Workshop ReportA More Informative and Reproducible Remote Homology Evaluation for Protein Language ModelsMoldwin, Asher; Kabir, Anowarul; Shehu, AmardaRecent studies exploring the abilities of transformer-based protein language models have highlighted their performance on the task of remote homology detection, but have not provided datasets or evaluation procedures geared toward properly measuring performance on this task. With the goal of obtaining more informative and reproducible results, we offer a detailed procedure for constructing datasets and evaluating remote homology detection performance in a way that allows detailed analyses to be performed that shed light on the remote homology detection performance throughout the “twilight zone” of low sequence similarity. Using the proposed procedures, we found that three stateof-the-art protein language models exhibit diminishing performance when the pairwise sequence similarity between the query sequence and other proteins is restricted to below 35% identity.LLMs4Bio2024-02-2610526869https://doi.org/2310113National Science Foundation