Revealing data leakage in protein interaction benchmarks

Investor logo
Investor logo

Warning

This publication doesn't include Faculty of Sports Studies. It includes Faculty of Science. Official publication website can be found on muni.cz.
Authors

BUSHUIEV Anton BUSHUIEV Roman SEDLAR Jiri PLUSKAL Tomas DAMBORSKÝ Jiří MAZURENKO Stanislav SIVIC Josef

Year of publication 2024
Type Article in Proceedings
Conference ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design
MU Faculty or unit

Faculty of Science

Citation
Description In recent years, there has been remarkable progress in machine learning for protein-protein interactions. However, prior work has predominantly focused on improving learning algorithms, with less attention paid to evaluation strategies and data preparation. Here, we demonstrate that further development of machine learning methods may be hindered by the quality of existing train-test splits. Specifically, we find that commonly used splitting strategies for protein complexes, based on protein sequence or metadata similarity, introduce major data leakage. This may result in overoptimistic evaluation of generalization, as well as unfair benchmarking of the models, biased towards assessing their overfitting capacity rather than practical utility. To overcome the data leakage, we recommend constructing data splits based on 3D structural similarity of protein-protein interfaces and suggest corresponding algorithms. We believe that addressing the data leakage problem is critical for further progress in this research area.
Related projects:

You are running an old browser version. We recommend updating your browser to its latest version.

More info