While scientific research using AI is attracting attention, 'fact-checking of papers' that point out huge errors are being ignored

In recent years, AI research has become more common in fields such as biology, and it is expected that AI will greatly accelerate scientific research. However, while AI-based research papers are attracting attention, papers that fact-check their contents and point out a huge number of errors are being ignored, points out Rachel Thomas, founder of the Center for Applied Data Ethics at the University of San Francisco.

Rachel Thomas, PhD - Deep learning gets the glory, deep fact checking gets ignored
https://rachel.fast.ai/posts/2025-06-04-enzyme-ml-fails/index.html

The paper that Thomas discussed this time was 'Functional annotation of enzyme-encoding genes using deep learning with transformer layers,' published in 2023 by a research team from the Korea Advanced Institute of Science and Technology (KAIST) and the University of California, San Diego .

Functional annotation of enzyme-encoding genes using deep learning with transformer layers | Nature Communications
https://www.nature.com/articles/s41467-023-43216-z

Enzymes catalyze a wide variety of chemical reactions and are extremely important for understanding what occurs in living organisms. Enzymes are assigned an EC number (enzyme number) according to the reaction format they perform, creating a hierarchical classification system for thousands of different functions.

The paper in question is an AI model that uses Google's deep learning model, Transformer , to predict enzyme numbers from the amino acid sequence of an enzyme. The research team trained and verified the AI model using UniProt , a database containing over 22 million enzymes and their EC numbers, and predicted EC numbers for approximately 450 enzymes whose functions are unknown. The paper was published in Nature Communications , a world-renowned academic journal, and has been viewed approximately 22,000 times, ranking in the top 5% of Altmetric, an index that measures the popularity of online papers.

A paper investigating the results of this study and showing that there were many errors was subsequently published on the preprint server bioRxiv, but according to Thomas, this 'fact-checked' paper has received little attention compared to the original paper.

Limitations of Current Machine-Learning Models in Predicting Enzymatic Functions for Uncharacterized Proteins | bioRxiv
https://www.biorxiv.org/content/10.1101/2024.07.01.601547v2.full

In the original paper, after making predictions about 450 enzymes, three enzymes were tested, one of which, yciO, was studied by Valérie de Crécy-Lagard , a professor at the University of Florida, more than 10 years ago. In the original paper, yciO was predicted to have the same function as another enzyme, TsaC, but de Crécy-Lagard knew from years of experience that this was incorrect.

Previous studies have shown that TsaC is essential for E. coli even when yciO is overexpressed, suggesting that they have different functions. The activity of yciO reported in the original paper was 10,000-fold lower than that of TsaC, again indicating that the two enzymes do not perform the same function.

After noticing this error, de Crécy-Lagard and his team conducted a detailed investigation of all 450 enzymes whose functions were predicted in the original paper. As a result, 135 enzymes were already listed in the UniProt database used for training, and 148 enzymes had the same function repeated at a very high frequency, with very specific functions appearing up to 12 times, making it biologically inconceivable. In some cases, the biological context and detailed literature search confirmed that the predictions were biologically incorrect.

There are two ways to identify the function of an enzyme: 'applying known enzyme labels to enzymes in the same functional family' and 'discovering truly unknown functions.' The de Crécy-Lagard team pointed out that 'by design, supervised machine learning models cannot be used to predict the function of truly unknowns,' and warned that various errors can occur even when applying known enzyme labels.

'These papers remind us how difficult, if not impossible, it can be to evaluate AI claims in research outside one's field of expertise,' Thomas said. 'How many of these seemingly impressive papers stand up to rigorous scrutiny? The task of validating hundreds of enzyme predictions is less glamorous than the task of building the AI models that generated them, but it is no less important.'