Why benchmarks and training data fall short | AI Learning Guide

Take the three weaknesses in turn. Test-set leakage is the subtlest: if the antibodies used to test the model were drawn from the same databases the model trained on, then the test set is not really unseen, and a high score partly measures recognition rather than design. Proxy mismatch is the most consequential: the benchmark scores predicted binding, but a usable antibody also has to fold, express, stay stable, and not provoke an immune response, and none of those are in the score. Simplification is the most misleading: benchmark tasks often fix the framework region or restrict the target class, which removes exactly the difficulty that makes a new target hard. A model can look strong on all three counts and still fail on a real problem.

0:00 / 0:00

Auto-play

Three recurring weaknesses in evaluation

Test-set leakage: because training and test data come from the same public databases, the test set is not truly unseen, and scores partly reflect familiarity.
Proxy mismatch: benchmarks score predicted binding, while the properties that decide whether an antibody is usable — stability, manufacturability, immunogenicity — are usually not scored at all.
Simplification: benchmark tasks fix parts of the problem, such as the framework or the target class, so they understate the difficulty of a genuinely novel target.

The gap between a proxy and the real outcome is not a technicality. A predicted binding score is computed from a model of molecular interaction; the measured affinity comes from an assay on a real protein in real buffer. Between the two sit folding, expression, aggregation, and measurement error. A benchmark that reports the first while the field needs the second will systematically overstate how close a design is to a usable drug.