Accurate but Uninformative | AI Learning Guide

The supplier shortcut example is the whole point in miniature. The model was not wrong on the test set — it genuinely separated the classes. It was right for a reason that has nothing to do with why the compounds work. And notice when that failure surfaces: not on the test set, which shares the same artifact, but on the first compound made a different way. That is the pattern to watch for. High accuracy tells you the model agrees with the labels you already have. It does not tell you the model would still be right if you changed something that matters.

0:00 / 0:00

Auto-play

The supplier shortcut

A model is trained to separate active from inactive compounds against one enzyme. It scores well on held-out compounds. Inspection shows it is partly keying on a synthesis by-product present in most of the active compounds — an artifact of how that series was made, not a cause of activity. The score was real; the understanding was not. Ask the same model about a compound made by a different route and the prediction collapses.

Accuracy is a statement about agreement with labels already collected. Mechanism is a statement about what would happen under a change — a different scaffold, a different target, a different patient. These are different claims, and the usual validation split tests only the first.