A model that has already been shown a drug's outcome can reproduce that outcome without having learned anything transferable. If you test it on the same drugs it trained on, you are measuring its memory, not its judgment. The fix is to split the known drugs before learning begins: one group is used to build the pattern, and a second group is locked away and never shown to the model during that process. Only afterward is the model asked to score the locked-away drugs, and its answers are compared with what actually happened to them.
This held-out group is a stand-in for the real situation. When the model is later pointed at a brand-new candidate, that candidate's outcome is genuinely unknown, and the model has never seen it. Testing on held-out drugs reproduces that condition as closely as the available history allows. A model that scores well on drugs it memorized but poorly on held-out drugs has learned the training set rather than the underlying pattern — a failure mode usually called overfitting, meaning the model has fitted the noise and quirks of its examples instead of a relationship that carries over.
One caveat matters even here: if the held-out drugs are drawn from the same narrow slice of history as the training drugs, the test is easier than reality. A convincing test holds back drugs that differ in some meaningful way — a different chemical family, a different era, a different disease area — so that success on it is evidence the pattern travels.