Why One Model Can Explain Itself and the Other Cannot | AI Learning Guide

The reason traditional models explain themselves is almost an accident of how they are built. Someone already decided that income, debt, and repayment history were the things that matter, and gave them names. So when the model produces a decision, you can point back at those same named quantities and say which one pushed the result. A deep model never received that list. It was given raw data and adjusted a huge number of internal settings until its outputs matched the examples, and nobody assigned a meaning to any of those settings. So there is no named quantity to point at, and the prediction cannot be traced back to a readable reason. Notice that this is not about the model being secret or badly built. It is about the vocabulary the model thinks in. One thinks in your words, the other thinks in its own. That is why the same accuracy can mean two very different things depending on whether you have to justify an individual decision.

0:00 / 0:00

Auto-play

Traditional models reason in human vocabulary

The inputs to a traditional model were chosen and named by a person, so the model's internal quantities are the person's own concepts. A prediction can be explained by pointing at those named inputs and their contributions, using the same words the inputs were written in.

Deep models store knowledge without assigned meaning

A deep model's knowledge lives in a very large number of internal parameters that no person named or interpreted. They were adjusted during training until the outputs matched the examples. Because no parameter corresponds to a human concept such as "existing debt," a single prediction cannot be traced back to a readable reason. That is the sense in which the model is a black box.

Interpretability is a requirement, not a score

Whether opacity is acceptable depends on what the prediction is used for. A decision about credit, employment, or medical care may need to be justified to the affected person or to a regulator, and there the ability to explain an individual case can outweigh a gain in accuracy. For low-stakes uses such as tagging personal photos, the reason rarely matters. The trade-off is real, but the situation decides which side of it you are on.