Distribution shift
The model learns the statistics of its training data. When the world it meets has different statistics — different lighting, weather, camera angle, or population — performance drops even though nothing about the model changed. This is the single most common reason a model that looked excellent in testing disappoints in deployment.
Bias is shift with consequences
When a group is rare in the training data, the model has fewer examples to learn from and its errors concentrate on that group. Aggregate accuracy hides this because the group is a small fraction of the total. The fix starts with disaggregating the metrics — measuring performance per group rather than overall.
Recurring failure patterns
- Unusual lighting, weather, or camera angle the training data did not cover.
- Small, distant, or partially occluded objects.
- Rare viewpoints — a bicycle seen head-on rather than from the side.
- Classes that were rare or absent in the training labels.
- Populations underrepresented in the data, where errors cluster.
Reading a performance claim
A single accuracy number tells you almost nothing about whether a system is trustworthy. Ask what data it was measured on, whether that data resembles your situation, and how the errors are distributed. A model that is 95 percent accurate overall but 60 percent accurate on the group you care about is not a 95 percent model for your purposes.