Lighting
A gradient measures change in recorded intensity, not change in the physical world. Dim the scene and every change shrinks, so weak but real edges drop below the threshold. Brighten it unevenly and shadows produce strong gradients that mark nothing real. The same object, the same detector, two different results.
Pose and scale
A corner viewed head-on can appear as a gentle curve from an oblique angle, so the two-direction change disappears and the keypoint is lost. Scale causes a parallel failure: a detector tuned to change over a few pixels finds nothing when the object fills the frame, and a detector tuned to large structures misses fine detail.
The same building, two photographs
A detector tuned on daytime street photos of a building finds plenty of strong corners and edges. Photograph the same building at night, lit only by a few windows, and most of those edges vanish below threshold while the window frames produce bright gradients that were never the structure you cared about. The detector has not learned anything; it is applying the same fixed rule to numbers that now mean something different.
The limitation is not that hand-designed features are wrong, but that they are fixed. Because the measurements are chosen in advance, every new condition requires new engineering, and fixes for one condition often degrade another. That is the gap that learned features were meant to close.