What a stuck agent looks like
The agent repeats one action from a state, its return stops improving, and the Q-values of the other actions from that state have not changed for many episodes. The values are not wrong in the sense of a bug; they are simply unvisited. No update has ever been applied to them, so they still hold whatever they were initialized to.
Why the greedy rule causes it
Recall that the Q-learning target for a transition is \(r + \gamma \max_{a'} Q(s', a')\), and the update moves \(Q(s, a)\) a fraction \(\alpha\) toward that target. The update only fires for the action \(a\) that was actually taken. If the greedy rule always takes \(a_1\), then \(Q(s, a_2)\) and \(Q(s, a_3)\) are never on the left side of an update, so they never move. A better action can sit at a low initial value indefinitely, and the greedy rule will keep avoiding it precisely because its value is low.
A corridor agent that never turns
An agent starts in a corridor with two exits. Turning left reaches a small reward of 1; turning right reaches a reward of 10, but the right turn looks blocked for the first few steps because a door opens only after the agent has moved twice. With \(\epsilon = 0\) the agent turns left, updates \(Q(\text{start}, \text{left})\) upward, and turns left forever. The right-turn value stays at its initial 0. Raising \(\epsilon\) to 0.1 makes the agent turn right about once in ten episodes. On those episodes it eventually passes the door, collects the 10, and the update pushes \(Q(\text{start}, \text{right})\) above 1. The greedy branch then switches on its own. The fix was not a change to the update rule; it was a change to how actions were selected.
Choosing a schedule
A fixed \(\epsilon\) is rarely the right answer. Two common schedules are constant \(\epsilon\) and decaying \(\epsilon\), where \(\epsilon\) starts near 1 and is multiplied by a factor slightly below 1 each episode, or reduced on a fixed schedule. Decay is appropriate when the environment is stationary and the agent has a long learning budget: explore widely at first, then exploit. A constant, small \(\epsilon\) is appropriate when the environment can change, because the agent must keep sampling to notice that a previously good action has become bad. If a bad action is expensive or dangerous, \(\epsilon\) should be small and the decay fast, accepting slower discovery in exchange for fewer costly mistakes. There is no schedule that is best for every task; the choice trades information against the cost of gathering it.
A common confusion
Decaying \(\epsilon\) to zero does not make the agent optimal. It makes the agent stop learning about actions it has not yet tried. If the decay is too fast, the agent freezes a suboptimal policy with high confidence, which is harder to notice than an agent that is still visibly exploring.