Epsilon-greedy action selection splits every decision into two branches. With probability \(1 - \epsilon\) the agent exploits: it takes \(a = \arg\max_{a} Q(s, a)\), the action with the highest current estimate. With probability \(\epsilon\) it explores: it picks an action uniformly at random from the action space, regardless of the estimates.
The parameter \(\epsilon\) is the exploration rate, a number between 0 and 1. At \(\epsilon = 0\) the agent is purely greedy and never tries anything new. At \(\epsilon = 1\) it ignores the table entirely and behaves randomly. Values in between mix the two behaviors, and the mix is what matters: the greedy branch keeps the agent collecting reward, while the random branch keeps feeding new transitions into the Q-learning update from the previous chapter.
Consider a state with three actions whose current estimates are \(Q(s, a_1) = 5.0\), \(Q(s, a_2) = 4.8\), and \(Q(s, a_3) = 0.2\). Under a purely greedy rule the agent takes \(a_1\) forever. The estimate for \(a_3\) never changes, so if \(a_3\) is actually the best action, the table can never discover it. Raising \(\epsilon\) to 0.2 means roughly one decision in five is random, so \(a_3\) gets sampled occasionally, its estimate is updated, and if it is genuinely better its value will climb past 5.0 and the greedy branch will start selecting it.
There is a cost. Every exploratory step is a step that probably earns less reward than the greedy choice would have. Exploration buys information at the price of short-term performance, and the size of \(\epsilon\) sets that price. This is why the exploration rate is usually not held constant: a large \(\epsilon\) early on gathers information quickly, and a small \(\epsilon\) later lets the agent cash in on what it learned.