跳到主要内容
Learn Motion
探索课程产品能力价格

下载 Learn Motion

用手机扫描二维码,下载 App。

App Store

扫码从 App Store 下载 Learn Motion打开商店页面

Google Play

扫码从 Google Play 下载 Learn Motion打开商店页面
登录
Learn Motion

How AI Learns from Rewards and Mistakes

1The Trial-and-Error Loop: How an Agent Learns2Formalizing the Problem: States, Actions, Rewards, and Policies3Measuring Success: Return, Discounting, and Value4Learning from Experience: Value Updates and Q-Learning5Exploring vs. Exploiting: Where Mistakes Come From6Learning Policies Directly and Scaling Up
Exploring vs. Exploiting: Where Mistakes Come From

Diagnosing a Stuck Agent and Choosing an Exploration Schedule

2 / 2
Look at the corridor example closely, because it shows the whole diagnosis. The agent starts between two exits. Turning left pays one, turning right pays ten, but the right door only opens after two moves, so early on the right turn looks worthless. With epsilon at zero the agent turns left, the update raises that value, and it turns left forever. The right-turn value is never updated, so it stays at its initial zero. Raise epsilon to a tenth and the agent turns right about once in ten episodes, reaches the door, collects the ten, and the update pushes the right value above the left. The greedy branch then switches by itself. Nothing about the update rule changed; only the way actions were selected changed.
0:00 / 0:00

What a stuck agent looks like

The agent repeats one action from a state, its return stops improving, and the Q-values of the other actions from that state have not changed for many episodes. The values are not wrong in the sense of a bug; they are simply unvisited. No update has ever been applied to them, so they still hold whatever they were initialized to.

Why the greedy rule causes it

Recall that the Q-learning target for a transition is \(r + \gamma \max_{a'} Q(s', a')\), and the update moves \(Q(s, a)\) a fraction \(\alpha\) toward that target. The update only fires for the action \(a\) that was actually taken. If the greedy rule always takes \(a_1\), then \(Q(s, a_2)\) and \(Q(s, a_3)\) are never on the left side of an update, so they never move. A better action can sit at a low initial value indefinitely, and the greedy rule will keep avoiding it precisely because its value is low.

A corridor agent that never turns

An agent starts in a corridor with two exits. Turning left reaches a small reward of 1; turning right reaches a reward of 10, but the right turn looks blocked for the first few steps because a door opens only after the agent has moved twice. With \(\epsilon = 0\) the agent turns left, updates \(Q(\text{start}, \text{left})\) upward, and turns left forever. The right-turn value stays at its initial 0. Raising \(\epsilon\) to 0.1 makes the agent turn right about once in ten episodes. On those episodes it eventually passes the door, collects the 10, and the update pushes \(Q(\text{start}, \text{right})\) above 1. The greedy branch then switches on its own. The fix was not a change to the update rule; it was a change to how actions were selected.

Choosing a schedule

A fixed \(\epsilon\) is rarely the right answer. Two common schedules are constant \(\epsilon\) and decaying \(\epsilon\), where \(\epsilon\) starts near 1 and is multiplied by a factor slightly below 1 each episode, or reduced on a fixed schedule. Decay is appropriate when the environment is stationary and the agent has a long learning budget: explore widely at first, then exploit. A constant, small \(\epsilon\) is appropriate when the environment can change, because the agent must keep sampling to notice that a previously good action has become bad. If a bad action is expensive or dangerous, \(\epsilon\) should be small and the decay fast, accepting slower discovery in exchange for fewer costly mistakes. There is no schedule that is best for every task; the choice trades information against the cost of gathering it.

A common confusion

Decaying \(\epsilon\) to zero does not make the agent optimal. It makes the agent stop learning about actions it has not yet tried. If the decay is too fast, the agent freezes a suboptimal policy with high confidence, which is harder to notice than an agent that is still visibly exploring.

参考资料

  1. [1]Sutton & Barto, Reinforcement Learning: An Introduction, Section 2.3 (epsilon-greedy action selection)incompleteideas.net
上一页2 / 2下一页

Learn Motion

生成课程,认真学会。

运营方:Wuhan Daoyin Technology Co., Ltd.

联系我们: [email protected]
隐私政策用户服务协议

© 2026 Learn Motion