跳到主要内容
Learn Motion
探索课程产品能力价格

下载 Learn Motion

用手机扫描二维码,下载 App。

App Store

扫码从 App Store 下载 Learn Motion打开商店页面

Google Play

扫码从 Google Play 下载 Learn Motion打开商店页面
登录
Learn Motion

How AI Learns from Rewards and Mistakes

1The Trial-and-Error Loop: How an Agent Learns2Formalizing the Problem: States, Actions, Rewards, and Policies3Measuring Success: Return, Discounting, and Value4Learning from Experience: Value Updates and Q-Learning5Exploring vs. Exploiting: Where Mistakes Come From6Learning Policies Directly and Scaling Up
Exploring vs. Exploiting: Where Mistakes Come From

Epsilon-Greedy in Action: Watching Which Actions Get Tried

1 / 2
Start with epsilon at zero and run a batch of decisions. The agent always takes the tallest bar, the action with the highest estimate, and the other two bars never move. Now raise epsilon to about a fifth. Roughly one decision in five comes from the random branch, so the short bar gets sampled, and if it is genuinely better its height starts to climb. Watch the branch indicator: teal means the agent exploited the table, amber means it deliberately tried something else. Every amber step probably earns less reward than the greedy choice would have, and that is the price of the information the table still needs.
0:00 / 0:00

Epsilon-greedy action selection splits every decision into two branches. With probability \(1 - \epsilon\) the agent exploits: it takes \(a = \arg\max_{a} Q(s, a)\), the action with the highest current estimate. With probability \(\epsilon\) it explores: it picks an action uniformly at random from the action space, regardless of the estimates.

The parameter \(\epsilon\) is the exploration rate, a number between 0 and 1. At \(\epsilon = 0\) the agent is purely greedy and never tries anything new. At \(\epsilon = 1\) it ignores the table entirely and behaves randomly. Values in between mix the two behaviors, and the mix is what matters: the greedy branch keeps the agent collecting reward, while the random branch keeps feeding new transitions into the Q-learning update from the previous chapter.

Consider a state with three actions whose current estimates are \(Q(s, a_1) = 5.0\), \(Q(s, a_2) = 4.8\), and \(Q(s, a_3) = 0.2\). Under a purely greedy rule the agent takes \(a_1\) forever. The estimate for \(a_3\) never changes, so if \(a_3\) is actually the best action, the table can never discover it. Raising \(\epsilon\) to 0.2 means roughly one decision in five is random, so \(a_3\) gets sampled occasionally, its estimate is updated, and if it is genuinely better its value will climb past 5.0 and the greedy branch will start selecting it.

There is a cost. Every exploratory step is a step that probably earns less reward than the greedy choice would have. Exploration buys information at the price of short-term performance, and the size of \(\epsilon\) sets that price. This is why the exploration rate is usually not held constant: a large \(\epsilon\) early on gathers information quickly, and a small \(\epsilon\) later lets the agent cash in on what it learned.

参考资料

  1. [1]Sutton & Barto, Reinforcement Learning: An Introduction, Section 2.3 (epsilon-greedy action selection)incompleteideas.net
上一页1 / 2下一页

Learn Motion

生成课程,认真学会。

运营方:Wuhan Daoyin Technology Co., Ltd.

联系我们: [email protected]
隐私政策用户服务协议

© 2026 Learn Motion