跳到主要内容
Learn Motion
探索课程产品能力价格

下载 Learn Motion

用手机扫描二维码,下载 App。

App Store

扫码从 App Store 下载 Learn Motion打开商店页面

Google Play

扫码从 Google Play 下载 Learn Motion打开商店页面
登录
Learn Motion

How AI Learns from Rewards and Mistakes

1The Trial-and-Error Loop: How an Agent Learns2Formalizing the Problem: States, Actions, Rewards, and Policies3Measuring Success: Return, Discounting, and Value4Learning from Experience: Value Updates and Q-Learning5Exploring vs. Exploiting: Where Mistakes Come From6Learning Policies Directly and Scaling Up
Measuring Success: Return, Discounting, and Value

Discounting a Reward Sequence

1 / 2
Each bar is one reward in the trajectory, and the shaded part is what survives after discounting. At a low discount factor only the first bar keeps most of its height, so the return is close to the immediate reward. Raise the discount factor and the later bars grow, because gamma to a small power stays near one. Watch the total: the same reward sequence is worth much more to a patient agent. That is the whole point of the discount factor, it decides how far ahead the agent is effectively looking.
0:00 / 0:00

A trajectory is a sequence of rewards r_1, r_2, ..., r_T collected at successive time steps. Its return is the discounted sum
\[G = r_1 + \gamma r_2 + \gamma^2 r_3 + \cdots = \sum_{t=1}^{T} \gamma^{,t-1} r_t,\]
where the discount factor gamma lies in the interval [0, 1). The exponent t-1 is the number of steps between the start of the trajectory and the reward r_t, so a reward received k steps later is multiplied by gamma to the power k. Because gamma is below 1, each successive power is smaller than the last, and rewards far in the future contribute less to G than rewards received soon.

The size of gamma sets the planning horizon. With gamma = 0 the return collapses to r_1, and the agent cares only about the immediate reward. With gamma = 0.9 a reward ten steps away is weighted by 0.9^10, roughly 0.35, so it still matters. With gamma = 0.99 the same reward is weighted by about 0.90, and the agent behaves almost as if it valued the distant future as much as the present. As gamma approaches 1 the sum can grow without bound on a non-terminating task, which is why gamma is normally kept strictly below 1.

Consider the reward sequence 0, 0, 10, 0, 0 received at steps 1 through 5. Under gamma = 0.5 the return is 0.5^2 * 10 = 2.5. Under gamma = 0.9 it is 0.9^2 * 10 = 8.1. The same trajectory is worth 2.5 or 8.1 depending only on how patient the agent is, and the more patient agent will prefer a policy that reaches the reward of 10 even if it takes several extra steps to get there.

参考资料

  1. [1]Reinforcement Learning: An Introduction, Chapter 3 (Finite Markov Decision Processes)incompleteideas.net
上一页1 / 2下一页

Learn Motion

生成课程,认真学会。

运营方:Wuhan Daoyin Technology Co., Ltd.

联系我们: [email protected]
隐私政策用户服务协议

© 2026 Learn Motion