Skip to content
Learn Motion
ExploreHow it worksMembership
Log in
Learn Motion

How AI Learns from Rewards and Mistakes

1The Trial-and-Error Loop: How an Agent Learns2Formalizing the Problem: States, Actions, Rewards, and Policies3Measuring Success: Return, Discounting, and Value4Learning from Experience: Value Updates and Q-Learning5Exploring vs. Exploiting: Where Mistakes Come From6Learning Policies Directly and Scaling Up
Measuring Success: Return, Discounting, and Value

Discounting a Reward Sequence

1 / 2
Each bar is one reward in the trajectory, and the shaded part is what survives after discounting. At a low discount factor only the first bar keeps most of its height, so the return is close to the immediate reward. Raise the discount factor and the later bars grow, because gamma to a small power stays near one. Watch the total: the same reward sequence is worth much more to a patient agent. That is the whole point of the discount factor, it decides how far ahead the agent is effectively looking.
0:00 / 0:00

A trajectory is a sequence of rewards r_1, r_2, ..., r_T collected at successive time steps. Its return is the discounted sum \[G = r_1 + \gamma r_2 + \gamma^2 r_3 + \cdots = \sum_{t=1}^{T} \gamma^{\,t-1} r_t,\] where the discount factor gamma lies in the interval [0, 1). The exponent t-1 is the number of steps between the start of the trajectory and the reward r_t, so a reward received k steps later is multiplied by gamma to the power k. Because gamma is below 1, each successive power is smaller than the last, and rewards far in the future contribute less to G than rewards received soon.

The size of gamma sets the planning horizon. With gamma = 0 the return collapses to r_1, and the agent cares only about the immediate reward. With gamma = 0.9 a reward ten steps away is weighted by 0.9^10, roughly 0.35, so it still matters. With gamma = 0.99 the same reward is weighted by about 0.90, and the agent behaves almost as if it valued the distant future as much as the present. As gamma approaches 1 the sum can grow without bound on a non-terminating task, which is why gamma is normally kept strictly below 1.

Consider the reward sequence 0, 0, 10, 0, 0 received at steps 1 through 5. Under gamma = 0.5 the return is 0.5^2 * 10 = 2.5. Under gamma = 0.9 it is 0.9^2 * 10 = 8.1. The same trajectory is worth 2.5 or 8.1 depending only on how patient the agent is, and the more patient agent will prefer a policy that reaches the reward of 10 even if it takes several extra steps to get there.

References

  1. [1]Reinforcement Learning: An Introduction, Chapter 3 (Finite Markov Decision Processes)incompleteideas.net
Previous1 / 2Next

Learn Motion

Generate a course. Learn it properly.

Operated by Wuhan Daoyin Technology Co., Ltd.

Contact: [email protected]
Privacy PolicyTerms of Service

© 2026 Learn Motion