跳到主要内容
Learn Motion
探索课程产品能力价格

下载 Learn Motion

用手机扫描二维码,下载 App。

App Store

扫码从 App Store 下载 Learn Motion打开商店页面

Google Play

扫码从 Google Play 下载 Learn Motion打开商店页面
登录
Learn Motion

How AI Learns from Rewards and Mistakes

1The Trial-and-Error Loop: How an Agent Learns2Formalizing the Problem: States, Actions, Rewards, and Policies3Measuring Success: Return, Discounting, and Value4Learning from Experience: Value Updates and Q-Learning5Exploring vs. Exploiting: Where Mistakes Come From6Learning Policies Directly and Scaling Up
Measuring Success: Return, Discounting, and Value

From Return to Value

2 / 2
Look at the delivery robot. Moving toward the goal pays plus one right now, stepping into the bay pays minus one. By immediate reward the robot should always move toward the goal. But if the battery is nearly empty, that move leads to a shutdown with a large penalty, while the bay leads to a full charge and a clean run. So the bay has the higher action value despite the worse immediate reward. That is the difference between the one-step signal the environment hands you and the long-run consequence you actually care about.
0:00 / 0:00

Two quantities, one difference

The state value \(V(s)\) is the expected return from state s under policy pi. The action value \(Q(s, a)\) is the expected return after taking action a in state s and following pi afterward. Both are expectations over future rewards, so both measure long-run consequence rather than immediate payoff.

A cheap action with a high value

A delivery robot in a corridor can move toward the goal, or step into a side bay to recharge. Stepping into the bay costs one unit of reward, so its immediate reward is -1, while moving toward the goal earns +1. If the battery is nearly empty, moving toward the goal leads to a shutdown state with a large negative reward, and the bay leads to a full charge and a clean run to the goal. Then \(Q(s, \text{bay}) > Q(s, \text{toward goal})\) even though the bay's immediate reward is worse. The value function sees the shutdown that the immediate reward does not.

Why value is the right basis for choosing

If the agent picked actions by immediate reward alone, it would be greedy in the worst sense: it would take the largest one-step payoff even when that payoff ends the episode badly. Value functions fix this by folding the entire future into a single number per state or per state-action pair. Once those numbers are accurate, choosing the action with the largest \(Q(s, a)\) is exactly choosing the action with the best long-run consequence. The remaining problem is that the numbers are unknown and must be learned from reward experience, which is where update rules enter.

参考资料

  1. [1]Reinforcement Learning: An Introduction, Chapter 3 (Finite Markov Decision Processes)incompleteideas.net
上一页2 / 2下一页

Learn Motion

生成课程,认真学会。

运营方:Wuhan Daoyin Technology Co., Ltd.

联系我们: [email protected]
隐私政策用户服务协议

© 2026 Learn Motion