Reinforcing Actions Instead of Scoring Them | AI Learning Guide

The key shift here is what gets stored. Q-learning stores a number per action and then picks the largest; the policy is a side effect. A policy gradient method stores the policy itself, as a probability distribution over actions, and tunes its parameters so that actions followed by high return become more likely. The update multiplies two things: the direction that makes the taken action more probable, and the return that followed. If the return was good, the action is reinforced; if it was bad, the same step is reversed and the action is suppressed. Notice the continuous-control example: when the action is a real-valued torque, there is no finite list to take a maximum over, so scoring every action is not an option. Outputting a distribution over torques and shifting its mean is.

0:00 / 0:00

Auto-play

The policy becomes the object being learned

Instead of estimating the value of each action and then choosing the best, a policy-based agent represents the policy itself as \(\pi_\theta(a \mid s)\) and adjusts the parameters \(\theta\) so that actions leading to higher return become more probable. There is no separate value table driving the decision.

Why the update looks like a weighted nudge

The gradient estimate \(\nabla_\theta J(\theta) \approx \sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, G_t\) has two factors. The first, \(\nabla_\theta \log \pi_\theta(a_t \mid s_t)\), is the direction in parameter space that increases the probability of the action that was taken. The second, \(G_t\), is the return that followed. Multiplying them means the parameter step is large and positive when a high-return action was taken, and reversed when the return was negative. The agent is not told which action was correct; it is told how good the outcome was, and the gradient machinery distributes credit to the actions that preceded it.

Where the two families differ

Value-based (Q-learning)

Learns \(Q(s,a)\), then acts greedily on it
Policy is implicit and derived after the fact
Needs a finite set of actions to take a maximum over
Reuses each transition to update one value estimate

Policy-based (policy gradient)

Learns \(\pi_\theta(a \mid s)\) directly
No value table needed for action selection
Handles continuous actions by outputting distribution parameters
Can represent stochastic policies such as 0.7/0.3

A continuous control case

Consider a robot arm that must choose a joint torque, a real number anywhere from \(-5\) to \(5\). A value-based agent would need to score infinitely many torques and pick the maximum, which is not directly possible without discretizing the range and losing precision. A policy-based agent can output the mean and standard deviation of a Gaussian over torques, and the gradient update shifts that mean toward torques that produced higher return. The action space never had to be enumerated.

References

[1]Reinforcement Learning: An Introduction, 2nd edition — Chapter 13: Policy Gradient Methodsincompleteideas.net