Writing a Task in Reinforcement Learning Notation | Visual AI Learning Guide

Look at the grid and the four labeled slots beside it. The state slot holds the robot's cell, so the state set is every cell it could occupy. The action slot holds the four moves, and notice that a corner cell has only two legal moves, which is why the action set can depend on the state. The reward slot holds a number attached to each transition: plus ten for entering the goal, minus one for a wall, and a small minus for every ordinary step. The policy slot holds a rule that reads a state and returns an action. The important point is that only the policy changes when the robot learns; the other three slots are the problem definition and stay fixed.

0:00 / 0:00

Auto-play

A reinforcement learning problem is written as a tuple of four objects. The state set \(S\) collects every situation the decision-maker can be in. The action set \(A\) collects every choice available; when the available choices depend on the situation, this is written \(A(s)\) for the actions legal in state \(s\). The reward function \(R(s, a, s')\) returns a number for moving from state \(s\) by action \(a\) into state \(s'\). The policy \(\pi\) maps a state to an action, written \(a = \pi(s)\), or to a probability over actions, written \(\pi(a \mid s)\), when the behavior is stochastic.

Take a delivery robot on a small grid. The state is its cell, for example \(s = (2, 3)\). The action space is \(A = \{\text{up}, \text{down}, \text{left}, \text{right}\}\), and in a corner cell only two of those are legal, so \(A(s)\) has two elements there. The reward function gives \(+10\) on entering the goal cell, \(-1\) on hitting a wall, and \(-0.01\) for every other move so that dawdling is mildly penalized. A candidate policy is: if the goal is to the right, move right; otherwise move up. That policy is a complete rule for behaving, and it can be evaluated and improved without changing the state set, the action set, or the reward function.

The notation is a modeling choice, not a discovery. A different engineer could define the state as the cell plus the remaining battery level, keep the same action space, and add a large negative reward for reaching zero battery. Both models describe the same robot, but they support different questions: the first can learn a route, the second can learn a route that also respects energy.

References

[1]Reinforcement Learning: An Introduction, Chapter 3: Finite Markov Decision Processesincompleteideas.net