Two quantities, one difference
The state value \(V(s)\) is the expected return from state s under policy pi. The action value \(Q(s, a)\) is the expected return after taking action a in state s and following pi afterward. Both are expectations over future rewards, so both measure long-run consequence rather than immediate payoff.
A cheap action with a high value
A delivery robot in a corridor can move toward the goal, or step into a side bay to recharge. Stepping into the bay costs one unit of reward, so its immediate reward is -1, while moving toward the goal earns +1. If the battery is nearly empty, moving toward the goal leads to a shutdown state with a large negative reward, and the bay leads to a full charge and a clean run to the goal. Then \(Q(s, \text{bay}) > Q(s, \text{toward goal})\) even though the bay's immediate reward is worse. The value function sees the shutdown that the immediate reward does not.
Why value is the right basis for choosing
If the agent picked actions by immediate reward alone, it would be greedy in the worst sense: it would take the largest one-step payoff even when that payoff ends the episode badly. Value functions fix this by folding the entire future into a single number per state or per state-action pair. Once those numbers are accurate, choosing the action with the largest \(Q(s, a)\) is exactly choosing the action with the best long-run consequence. The remaining problem is that the numbers are unknown and must be learned from reward experience, which is where update rules enter.