A state is the information that determines what happens next; an observation is what the agent actually receives. They coincide only when the agent has full information about everything that matters.
When the agent sees less than the state
A card game makes the gap concrete. The full state includes every player's hand, but a player observes only their own cards and the cards already played. Two situations with identical observations can require different actions, because the hidden cards differ. An agent that treats its observation as if it were the state will make decisions that look inconsistent: the same input leads to different correct answers. The usual repairs are to keep a memory of past observations or to maintain a belief about the hidden part. Both are ways of reconstructing enough of the state to decide well.
State design changes the problem
Because the state is chosen rather than given, the same task can be modeled several ways. A thermostat task can use the state "current temperature" alone, or "current temperature plus whether the heater is on." The second model can express that a heater already running should not be switched on again; the first cannot, because that fact is not in the state. Adding information to the state enlarges the state set and slows learning, since each distinct state must be experienced. Leaving out information makes the state set smaller but can make the task unsolvable, because the missing fact is exactly what the decision depends on. The practical rule is to include what the next outcome depends on, and nothing more.
Episodes and horizon
An episode is one run from a start state to a terminal state, such as one game or one delivery trip. The horizon is how far ahead the task is scored. A task with a short horizon rewards quick gains; a task with a long or unbounded horizon can reward a slow build-up that pays off much later. This is why the same reward function can produce different behavior depending on how far the episode is allowed to run, and it is the reason the next chapter has to define a quantity that sums rewards over time rather than looking at one step.