Accumulating adenosine only matters because something reads it. Two receptor subtypes do most of that reading in the context of sleep and arousal: A1 and A2A. Both are G-protein-coupled receptors that couple to Gi or Gs signaling depending on subtype and cell type, and both are engaged by adenosine at concentrations that rise across a waking period.
When adenosine binds these receptors on wake-promoting populations, the result is inhibition. In the tuberomammillary nucleus, A1 receptor activation reduces the firing of histaminergic neurons that help sustain arousal. In the basal forebrain, A2A receptor activation inhibits cholinergic neurons that support cortical activation and attention. Adenosine also acts in the ventrolateral preoptic area, where its signaling promotes the activity of sleep-active neurons and thereby suppresses the arousal systems they project to. The net effect across these sites is the same: the circuits that keep you alert are turned down.
This is why adenosine is described as an inhibitory sleep-promoting signal. It does not create sleepiness by adding an arousing signal; it produces sleepiness by subtracting from the arousal systems already running. The higher the adenosine concentration, the more receptors are occupied, and the more strongly those wake-promoting circuits are suppressed. That relationship, concentration in and arousal out, is the target that caffeine will later interfere with.