What a p-value Does and Does Not Tell You | AI Learning Guide

The two trials in the example make the point sharply. Both report a p-value of zero point zero four, so both clear the conventional significance threshold. But one found a risk difference of fifteen percentage points and the other a risk difference of half a percentage point. The p-value is the same because it is answering a question about compatibility with the null hypothesis, not a question about how large the effect is. A very large trial can push a trivial effect to a small p-value, and a small trial can leave a real and important effect with a large p-value simply because it lacked precision. So when you see a significant result, the p-value tells you the data are unlikely under the null, and nothing more. You still need the effect estimate and its confidence interval to know whether the result matters.

0:00 / 0:00

Auto-play

A p-value measures how incompatible the observed data are with the null hypothesis. It does not measure the size of the effect, the probability that the null is true, or the clinical importance of the result.

Two trials, the same p-value, different consequences

Trial A reports a risk difference of 15 percentage points with p = 0.04. Trial B reports a risk difference of 0.5 percentage points with p = 0.04. Both results are statistically significant at the conventional threshold, and both p-values are identical. Yet only Trial A describes a benefit large enough to change how a patient is treated. The p-value cannot distinguish them, because it was never designed to measure magnitude.

A p-value is not the probability that the null hypothesis is true, and 1 minus the p-value is not the probability that the alternative is true. Those quantities require prior assumptions that the p-value does not contain.