Rating Certainty of Evidence Across Domains | AI Learning Guide

The example on this page is where the domains stop being a list and start being a judgment. Eight well-conducted trials, a pooled relative risk of 0.85, a confidence interval that excludes no effect — on the surface, a strong result. But walk through the domains. Two trials have unclear allocation concealment, so risk of bias is downgraded. The I-squared is 60 percent, so inconsistency is downgraded. The funnel plot is asymmetric, so publication bias is downgraded. Indirectness and imprecision are fine. Three downgrades from high leaves low certainty. Notice what that means: the estimate is precise and the effect looks real, but our confidence in it is low because the body of evidence has problems that no single number reveals. That is the whole point of rating certainty across domains — it forces you to look past the headline estimate at the quality of the evidence behind it.

0:00 / 0:00

Auto-play

The five downgrading domains

For randomized evidence, certainty starts high and is downgraded for problems in any of five domains. Each domain is a structured question about the body of evidence, and the judgments from earlier chapters feed directly into them.

Risk of bias: are the studies' internal validity safeguards — randomization, allocation concealment, blinding, complete follow-up — actually intact?
Inconsistency: do the studies agree, or is there unexplained heterogeneity?
Indirectness: do the population, intervention, comparator, and outcome match the question you are trying to answer?
Imprecision: is the confidence interval narrow enough to support a decision, or does it include both meaningful benefit and meaningful harm?
Publication bias: is the set of studies likely to be a biased sample of all studies conducted?

Applying the domains to a body of evidence

Suppose a systematic review of eight randomized trials finds that a drug reduces the risk of a composite cardiovascular outcome, with a pooled relative risk of 0.85 and a confidence interval of 0.78 to 0.93. The trials are generally well conducted, but two have unclear allocation concealment, the I-squared is 60 percent, and the funnel plot is asymmetric. Working through the domains: risk of bias is downgraded once for the two unclear trials; inconsistency is downgraded once for the unexplained heterogeneity; publication bias is downgraded once for the funnel asymmetry; indirectness is not downgraded because the population and outcome match the question; imprecision is not downgraded because the interval is narrow and excludes no effect. Three downgrades from high gives a rating of low certainty. The recommendation that follows would have to acknowledge that the true effect could be appreciably different from the estimate.

Certainty is not the same as effect size

A high-certainty rating means we are confident the effect is close to the estimate; it does not mean the effect is large or clinically important. A large effect estimated with low certainty and a small effect estimated with high certainty are both possible, and they call for different recommendations. Keep the two judgments separate.

References

[1]GRADE Handbook: Rating quality of evidence and strength of recommendationsgdt.gradepro.org