NOTES / HEALTHCARE AI

Evidence appraisal

How to judge a medical AI claim: GRADE certainty, the difference between reporting guidelines and risk-of-bias tools, and which guideline fits which study stage.

UPDATED 18 NOV 2026#healthcare#evidence#grade#tripod-ai#consort-ai

Concepts only. Not medical advice.

Two different questions

“Was this study reported clearly?” and “Is its evidence trustworthy?” are different. The EQUATOR Network makes the point directly: reporting guidelines say what to write up, but you cannot use them to judge whether a study was well designed.7

GRADE: certainty of a body of evidence

Diagram: GRADE start, rate down, rate up Diagram: my own summary of the GRADE logic.

GRADE rates certainty as high, moderate, low or very low. Randomised trials start high and non-randomised studies start low; the rating goes down for risk of bias, inconsistency, indirectness, imprecision or publication bias, and observational evidence can go up for a large effect or a dose-response pattern.1 A downgrade can be one or two levels, and the rating cannot fall below very low.2 GRADE was built for interventions and guidelines, so applying it to AI accuracy studies takes care.

Reporting guidelines by stage

Diagram: reporting guidelines mapped to AI study stage Diagram: my own mapping.

  • TRIPOD+AI guides reporting of prediction-model development and validation, covering regression and machine learning.3
  • PROBAST+AI is a tool for appraising risk of bias and applicability of such models.4
  • DECIDE-AI covers early live evaluation of decision support: small-scale clinical utility, safety, human factors and preparation for larger trials.6
  • CONSORT-AI (and its protocol twin SPIRIT-AI) extends the trial standards to AI interventions.5

A quick appraisal routine

  1. What is the claim, and which stage of evidence supports it?
  2. Was the model tested on data from other sites or times?
  3. Were clinicians in the loop, and what happened to patient outcomes?
  4. Is the study reported against the right guideline, and by whom was it funded?

Then see validation and, for the same instincts applied to LLM systems, why evals.

Sources

  1. [1]ACIP GRADE Handbook, Chapter 7: GRADE criteria determining certainty of evidence — US CDC · 2024PARAPHRASED
  2. [2]Assessing certainty of evidence — NHMRC (Australia) · 2024PARAPHRASED
  3. [3]TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods — Collins et al. · 2024PARAPHRASED
  4. [4]PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or AI methods — Moons et al. · 2025PARAPHRASED
  5. [5]Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension — Liu et al. · 2020PARAPHRASED
  6. [6]Reporting guideline for the early-stage clinical evaluation of decision support systems driven by AI: DECIDE-AI — Vasey et al. · 2022PARAPHRASED
  7. [7]Reporting guideline for the early-stage clinical evaluation of decision support systems driven by AI: DECIDE-AI (EQUATOR Network entry) — EQUATOR Network · 2022PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.