Groundedness: The Metric That Catches Confident AI Lies

Groundedness is the LLM metric that scores whether every factual claim in an AI answer is supported by the evidence the system actually retrieved. The procedure is simple to state: split the answer into individual claims, check each claim against the retrieved context or tool results, and report the fraction supported. An answer where three of four claims trace back to evidence scores 0.75. The reason this one metric earns its keep: it does not ask whether the answer is true in general, it asks whether the answer is backed by what the system saw, which is exactly the question that catches an AI confidently summarizing an empty search result or embellishing beyond its sources. Accuracy needs ground truth you usually do not have in production. Groundedness only needs the trace.

How is groundedness actually measured?

Two moving parts. First, claim decomposition. Openlayer's groundedness guide defines the metric as measuring "whether each factual claim in your RAG system's output traces back to the retrieved context", and the mechanics follow from the definition: the response is split into individual factual statements, and each is verified separately against the context.

Second, the judge. In production this checking is done by an LLM-as-judge that receives the question, the retrieved evidence, and the answer, then flags unsupported claims and returns a score with reasoning. Openlayer reports judge models reaching about 80 percent agreement with human evaluators on groundedness tasks, which is good enough to be useful and low enough to justify validating your judge before trusting it, a discipline I covered in LLM Evals Explained for Product Leaders.

The open-source reference implementation is Ragas, whose faithfulness metric is the same idea under its other common name: "a response is considered faithful if all its claims can be supported by the retrieved context", scored 0 to 1 as supported claims over total claims. Their worked example is instructive in its bluntness: one supported statement plus one unsupported statement scores 0.5.

Why does it catch what other checks miss?

Because it compares the answer to the evidence rather than to the world. The most dangerous production failure is not the answer that is wrong about a fact you can look up; it is the answer that is fluent, plausible, and built on nothing: the tool returned an empty result, the retrieval missed, and the model wrote a confident paragraph anyway. Status codes are green, latency is normal, and no accuracy benchmark exists for this one user's question. A groundedness check sees it immediately, because the claims trace back to an empty context. I dramatized exactly this failure as "the confident liar" in Your AI Agent Has the Same Red Flags as Your Ex: empty input, confident output, red verdict.

The flip side matters just as much. On a healthy run, the same judge goes green, and that is the sentence your leadership actually wants: not "we think it works" but "answers are scored against their evidence, and here is the score."

What score should you require?

Calibrate rather than copy, but the published starting points cluster usefully. Openlayer's guidance: critical domains should stay above 0.85, while something like customer support "might accept 0.75", with pass and fail criteria defined before deployment rather than negotiated after an incident. Two of their operational points are worth adopting verbatim. Watch trends, not just thresholds: their example alert is a drift from 0.88 down to 0.76, which is a regression story long before it is a threshold story. And you do not need to score everything: starting with roughly 10 percent of production traffic balances signal against judge cost, then adjust.

One honest caveat: these thresholds are practitioner guidance, not standards, and the right number depends on how costly a fabricated sentence is in your product.

Where does it fit in an agent stack?

Groundedness is usually the first of the three to five evaluations worth running, because it covers the failure mode with the worst blast radius for trust. It slots into the stack as a scored check on every traced run: trace captures what the tools returned, the judge scores the answer against it, and the score lands next to cost and latency. If you want to see what a rigorous groundedness judge prompt looks like for your own agent, my free tool Judgekit generates one from a pasted trace. And setting this up end to end on real traffic, judge validation included, is the core of my Agent Trust Sprint: 2 weeks at about 4 working days, EUR 2,400 fixed.

FAQ

Is groundedness the same as accuracy? No. An answer can be grounded and wrong, if the retrieved evidence itself was wrong, and correct but ungrounded, if the model knew something its sources did not say. Groundedness tells you the system is honest about its evidence; retrieval quality tells you the evidence was worth being honest about. You need both signals.

Groundedness or faithfulness, which name is right? They name the same core idea: claims supported by context. Vendor vocabularies differ at the edges, with some treating one as a hard gate and the other as a continuous score. Agree on one definition inside your team and ignore the naming wars.

Can the judge itself hallucinate? Yes, which is why the roughly 80 percent human agreement figure matters: validate your judge on a labeled sample before trusting it, and re-validate when you change its rubric or model.

Does this only apply to RAG? No. Any system whose answers should follow from tool results can be scored this way: an agent summarizing tickets, an assistant reading your calendar, a support bot quoting policy. If there is evidence in the trace, groundedness is measurable.