An answer is not a data point
When an AI tool answers a question, summarizes a document or classifies a set of records, it delivers a result that looks a lot like data: it is neatly formatted, written with confidence and often correct. But it is not observed data. It is the output of a model, and like any model output, it can be wrong.
This chapter proposes looking at those answers the same way the first edition looked at an estimate. Point and interval estimation stated that estimating is approximating, and that the variation in the results of estimators plays a fundamental role in managing uncertainty. Something similar happens with a machine's answers: the point is not to trust or distrust them wholesale, but to measure how often they are right, on which tasks and with what variability.
Why they get things wrong
Language models generate text by choosing, word by word, plausible continuations based on what they learned during training. That way of working explains both their fluency and their main flaw: they can produce statements that are plausible but false, such as a citation that does not exist, a made-up figure or a library function with a name that sounds right. This phenomenon is usually called hallucination.
The causes are varied and are still being researched. A 2025 paper, for example, argues that the training and evaluation procedures themselves tend to reward a model for venturing an answer rather than admitting it does not know, in the same way that on a multiple-choice exam with no penalty for wrong answers it pays to guess rather than leave a question blank. Whatever the explanation, the practical consequence is clear: the tone of an answer says nothing about its accuracy.
The answer as a sample
One peculiarity of these tools is that they can give different answers to the same question. This, puzzling at first sight, admits a familiar statistical reading: each answer can be thought of as an observation of a random variable, with its own distribution.
From that perspective, asking once amounts to drawing a sample of size one. If the question is repeated several times and the answers agree, there is some evidence of stability (though not necessarily of accuracy: a model can be consistently wrong). If the answers vary widely, that dispersion is in itself a sign that the tool has no solid basis for answering. The notions of variability covered in Describing variables and of sampling distribution covered in Applied probability find a direct application here.
Building an evaluation set
The most powerful verification tool is also the simplest: a set of cases whose correct answer is known in advance. For example, to use a model to classify customer complaints into categories, one can take a sample of complaints, classify them carefully by hand and compare that classification with the model's.
That comparison is nothing other than the confusion matrix presented in Logistic regression, with its associated measures: accuracy, sensitivity, specificity and precision. And since the evaluation sample is exactly that, a sample, the results are subject to sampling error: an accuracy of 90% measured on 20 cases says much less than the same accuracy measured on 500. What was covered in Sampling techniques about sample size and representativeness applies unchanged: if the evaluation cases are only the easy ones, the measurement will be optimistic.
Some recommendations for building these sets:
- Include hard cases and edge cases, not just the typical ones.
- Keep evaluation separate from tuning. If the tool's instructions are adjusted while looking at the same cases used to evaluate it, there is a risk of overfitting, just as with any model.
- Repeat the evaluation every time the tool, the model version or the instructions change.
Comparing against a baseline
Knowing that a tool is right 85% of the time is not enough to decide whether it is worth using: a point of comparison is needed. How often is a simpler method, such as a keyword-based rule, right? How often is a trained person right? What is the cost of each type of error?
The logic is the same as that of a hypothesis test: start from a null hypothesis ("the tool is no better than the simple alternative") and assess whether the evidence allows rejecting it. When several alternatives are compared at once, the reasoning behind ANOVA or, if its assumptions do not hold, nonparametric statistics techniques such as the bootstrap, makes it possible to do so with sound judgment.
The machine as evaluator
When the answers are long texts (summaries, explanations, reports), evaluating them by hand is costly. An increasingly common practice is to use another model as an evaluator, asking it to grade the answers according to defined criteria. This practice can be useful, but it introduces a new source of error: the evaluator also has biases (for example, a preference for longer answers or for a certain style). That is why the evaluator must also be evaluated, by comparing its grades with those given by people on a sample of cases.
Demanding traceability
Finally, a very effective way to verify is to require the tool to show where each statement comes from: the code that produced a number, the query that generated a table, the passage of the document that supports a summary. A traceable result can be reviewed; one that is not can only be believed or discarded.
Recap
Verifying what a machine says does not require new tools, but applying the ones this book has been presenting since the first edition: thinking in terms of samples and variability, measuring errors with a confusion matrix, comparing against a baseline with the logic of a hypothesis test. Statistics, which was born to reason under uncertainty, turns out to be the best ally for working with tools that produce it.
Further reading
- Why Language Models Hallucinate. Kalai, Nachum, Vempala and Zhang, 2025. The paper mentioned above on training and evaluation incentives.
- Survey of Hallucination in Natural Language Generation. Ji et al., 2022. An overview of the types of hallucination and the approaches to measuring them.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Zheng et al., 2023. An analysis of the scope and the biases of using a model as an evaluator.
- Your AI Product Needs Evals. Hamel Husain. A practical guide to building evaluations for products based on language models.
- AI Risk Management Framework. NIST. A reference framework for identifying, measuring and managing the risks of AI systems.