Validating is putting to the test
Validating an agent's work means testing what it delivers against checks defined in advance, before using it. A loaded table, a fitted model, a report with figures: none of these becomes an input for a decision until it passes a set of tests that someone set before seeing the result.
Hypothesis tests described putting what is known to the test, using evidence, as the way science works. Validating an agent's work is the same operation applied to a smaller case: one starts from a claim (the result is correct) and looks for evidence that allows rejecting it. What changes is the starting point. An agent's work is not believed until it is validated, and not the other way around. This chapter presents what research measures about the effect of validating, how to read those figures with the tools of this book and where it makes sense to place the tests.
What research measures
Two recent papers provide concrete figures, and they are worth knowing in context.
The first is AutoKaggle (Li et al., 2024), a multi-agent system that solves Kaggle competitions with tabular data: cleaning, feature engineering, modeling and submission of predictions. Among its tools it includes unit tests that run on the code and the data of each stage (missing values, outliers, duplicates and other consistency problems). When those tests were removed and everything else was left the same, the average completion rate fell from 0.85 to 0.10, that is, from 85% to 10% of runs. The authors sum it up by saying that, without unit tests, completing the tasks becomes almost impossible.
The second is a scaling study by Google Research, Google DeepMind and MIT (Kim et al., 2025), which compared 260 configurations of agent systems across six task sets, with five architectures and three model families. One of its measures is an error amplification factor: how much errors multiply when going from a single agent (factor 1) to several. With independent agents, which work in parallel and whose outputs are combined without any cross-validation, the factor was 17.2. With a centralized architecture, where an orchestrator receives the others' work and verifies it before combining it, it was 4.4. It is worth noting that this measure is taken on execution traces; the paper itself reports another one, on the error in the final result of the task, where the differences between architectures are much smaller (between 1.1 and 1.3 times). The direction is the same; the magnitude depends on what is measured.
Reading those figures with a statistician's eye
The figures in a research paper are estimates too, and they deserve the same treatment as any other (see Point and interval estimation).
The AutoKaggle comparison was made on 4 tasks. If, as in the rest of the paper's experiments, each task was repeated 5 times, that makes 20 runs per condition: 17 complete with tests and 2 without them. With so few observations, a 95% confidence interval (computed with the Wilson method, which is better suited than the normal approximation when the sample is small or the proportion is close to 0 or 1) runs roughly from 0.64 to 0.95 with tests and from 0.03 to 0.30 without them. The intervals are wide, but they do not overlap: the effect is large enough to show even with a small sample. What the sample does not guarantee is generality. The 4 tasks were chosen by the authors and are not a random sample of all possible data tasks, so what was covered in Sampling techniques about representativeness applies unchanged.
The same reasoning holds for one's own measurements. Saying that an agent "got it right 9 out of 10 times" is not the same as saying it is right 90% of the time: with 10 cases, the Wilson interval runs roughly from 0.60 to 0.98. A success rate without its interval is a small sample presented as a parameter.
A validation is a test
The analogy with a hypothesis test is not just a figure of speech. A check on an agent's work has the same two possible errors:
- Rejecting correct work (the equivalent of a Type I error). Its cost is usually rework: the agent redoes something that was fine.
- Accepting wrong work (the equivalent of a Type II error). Its cost is a decision made on false data, and it is usually much higher.
A test that only checks that the file exists almost never rejects anything, that is, it has very little power. Tests with power are the ones that could fail if the work were wrong. Some worth always having:
- Structure. Expected columns and types, a unique key, no missing values where they do not belong.
- Control counts and totals. The rows read must equal the rows written plus the rows discarded, and a total in the table must match the one in the source.
- Business rules. Non-negative amounts, dates within the period, statuses within a closed list.
- Independent recomputation. Rerunning the queries cited in the report on the same version of the data, using the format proposed in Every number with its source.
- Cases with a known answer. An evaluation set like the one described in Verifying what the machine says.
There is also the problem of multiple comparisons. An agent can run dozens of statistical tests in seconds and return only the ones that came out significant; at a 5% level, some will be significant by chance. Validation must require the agent to report all the tests it ran, not just the ones that support the conclusion.
Where to put the verifier
The scaling study suggests something about how tests are organized, and not just about their content: errors are better contained with a central point that verifies before combining than with agents that check one another, or that do not check at all. In the terms of Agents that analyze, that point is a natural one: the central warehouse receives each change from each agent and can validate it at the door (types, declared key, the table's rules, a cap on deletions per push) before accepting it, and send it back with the reason when it does not pass.
Two caveats. The first is that the verifier must also be evaluated: if it is another model, it has its own biases, and if it is a set of rules, it can miss precisely the error that matters. The second is that not everything can be validated by a program. Which tests to require for each table, what to do when a piece of work fails several times and who authorizes an exception are decisions of organization, which are developed in the book Business factories, by the same author.
Estimating the error, not just approving or rejecting
When the volume is large (thousands of records classified by an agent, for example), reviewing everything by hand is not possible, and approving without reviewing is not acceptable. The way out is the usual one: review a random sample and estimate the error rate with its interval. If the upper end of the interval exceeds the error one is willing to tolerate, the batch does not pass, even if the point estimate looks good. It is the logic of the acceptance sampling that industrial quality control has used for decades, applied to a process now carried out by a machine.
Recap
The measured evidence is consistent: without tests, a system of agents working with data completed its tasks in 10% of runs instead of 85%, and independent agents without a central verifier multiplied their errors 17.2 times, compared with 4.4 with one. Read with the tools of this book, those figures are estimates with their uncertainty, and the same view applies to the success rate of any agent. Validating is running a test: with a hypothesis set before looking, with tests that have power, with every comparison reported and with an interval instead of a single number. Statistics is, once again and in the end, the tool for deciding how much to believe.
Further reading
- AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions. Li, Zang, Ma et al., 2024. The paper whose comparison with and without unit tests is cited in this chapter (table 3).
- Towards a Science of Scaling Agent Systems. Kim, Gu, Park et al. (Google Research, Google DeepMind and MIT), 2025. The scaling study and its error amplification factor by architecture.
- Why Do Multi-Agent LLM Systems Fail?. Cemri, Pan, Yang et al., 2025. A classification of the failures of multi-agent systems, including weak task verification.
- Add data tests to your DAG. dbt documentation. How to declare structure and rule tests on the tables of a warehouse.
- Improving Your Statistical Inferences. Daniël Lakens. An open book on power, effect sizes and multiple comparisons.