A business factory spends in two currencies: the money each turn of a model costs and people's attention. This chapter gathers the practices that lower the first without lowering quality, and that protect the second, which is the more expensive one.
The model, by task
Not all work needs the most capable model. One practice that works well is to assign the model by task:
| Task | Model |
|---|---|
| searching, listing, taking inventories, reading a lot | a small, cheap one |
| implementing, reviewing with narrow criteria, researching | a mid-range one |
| designing, judging, reviewing what is sensitive (access, secrets, real data, anything irreversible) | the most capable one |
And one rule makes the scheme safe: what a cheap model writes is reviewed by a capable one before it is accepted, and every serious finding from a cheap reviewer goes through a capable one that confirms or dismisses it. A small model never edits what matters; a mid-range one that could not verify something central, or that strayed from its instruction, is run again with the most capable one.
Elephants and goldfish
One image helps in thinking about quality in a system of agents. The elephant is the main conversation: it remembers everything, knows the context, knows why each decision was made. The goldfish is the reviewer: it starts with no memory and no premises, and reads the work as someone arriving from outside would.
Each one sees what the other does not. The elephant understands, but carries its assumptions along; the goldfish does not understand the why, but shares none of the assumptions. Quality comes from crossing the two. That is why important reviews are done by new subagents (chapter 4), and why a reviewer is given the work, not the hypothesis: the code, the text or the plan, and not the conclusion of whoever made it. A reviewer who is told what to look for tends to come back with it confirmed.
Reviews sized to the risk
Reviewing everything with the same intensity is expensive and, moreover, does not improve quality: it spreads attention over what does not need it. The practice is to size the review to the risk (chapter 12):
- a small change or a fix is reviewed from two or three angles, chosen for what could break;
- a new design, or the first thing that writes to real data, gets the full set of reviews;
- later rounds review only what changed;
- there is a maximum number of rounds; if a serious problem remains after it, it is taken to the person who decides, with the problem written down, instead of one more round.
Each finding is written in one line: where, how serious, what the problem is and what the fix is. No prose around it.
«Done» with evidence, and validating before believing
An agent that says «done» with confidence can be wrong with the same confidence. Two rules hold it back:
- every «done» comes with its evidence: a link, an identifier, a screenshot, the output of a check;
- before trusting something, it is validated against reality: the change is tested in the real system (in the test environment), not only in what the agent says it did.
One particular case deserves its own rule: testing happens in one environment, and exactly what was tested is what goes to production, not a similar version put together afterward.
Context costs too
Each turn of a model charges for everything it reads. A conversation that drags along hours of history pays for that history in every response. For that reason:
- what is always loaded is kept short (chapter 5);
- long work is given to a new conversation with a written, targeted request, or to one that starts from a summary made for the purpose, and not to a copy that inherits the entire context of whoever opens it;
- internal reports are brief: first the result, then where it is and the next step, and at the end a line with what was not verified.
Subtracting before adding
The cheapest way to do something is not to do it. Before building, it is worth going through three questions, in order: Does it need to exist? Does it already exist somewhere? Would one line be enough? Only if all three say no is it built. It is a well-understood form of laziness, and it avoids most of what would later have to be maintained.
Thinking before passing along
A role that receives something and forwards it without thinking multiplies the cost: each hop adds a turn and none adds judgment. Before passing a request along, it is understood, whatever can be answered is answered, and only what truly belongs to someone else is passed on.
And one rule without exceptions: an approval is never «laundered» through another conversation. If a conversation was denied an action, it does not ask another to do it in its place; it takes it to whoever decides.
The measure
All of the above is judged by a single measure: a person's total attention for each task finished right, counting the request, the questions, the approvals, the reviews and whatever had to be redone (chapter 11). A practice that saves money but asks a person for more attention generally does not save anything.
To recap, the model is chosen by task and what is cheap is reviewed by what is capable, quality comes from crossing elephants and goldfish, reviews are sized to the risk, every «done» comes with evidence, context is treated as a cost, and before building, one subtracts. The next chapter looks at the typical failures these practices help stop.