28. From insight to action

The last stretch

Concepts and definitions described data analysis as an activity that makes it possible to draw conclusions and translate them into recommendations or actions, and that, framed within an analytics process, supports decision making in organizations. Throughout the book, however, the focus was almost always on the first part: how to obtain, describe, infer and model. The last stretch, the one that goes from a conclusion to an action, lay outside the scope of analysis proper.

AI tools shorten that stretch considerably. A single system can detect a pattern, write a recommendation and, connected to the right tools, carry it out: send an alert, adjust a price, prioritize a list of cases. This chapter, which closes the new part of the book, is about what it takes for that step, from conclusion to action, to be well grounded.

Closing the loop

When the result of an analysis automatically becomes an action, the analysis stops being a report that someone reads and becomes part of a system that acts on the world. And systems that act have consequences that reports do not: an error in a report can be caught before anyone uses it; an error in an automated system repeats every time the system acts.

Connecting a model to the tools that let it act (a database, a management system, a messaging channel) is technically simple today, and there are even open protocols designed to standardize those connections. Precisely for that reason, the relevant question is no longer "can it be automated?" but "what evidence is needed to do it, and how will anyone know whether it keeps working?".

Deciding under uncertainty

Every conclusion drawn from data carries some degree of uncertainty, as has been seen from Data, units and variables of analysis onward. When moving to action, that uncertainty does not disappear: it becomes risk. Some ideas from the first edition are especially useful for managing it:

  • Intervals, not just points. A point estimate says what the most likely value is; a confidence interval says how far off it could be. An action that only pays off if the true value lies at the optimistic end of the interval is a bet, not a grounded decision.
  • The cutoff point is a decision. Logistic regression mentioned that, to go from a probability to a classification, 0.5 is usually used as the cutoff point, although other criteria can be set. When an action is automated, that criterion stops being a technical detail: it defines how many false positives and false negatives are accepted.
  • Errors do not cost the same. Failing to detect a fraud is not the same as blocking a legitimate customer by mistake. The appropriate cutoff point depends on the relative cost of each type of error, something that can only be defined with knowledge of the business.

Experimenting before automating

An analysis of historical data can show that a certain action is associated with better results, but that does not prove the action causes them. Perhaps the customers who received an offer bought more because they were already better customers, and that is why they were sent the offer. The soundest way to find out whether an action has the expected effect is to test it in a controlled way.

A controlled experiment (in the digital world it is usually called an A/B test) randomly assigns the units of analysis to different groups, applies the action to only one of them and compares the results. Random assignment is what makes it possible to attribute the difference to the action and not to other factors. The analysis of those results is exactly what was covered in Hypothesis tests and, if more than two variants are compared, in ANOVA, whose very formulation starts from experiments with different treatments.

It should be noted that experimenting has its own requirements: defining in advance what will be measured, calculating the sample size needed to detect a relevant difference, and not stopping the experiment as soon as the results look favorable. These are the same precautions as in classical inferential statistics, applied to everyday decisions.

Monitoring what was automated

A model is fitted with data from a given moment, but it is used in a world that changes: customers change their habits, new products appear, rules are modified. Over time, the data the model acts on may stop resembling the data it was built with, and its performance degrades with nothing to announce it.

That is why automating an action is not the end of the analysis, but the start of a follow-up task. It pays to monitor at least three things: the input data (do they still have distributions similar to the original ones?), the system's decisions (has the proportion of cases classified each way changed abruptly?) and the results (does the action still have the effect that was measured in the experiment?). The descriptive tools of the first edition, such as frequency tables, summary measures and charts over time, are enough for much of this follow-up.

People in the loop

Not all decisions should be automated to the same degree. A useful criterion is to consider two dimensions: how much harm an error can cause and how easy it is to reverse. A wrong content recommendation corrects itself on the next visit; a decision on a loan, a diagnosis or a penalty affects a person in a much more serious and lasting way.

Between full automation and a completely manual decision there are many intermediate points: the system can propose and a person approve, it can act only on the clear cases and refer the doubtful ones, or it can act within limits (of amount, of quantity, of scope) that a person sets. Choosing the right point for each decision is itself part of the work of analysis, and it builds on everything covered in Privacy, bias and responsibility.

Recap

The value of an analysis is realized when it turns into a good decision. AI tools make it possible to cover that last stretch at an unprecedented speed, and for that very reason they demand more rigor, not less: managing uncertainty instead of ignoring it, experimenting before automating, monitoring what was automated and reserving human judgment for what cannot be undone. It is, in the end, the same statistics as always, put to work for action.

Further reading