19. From copilot to colleague

From copilot to colleague

In February 2023, the first edition of this book described AI tools with a flight analogy: we were the pilots, and the assistant acted as copilot. It suggested the next line of code, answered questions, proposed corrections. The chapter closed with a warning: the responsibility for defining, thinking and anticipating errors still falls on human skill (at least for now). That text is preserved intact in the appendix Copilots and chatbots.

Since then, the copilot has stopped limiting itself to suggestions. Current tools can receive a goal, break it into steps, write and run code, read the results and try again when something fails, until they deliver a complete analysis with tables, charts and written conclusions. In the jargon they are often called agents. The flight analogy, then, falls short: rather than a copilot assisting in the cockpit, the tool resembles a colleague to whom an entire task can be assigned.

This chapter opens the new part of the book with a simple question: if a tool can cover a good part of the analysis process on its own, what changes in the work of the person who analyzes data? The short answer is that the form changes a great deal, but not the substance.

What an AI tool can do in an analysis today

It is worth reviewing the stages of the data analytics process mentioned in Concepts and definitions and identifying where these tools come in:

  • Framing the question. They can help turn a vague concern into concrete questions, suggest variables to consider or point out missing information.
  • Obtaining and wrangling the data. They write SQL queries and code to join, clean and transform tables, and explain what each step does. That is, much of what was covered in Data wrangling.
  • Exploratory analysis. They compute measures of location and dispersion, build frequency tables and propose visualizations, as in Describing variables and Data visualization.
  • Inferential analysis and models. They suggest which statistical test or which model to apply, implement it and report the results.
  • Communication. They write reports, executive summaries and presentations tailored to different audiences.

What is notable is not that each task is possible on its own (many already were, with libraries and templates), but that a single tool can chain them together from an instruction in natural language.

What still belongs to the analyst

However, several of the issues that the first edition pointed to as the foundation of all analysis are not solved with more computing power.

The definition of the unit of analysis and of the variables, for example, depends on the problem to be solved and on knowledge of the domain. A tool can propose a definition, but it does not know, unless told, what that organization means by "active customer" or which period makes sense to compare. The same happens with assumptions: knowing whether a sample is representative, whether a variable can be considered random or whether the groups being compared are really comparable requires knowing how the data were generated, something rarely written in the table.

The critical reading of the results also still belongs to the analyst. A conclusion can be correctly computed and still be irrelevant or misleading for the decision to be made. And, above all, responsibility still belongs to the analyst: whoever signs the report or recommends an action answers for it, regardless of which tool did the calculations. In that sense, the "at least for now" of the 2023 chapter can be read differently today: it is not a question of technical capability, but of who is accountable.

Uneven performance

One characteristic of these tools worth keeping firmly in mind is that their performance is uneven. They can handle with ease a task that would take a person hours and, in the same conversation, get something apparently trivial wrong, such as a count, a date or the name of a column. Moreover, they are wrong in the same confident tone in which they are right.

A practical consequence follows: it is not enough to know what a tool can do in general; one has to know how much to trust it for a particular task. That trust is not presumed; it is measured. The chapter Verifying what the machine says returns to this question with the tools of inferential statistics.

How to work with an artificial colleague

Some practices, closer to common sense than to a technique, help make good use of these tools without losing control of the analysis:

  • Giving context. The clearer the description of the problem, the data and the definitions, the better the results. A vague request produces a generic analysis.
  • Asking for the path, not only the answer. Requesting the code, the query or the reasoning that led to each result makes it possible to review it and, if needed, correct it.
  • Reviewing in parts. An error is easier to detect in an intermediate step (a filter, a join between tables) than in the final report.
  • Keeping the trail. The instructions, the generated code and the versions of the data are part of the analysis. The desirable properties mentioned in Working environment (transparency, replicability, interoperability and scalability) apply just as much, or more, when part of the work was done by a tool.
  • Distrusting the convenient. A result that confirms exactly what was expected deserves a second look, whether it was computed by a person or by a machine.

Recap

The arrival of tools capable of carrying out a complete analysis does not make the fundamentals obsolete: it makes them more necessary. To assign a task well, one has to know what it consists of; to review a result, one has to understand how it was obtained; to decide how much to trust it, one has to know how to measure uncertainty. The chapters that follow go, one by one, through the points where the practice of data analysis changes with these tools, always connected to the topics of the first edition.

Further reading

  • Building effective agents. Anthropic. A clear description of what is meant by an agent and of the common patterns for building agents.
  • AI Index. Stanford Institute for Human-Centered Artificial Intelligence. An annual report on the state of artificial intelligence.
  • People + AI Guidebook. Google PAIR. A guide to designing products in which people and AI systems work together.
  • Attention Is All You Need. Vaswani et al., 2017. The paper that introduced the Transformer architecture, already cited in the first edition.