9. Data wrangling

Summary

Data wrangling involves joining, transforming, aggregating and/or filtering data, not necessarily in that order. A common saying holds that 80% of a data analyst's time goes to tasks related to this, and it may well not be a bad estimate.

Detecting missing values (which will not always mean the same thing), outliers (which will not necessarily always be outliers) and duplicate records (which will not necessarily always be wrong to have duplicated) is only the first step in a series of tasks that are complex both technically and conceptually, and that do take a great deal of time. Added to this is the need to integrate different data sources, performing join and concatenation operations between datasets (or tables in a database), with filtering and/or aggregation of the data before or after, not only to summarize the information or to work with different units of analysis, but perhaps also for the creation of new variables and/or the transformation of existing ones (known as feature engineering, which deserves a whole chapter of its own).

The key point is that the process itself (every operation performed) must be known and understood by the data analyst, because the results of the analysis, especially when working with inference, depend heavily on what was done, including of course the origin of the data itself.

In the best case, within companies and organizations with well-established analytics processes, the role of the data engineer tackles these aspects and makes the information available as needed. However, even when this holds, some degree of data wrangling will inevitably be required.

Update 2026 Data contracts and data tests

The 80% saying has no verifiable source, and what matters is not the exact proportion but that this work determines the validity of everything that follows. What has changed since the first edition is where it is done and how it is controlled. In teams of some maturity, much of the cleaning and joining of tables has moved to versioned SQL transformations, owned by a role between engineering and analysis (analytics engineering). And AI assistants now write a significant share of transformation code (see From copilot to colleague), so the bottleneck has shifted from writing the transformation to verifying that it is correct.

Data tests

A data test is an assertion about a table that is checked automatically every time the data are updated. The most common are:

  • Uniqueness of the key (each order appears only once).
  • Absence of nulls where there should be none.
  • Accepted values for categorical variables (the possible statuses of an order).
  • Referential integrity: every value of a foreign key exists in the reference table.
  • Ranges and freshness: non-negative amounts, plausible dates, data updated as of the expected date.

dbt ships the first four as built-in generic tests; in Python, libraries such as pandera or Great Expectations allow declaring similar expectations on a DataFrame. An example tied directly to this chapter: a join between tables with duplicate keys multiplies rows without raising any error, and a uniqueness test after the join catches it before total sales show up inflated in a report.

Data contracts

A data contract is an explicit agreement between whoever produces a dataset and whoever consumes it, covering its schema (columns and types), the meaning of each field and the quality expectations. Its value is that a change that breaks the agreement (a renamed column, a changed type) fails on the producer's side, instead of surfacing weeks later as a strange number in an analysis. There are concrete implementations, such as dbt model contracts, and open specifications, such as the Open Data Contract Standard.

The tools

pandas remains the reference and the basis of this chapter's readings; Polars and DuckDB are alternatives when the volume grows. The checks described apply to any of them. It should be added that an AI assistant can propose tests from a profile of the data, but what counts as a valid value is a domain decision, not the tool's. Tests are, in the end, the way to check generated code against explicit expectations instead of reviewing it line by line (see Verifying what the machine says).

Further reading

Update 2026