2. Working environment

Summary

One of the most common points of friction for anyone starting out in data analysis is the overwhelming number of tools and working environments available. Even once a choice is made, setting them up is rarely simple and the learning curve is slow, particularly for people without a background in programming. However, together with a clear understanding of theory and practice, using the tools well is a key factor in professional success.

So before starting on the tasks proper to a data analyst, it is necessary to survey the landscape of leading platforms, for each of the different concerns involved in sound development and work.

Starting with spreadsheets, these are the tools with the longest track record in the field. Although today there are countless complex tasks and volumes of data they cannot handle, they tend to appear in the final stretch of a project (making results available to end users), both because of their simplicity and because those same users are accustomed to them.

Then there are notebooks, generally associated with Jupyter, which are essentially documents that can hold text and code (Python, R, etc.), with a cell-by-cell execution model, which makes them the tool of choice for data scientists. Their use has since spread, and practically any role involved with data understands how they work and what they offer, to the point that they are considered something of a standard. Through different kernels, notebooks support several languages, although they are most often associated with Python. For those who program in R, Posit (formerly RStudio) is more comfortable, although it is now becoming a platform that supports more languages and can be an excellent choice as a working environment.

There are also other tools that copy the notebook model and/or extend it with new features. The range of options is quite wide, with Colab standing out for being free and offering generous processing capacity, and JupyterLab, an IDE that handles notebook management, plugin installation and more. JupyterLab, in fact, has been adopted by several companies as the base for much more powerful cloud services such as GCP, AWS and Azure, forming data analytics suites that today's companies and organizations keep adopting.

To all of this must be added the tools designed exclusively for data visualization, which are now beginning to integrate querying capabilities (once attributable only to a database client) and data transformation (beyond ETL tools as such) within the same environment.

As may already be clear, this first look has brought quite a few new things to investigate and understand. What matters, in the end, is having enough flexibility to adopt, along the way, whatever tools are available at work or in particular projects. It should not be overlooked that there will be certain features common to all of them, as well as certain desirable properties (not only for the tools, but for any data analysis itself): transparency, replicability, interoperability and scalability.

With that in mind, another set of important tools comes into play. The main ones are Git for transparency (versioning and experimentation) and Docker / Anaconda for replicability (defining and managing dependencies). A piece of work that cannot be replicated or properly used by other collaborators, or even by end users, is worth nothing: in the era of collaborative work these tools are practically mandatory.

Update 2026 Reproducible environments

Since the first edition, the usual way of guaranteeing replicability has shifted. The first resource is no longer usually a container or a full package distribution, but a lockfile: an automatically generated record of the exact version of every library the project uses (including the dependencies of its dependencies), which makes it possible to rebuild the same environment on another machine with a single command. In Python, tools such as uv (which works from the pyproject.toml file) or pixi (built on the community packages of conda-forge) fill that role; in R, renv does.

In practice, reproducibility is best thought of as a ladder, where each rung is added only if needed:

  • Project file and lockfile, versioned with Git alongside the code. This is the minimum rung and is enough for most analyses.
  • Container (Docker, or dev containers, which usually rely on it under the hood) when the operating system and its libraries also need to be pinned, or when the analysis is deployed as a service. Docker is still valuable, but as a second step rather than a condition of entry.
  • Identified input data: the cutoff date, version or hash of each file or table. The same code run on different data produces different numbers, and no locked environment prevents that. Tools such as DVC, mentioned above, handle this part.

There is also an issue that is not technical. Since 2020, Anaconda's terms of service have required a paid license to use its package repositories (the defaults channel) in for-profit organizations with more than 200 people. conda as a tool and the community channel conda-forge remain free to use. Recommending "Anaconda" without qualification can therefore create a licensing problem in a company; it is worth knowing which channel is in use.

The criterion for knowing whether an analysis is reproducible is simple and verifiable: another person clones the repository, runs one command and gets the same results. The cheapest test is to do it oneself in a clean environment, or to automate it so that it runs with every change.

Finally, AI assistants (see From copilot to colleague) readily write code that imports libraries, and may assume versions or functions different from those installed. A declared, locked environment makes those discrepancies surface explicitly at run time, rather than as a silently different result; it also gives the assistant precise context about which versions the project uses.

Update 2026 From the notebook to the team's data flow

Notebooks remain the tool of choice for exploring: trying out ideas, looking at data, iterating on a chart. What has changed is the place they hold in a team. As the definitive record of an analysis they have well-known limitations: hidden state (cells can be run out of order, so the visible result may not match the visible code), outputs embedded in the file, which make diffs between versions in Git nearly unreadable, and the difficulty of testing and reusing what they contain.

In many organizations, the source of truth for analysis has moved to a data warehouse or a cloud lakehouse, with transformations written in SQL and versioned in Git. The usual pattern is to load the raw data and transform it inside the warehouse (known as ELT), through SQL models with tests and documentation; dbt is the most widespread tool for that task. Data is increasingly stored in open columnar formats, such as Parquet, and in table formats such as Apache Iceberg, which different engines can read without copying it. For work on a personal computer, DuckDB runs SQL directly on Parquet or CSV files, without installing a server.

The path data takes within a team usually looks like this:

  • Ingestion from the sources (transactional systems, events, files, APIs).
  • Storage in the warehouse or lakehouse.
  • Transformation in versioned, tested SQL (see the update in Data wrangling).
  • Semantic layer and BI tools, where metrics are defined only once (see Asking the data in natural language).
  • Notebook or report, to explore, model and communicate on data that can already be trusted.

Within that pattern, a few practices greatly improve the use of notebooks: restarting and running end to end before sharing; moving reused logic into modules or SQL models; and considering text-based formats that version well, such as notebooks paired with scripts through Jupytext, marimo notebooks (saved as Python files) or Quarto documents.

The properties listed in this chapter (transparency, replicability, interoperability and scalability) are precisely what this pattern pursues. And AI assistants, which today write a good share of SQL queries (see From copilot to colleague), perform better when tables and metrics are defined, versioned and documented in a single place.

Further reading

Spreadsheets

Jupyter Notebooks and similar platforms

Data visualization and querying tools

Data analytics suites

Versioning

Environment managers

Containerization

Update 2026