Many agents, many copies
An analysis agent is an AI tool that carries out a stretch of the analysis on its own: it queries data, writes and runs code, loads results and drafts conclusions (see From copilot to colleague). When several of them work at once on the same sources, a problem appears that teams of people know well and that agents make far more frequent: each one has its own copy of the data, and the copies drift apart.
This chapter is about how to organize the data of a team of agents. The proposal has two pieces: a local warehouse per agent, to work fast without stepping on anyone, and a central team warehouse, which records who loaded what, keeps the history of each table and defines how conflicts are resolved. What follows is the technical side; the organizational side (what role each agent plays, who approves what and how they are directed) is developed in the book Business factories, by the same author.
A local warehouse for each agent
Working environment presented DuckDB as a way to run SQL on files without installing a server. That same property makes it suitable for an agent: it is an embedded analytical database (it runs inside the same process as the query), columnar and with no network in between.
On top of DuckDB, DuckLake adds what a folder of loose files lacks. It is an open table format (MIT-licensed) that stores the data in Parquet files and the metadata (which tables exist, which files make them up, what changed in each version) in a separate database called the catalog. Each load is recorded as a version or snapshot, which can carry an author and a message, just like a commit, and any earlier version can be queried with AT (VERSION => n). If, in addition, each table and each column carries a comment, a new agent can discover the tables by reading the catalog.
The limit of a local warehouse is obvious: what is in one agent's folder is invisible to every other agent, and in many environments it disappears when the session that created it ends.
A central warehouse for the team
The second piece is a shared warehouse, which in the DuckDB ecosystem can be built with DuckLake on a catalog in Postgres and the data in object storage. Its job is to answer three questions:
- Who loaded what. Each change that reaches the central warehouse is a snapshot with its author and its message. It is worth noting that in DuckLake the author is a text string written by whoever makes the change, and nothing authenticates it; that is why it is better for the service that receives the changes to stamp it, based on the agent's identity, rather than for the agent to declare it.
- What the history is. Snapshots make it possible to see a table as it was at any point in time.
- How conflicts are resolved. When two agents change the same thing, the central warehouse decides what happens, according to rules set in advance.
The key point is that agents do not write directly to the central warehouse. They work on their local copy, and a synchronization process carries changes back and forth. The analogy with git is direct: it is the same problem that software development solved for code. However, git does not work for the data itself. The catalog is a binary file, and when two agents load rows at the same time and an attempt is made to merge their versions, git responds Cannot merge binary files and leaves one of the two out. Tables need to be merged row by row, not file by file.
Clone, push and pull
The three git operations have their equivalent for tables:
- Clone. The agent receives the schema and the rows of the central warehouse as they are at a given snapshot, with their columns and comments, and records that number as its base.
- Push. The agent sends only what changed since its base. For this, DuckLake offers a change feed between snapshots (the
ducklake_table_changesfunction), which returns the inserted rows, the deleted rows and the before and after image of each modified row. The central warehouse applies everything in a single transaction, so that each push is a central snapshot with the agent as its author. - Pull. The agent receives what others changed since its base and moves its base forward.
It is better for synchronization to be done by the environment at fixed moments (when a session opens, when each task ends) and not by the agent whenever it sees fit: that way nothing is left unpushed by oversight, and the agent does not need the central warehouse's credentials.
How conflicts are resolved
A conflict occurs when an agent tries to push a change to data that another agent has already modified. Four rules are enough to handle it in an orderly way.
One key per table. Data, units and variables of analysis defined a data point as the intersection of a unit of analysis and a variable. The key is what identifies the unit: the id of a customer, of an order, of a campaign. Without a declared key there is no way to know whether two rows coming from different agents refer to the same thing, and therefore no conflict rule is possible. It is the same requirement as for a join between tables (see Data wrangling).
One owner per table. Each table has a single agent, or a single group of agents with the same function, that can write to it; the others read it. If two functions need to record data about the same entity, those are two tables with the same key, joined at analysis time. The central warehouse rejects any write from a non-owner, and conflicts between different functions disappear by construction.
Reject by key and rebase. Within a single owner, several agents can work at the same time. A push is rejected when another agent changed any of its keys in the central warehouse after the base; keys that do not overlap go through without trouble. For example, an agent marks a lead as lost based on its copy, unaware that another agent has just marked it as qualified. The central warehouse rejects the change and reports which key conflicted and in which snapshot it changed; the first agent pulls the new version, sees the current state, decides and pushes again.
Set aside, never overwrite. The rejected change is not lost: it is stored separately, with an identifier, so that the agent (or a person) can review it. The usual alternative in synchronization tools, last write wins, would have silently erased the other agent's qualified, and nobody would have found out. Append-only tables (events, measurements, costs) do not run into conflicts, so it is advisable to model everything that allows it that way.
The catalog defines how many can write
In DuckLake, the type of catalog determines how many processes can write at the same time:
- Catalog in a DuckDB file. Only one process can attach it at a time (unless all of them open it in read-only mode). If two agents try to load data into the same local warehouse at once, the second one fails immediately with a lock error (
Could not set lock on file … Conflicting lock), and not even a reader can attach it while the other one is writing. - Catalog in SQLite. SQLite supports a single writer at a time, but DuckLake retries lock conflicts, which in practice lets several local processes write interleaved. In a test with two processes that simultaneously loaded 50 batches of 20 rows each, the table ended up with all 2,000 rows, with no losses.
- Catalog in Postgres. This one is designed for many users, including remote ones, and that is why it is the one for the central warehouse.
Hence a simple rule: a DuckDB file for a single process, SQLite when several processes share a local folder, and Postgres for the central warehouse. It is worth noting that DuckLake resolves conflicts between simultaneous writes to the same catalog (it retries if the changes are compatible and aborts if they are not), but that is not the same as reconciling out-of-date local copies: as of this writing it offers no clone, push or pull, no branches and no merge, and that thin layer has to be built. Dolt does offer the full git model, with cell-by-cell merge, although it is a row-oriented database that its own authors do not recommend for large-scale analytics.
Recap
A team of agents needs the same things as a team of people sharing data, only more often. Each agent works in its local warehouse, the team maintains a central warehouse, and changes are cloned, pushed and pulled between them with an author and a message. Conflicts are handled with one key per table, one owner per table and a rule that rejects, sets aside and asks for a decision, instead of overwriting. DuckDB and DuckLake provide the engine and the history; synchronization, in the end, is still a piece the team must own.
Further reading
- Choosing a Catalog Database. DuckLake documentation. What each catalog (DuckDB, SQLite, Postgres) supports in terms of concurrent processes and writers.
- Conflict Resolution. DuckLake documentation. How transactions that conflict on the same catalog are resolved.
- DuckLake 1.0: The Lakehouse Format Built on SQL Reaches Production-Readiness. DuckLake team, 2026. What the first stable release includes and what is left for later.
- DuckLake Extension. DuckDB documentation. Snapshots, commit messages and queries on earlier versions.
- Conflicts. Dolt documentation. An example of cell-by-cell conflict detection in a database versioned like git.