The questions that are not technical
The previous chapters dealt with how to use AI tools to analyze data: how to assign them tasks, how to query in natural language, how to verify their results, how to turn texts and images into variables and how to generate synthetic data. All of them assume a prior question that is not technical: should it be done? Can these data be used for this purpose? Can they be sent to this tool? Whom can the result harm?
These questions existed before AI, but they became more urgent for two reasons. The first is scale: what used to take weeks of manual work is now done in minutes, on millions of records. The second is distance: many tools process the data outside the organization, on third-party servers, and their decisions can be hard to explain. This chapter goes through three axes, privacy, bias and responsibility, with the same applied outlook as the rest of the book.
Personal data
A large share of the data analyzed in organizations refers to people: customers, employees, patients, students. In Argentina, its processing is regulated by Law 25,326 on the Protection of Personal Data, which sets out, among others, principles of data quality (that the data be adequate and not excessive in relation to the purpose for which they were obtained), of consent, of informing data subjects, of security and of confidentiality, and which gives special treatment to sensitive data. Other jurisdictions have their own rules, and it pays to know the ones that apply in each case.
From the point of view of analysis, some concrete practices follow from these principles:
- Minimize. Use only the variables needed to answer the question. If the analysis is by region, the exact address is not needed; if it is by age group, the date of birth is not needed.
- Respect the purpose. That a data point is available does not mean it can be used for any purpose. Data collected to manage one service cannot necessarily be reused for another end.
- Aggregate when possible. Many questions can be answered with frequency tables or summary measures, without working with individual records. It should be noted, however, that very small groups may make it possible to identify the people in them.
- Pseudonymize with awareness of its limits. Replacing names with codes reduces the risk but does not eliminate it: the combination of several apparently harmless variables (age, town, occupation) can be enough to identify someone.
Sending data to an external tool
When an AI tool hosted by a third party is used, the data sent to it leave the organization. Before doing so, it pays to answer a few questions: who processes the data and where? are they retained, and for how long? can they be used to train other models? what do the contractual terms and the organization's policies say?
In many cases, the appropriate answer is not to send the original data, but a version that does not expose them: aggregated or pseudonymized data, or a synthetic sample with the same structure (with the precautions noted in Synthetic data and simulation). Another alternative is to ask the tool for the code to perform the analysis and run it locally, so that the data never leave the controlled environment.
Bias starts in the sample
Sampling techniques and Data sources and collection stressed the importance of the sample being representative of the population. That requirement takes on a new dimension with models: a model learns from the data it was trained on, and it reproduces their imbalances. If certain groups are underrepresented in the data, the model will tend to work worse for them; if the data reflect past decisions that were biased, the model will learn to repeat them.
A frequently cited case is a 2018 study that evaluated commercial gender classification systems based on photos of faces and found that errors were much more frequent for darker-skinned women than for lighter-skinned men. None of those systems had been designed to discriminate; the problem lay in the data and in how their performance had been evaluated.
Measuring bias
The practical lesson of that case is that an overall indicator can hide important differences between groups. An accuracy of 95% over the total can be the average of 99% in one group and 70% in another. That is why, when evaluating a model that affects people, it pays to calculate the confusion matrix separately for each relevant group and compare the error rates.
It should be noted that there is no single definition of a "fair model". It can be required that the proportion of positive cases be equal across groups, that the false positive rates be equal, or that the precision be equal, and in general it is not possible to meet all those conditions at once. Choosing which to prioritize is a decision that depends on the context and on the consequences of each type of error, and it cannot be delegated to the algorithm.
Documenting
A practice that helps a great deal is to document data and models systematically. Specific formats have been proposed for this: sheets that describe how a dataset was collected, what it contains, what it was intended for and what it should not be used for; and cards that describe a model, its intended use, how it was evaluated and how it performs across different groups. Beyond the format, the goal is that anyone who uses those data or that model knows their limitations without having to rediscover them.
Responsibility is not delegated
Finally, the axis that runs through the other two: decisions based on an analysis have people responsible for them, and those responsible are people. That a result was produced by an AI tool does not transfer the responsibility to the tool.
Regulatory and reference frameworks point in that direction. In 2024 the European Union approved a regulation on artificial intelligence that classifies systems by their level of risk and sets stricter obligations for high-risk ones. The US NIST published a voluntary framework for managing AI risks, and UNESCO adopted a recommendation on the ethics of artificial intelligence. Whichever applies in each case, they all agree on a few points: know the risks, document, supervise and be able to explain.
Recap
Privacy, bias and responsibility are not side topics dealt with at the end of an analysis, but conditions defined at the start: which data are used, how they were obtained, whom they represent, where they are sent and who answers for the conclusions. The same statistical tools this book presents for analyzing data also serve to detect when an analysis may harm someone; using them for that purpose is part of the craft.
Further reading
Regulations and reference frameworks
- Ley 25.326 de Protección de los Datos Personales (in Spanish). Argentina.gob.ar.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act). EUR-Lex.
- AI Risk Management Framework. NIST.
- Recomendación sobre la ética de la inteligencia artificial (in Spanish). UNESCO.
Bias and documentation
- Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Buolamwini and Gebru, 2018. The study mentioned in this chapter.
- Fairness and Machine Learning: Limitations and Opportunities. Barocas, Hardt and Narayanan. An open book on the definitions of fairness and the tensions between them.
- Datasheets for Datasets. Gebru et al., 2018. A proposal of sheets for documenting datasets.
- Model Cards for Model Reporting. Mitchell et al., 2018. A proposal of cards for documenting models.