Beyond the cell
Data, units and variables of analysis proposed thinking of a data point as anything that can be placed inside a cell: a number, a letter, a word, a true or false. That definition is still useful, but it leaves out a vast amount of information that organizations generate every day and that does not fit comfortably in a spreadsheet: the comments in a survey, complaint emails, the photos from a field survey, the recordings of a call center.
For a long time, analyzing that kind of information meant reading it, looking at it or listening to it, and coding it by hand, which limited the analysis to small samples. AI tools changed that situation radically: today it is possible to turn text, images and audio into variables at a scale that used to be impractical. This chapter is about how to do it with judgment.
The model as a measuring instrument
The central idea is simple: an AI model can work as a measuring instrument. Just as a scale turns an object into a number of kilograms, a model can turn a comment into a category ("complaint about a delay", "price inquiry"), a rating (positive, neutral, negative), a list of the topics mentioned or an amount extracted from the text.
The result of that process is, once again, a data matrix: each row is a unit of analysis (a comment, a photo, a call) and each column is a variable obtained from it. From there, everything covered in the first edition applies unchanged: frequency tables, descriptive measures, visualizations, tests and models.
But like every measuring instrument, the model has error. And the quality of the analysis that follows depends on knowing that error.
Coding with judgment
Before asking a tool to classify thousands of texts, it pays to do the work that the social sciences call building a codebook: defining precisely which categories exist, what each one means and what to do with doubtful cases. The condition required of the classes of a frequency table holds here too: the categories must be mutually exclusive (each case falls into only one, or it is stated explicitly that a case can fall into several). And it pays to add another: that they be exhaustive, that is, that every case has a place, even if it is "other".
A reasonable procedure is the following:
- Define the categories and write down their criteria, with examples.
- Code a sample of cases by hand, ideally by more than one person, to detect ambiguities in the definitions.
- Apply the model to that same sample and compare its results with the manual coding, using a confusion matrix.
- Adjust the definitions or the instructions and repeat until the level of agreement is acceptable for the intended use.
- Only then, apply the model to the full set.
This procedure is not a formality. There is evidence that, for certain text annotation tasks, conversational models can match or outperform people hired on crowdwork platforms; but that depends on the task, the definitions and the data, and it can only be known by measuring in the specific case. What was developed in Verifying what the machine says applies here directly.
Representing meaning with numbers
Besides classifying, there is another way to turn texts (and images) into variables: embeddings, or vector representations. An embedding assigns each text a list of numbers, in such a way that texts with similar meanings are represented by similar lists. There is no need to define categories in advance: similarity emerges from the representation.
With that representation it is possible, for example, to find the comments most similar to a given one, to group automatically the texts that talk about the same thing, or to use those numeric columns as input variables for a model. The idea of "closeness" between observations already appeared in KNN and CART: the k nearest neighbors algorithm classifies a case according to the cases closest to it. Embeddings make it possible to apply that same logic to objects that previously had no natural notion of distance.
One limitation is worth pointing out: the dimensions of an embedding do not, in general, have a direct interpretation. They are useful for measuring similarity and feeding models, but they do not replace an explicitly defined variable when what is needed is to explain a result.
Images and audio
The same principles extend to other formats. In the case of audio, automatic transcription models turn a recording into text, which can then be coded as shown above. In the case of images, there are models that can describe them, classify them or relate them to descriptions in natural language, which makes it possible, for example, to label the photos from a field survey according to what they show.
In both cases it is worth remembering that each conversion step adds a layer of error: a transcription with errors produces a coding with errors, even if the classification model works perfectly. Reviewing a sample of each intermediate stage is the way to know where quality is lost.
Measurement error does not go away
Suppose a model classifies comments as "positive" and "negative" and that, on a hand-coded sample, it is observed to tend to confuse ironic comments with positive ones. If the proportion of positive comments over the total is then calculated, that proportion will be biased upward. Measurement error is not noise that cancels out; it can have a direction.
That is why, when reporting results obtained this way, it is good practice to also report the quality of the measurement: how it was validated, on how many cases and with what level of agreement. Just as an estimate without a confidence interval says little, a proportion calculated on automatic classifications with no information about their accuracy should be read with caution.
Recap
The possibility of turning text, images and audio into variables greatly expands what can be analyzed. But the principle is the same as always: a data point is the result of a measurement, and every measurement has an error worth knowing. Defining the categories well, validating against a carefully coded sample and reporting the quality of the measurement are the conditions for that expansion to also be an improvement.
Further reading
- The Illustrated Word2vec. Jay Alammar. A visual introduction to the idea of representing words as vectors.
- ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks. Gilardi, Alizadeh and Kubli, 2023. The study on text annotation mentioned in this chapter.
- Sentence Transformers. Documentation for a widely used Python library for obtaining text embeddings.
- Robust Speech Recognition via Large-Scale Weak Supervision. Radford et al., 2022. The paper that introduced Whisper, an open audio transcription model.
- Learning Transferable Visual Models From Natural Language Supervision. Radford et al., 2021. The paper that introduced CLIP, a model that relates images and text.
- Text as Data: A New Framework for Machine Learning and the Social Sciences. Grimmer, Roberts and Stewart, Princeton University Press, 2022. A rigorous treatment of text as a data source in the social sciences.