7 Experiments
Experiments are the backbone of causal inference, and text analysis is no exception. Whether in a laboratory or on Amazon’s Mechanical Turk, experiments can be carefully controlled and are a good way to mitigate the effects of confounding variables. Though many people associate advanced natural language processing with “big data,” the methods discussed in this book can be used effectively even in small-scale laboratory experiments.
An example of using experiments in quantitative language research: Sap et al. (2020) had online participants write either true stories that happened to them recently, or fictional stories about the same topic. They then used a large language model, GPT, to measure two likelihoods for each sentence in the story: the likelihood of the sentence given the previous sentence, and the likelihood of the sentence given a rough summary of the story. The ratio of these two likelihoods is a measure of how predictably the story flows from one point to another. Sap et al. (2020) found that fictional stories flow much more predictably than true ones. They also found that true stories begin to flow more predictably when they are retold 2-3 months later. Sap et al. (2022) reproduced these findings using a more advanced language model, GPT-3. We will discuss these and other methods of measuring linguistic complexity in Chapter 22.