16. Use LLM outputs as research data
16. Use LLM outputs as research data
Large language models (LLMs) can help researchers turn large collections of documents, recordings, or images into data for analysis. They can assign categories to interview passages, extract reported events, or assess written reviews. An agent can build the code that prepares the material, requests these outputs, and saves them as research variables. The preceding chapters explained how to check that code; this chapter examines whether the outputs measure what the study requires.
This chapter uses a running example: a research team wants to estimate the proportion of interview excerpts that describe an unmet service need. An LLM could label each excerpt, allowing the team to examine more material than it could read manually.1 The team needs to establish what counts as an unmet need, evaluate the labels against human judgments, and account for labeling errors when estimating that proportion. The following sections explain how an agent can help with each step.
Define what the LLM should produce
Ask the agent to draft a rubric: instructions defining the categories, exclusions, and treatment of ambiguous cases. Researchers familiar with the subject should review it and label a set of examples, resolving disagreements and recording their reasons. These human judgments provide a reference against which to evaluate the LLM. Sometimes called a gold standard, the reference also needs scrutiny when people disagree about what a passage means.
Work with the agent to develop the rubric and prompt on examples reserved for that purpose. Include difficult cases and specify when the LLM should leave a case for human review. Ask the agent to record what the labels represent in SPECIFICATION.md, with links to the rubric and evaluation criteria. It should retain the exact rubric and prompt used for each completed run and record approved revisions and their reasons in METHODOLOGY.md.
For recorded interviews, the agent might first use speech-recognition software to transcribe the audio, then ask an LLM to label passages. Have it select a sample for human review against the recordings so the evaluation captures transcription errors as well as labeling errors. Some LLMs can process images, audio, or video alongside text. These are called multimodal LLMs. Ask the agent to check whether the chosen LLM supports the material you want to analyze. Chapter 12: Bring data into the project covers collecting and documenting those inputs.
Using a trained LLM to generate a response is called LLM inference. This is a form of statistical prediction: the LLM uses patterns learned during training and the supplied context to predict successive tokens2. “Inference” here therefore means something different from statistical inference about a population or an effect. For the interview-labeling task, the agent can write a script that submits each excerpt with the rubric and saves the returned label. If the task also requires tools, such as retrieving passages from several files, the script can launch a separate agent session to do that work.
Evaluate the labels against human judgments
Evaluating LLM-generated labels follows a familiar principle from predictive modeling: assess performance on examples that were not used to develop or select the method.3 Even without training an LLM, researchers make choices about the rubric, prompt, and which LLM to use. Repeatedly adjusting those choices to improve results on the same examples can give an overly optimistic picture of performance on new material.
For the interview example, ask the agent to divide the human-labeled excerpts into a development set and a separate held-out test set before work on the prompt begins. Use the development examples to refine the rubric and prompt and compare LLMs and generation settings. Keep the test set separate from all those decisions. All excerpts from one participant or document should belong to the same set, so closely related passages do not appear in both development and testing.
If the team also fine-tunes an LLM, a training set supplies examples used to update its learned parameters. The development set helps compare candidate LLMs, prompts, and settings. Both contribute to preparing the labeling method; the test set remains separate for final evaluation. Chapter 17: Run and adapt LLMs for your research explains fine-tuning. The interview example here uses an already-trained LLM without further training.
Using the research question, development material, and reviewed human examples, help develop a rubric and prompt for the labeling task. Identify ambiguous cases and category definitions that need my decision. Compare the agreed candidate prompts and LLMs on development material only, reporting category-specific errors and cases requiring human review. Keep all excerpts from the same participant or document together when assigning material to development and test sets. Do not inspect the test set while selecting the method. Save the candidate versions and comparisons, then record the chosen rubric, prompt, LLM settings, and final-test criteria in
SPECIFICATION.mdafter my approval.
Set criteria for acceptable performance before the final test, including which errors matter for the study. Once the rubric, prompt, LLM version where available, and generation settings are fixed, ask the agent to apply them to the test excerpts. The LLM should receive those excerpts without their human labels. The agent can then compare its predictions with the withheld human judgments and report whether they meet the criteria.
A confusion matrix summarizes agreement and errors by category. In this illustrative matrix for 200 test excerpts, rows show the human reference labels and columns show the LLM’s predictions. Each cell counts the excerpts with that combination of labels:
| Human reference | LLM: need present | LLM: need absent |
|---|---|---|
| Need present | 10 | 10 |
| Need absent | 5 | 175 |
Overall agreement is 92.5 percent: the LLM and human reference agree on 185 of 200 excerpts (10 positive and 175 negative). Yet the LLM finds only half of the 20 positive cases, and five of its 15 positive labels are false positives.4 Ask the agent to report these category-specific errors and their uncertainty, including performance for relevant document types or groups of participants. Also count failed requests and cases sent for human review, because excluding them can conceal errors and understate the work remaining.
Apply the fixed rubric, prompt, and LLM settings to the held-out test excerpts without supplying their human labels to the LLM. Save each raw response, the label extracted from it, failed requests, and cases requiring human review. Then compare the labels with the reviewed human reference. Report a confusion matrix, uncertainty in error rates, differences across the specified groups, and remaining human review. Assess the acceptance criteria supplied before the run. Do not revise the labeling method using this test set or omit difficult cases.
An LLM’s stated confidence also needs evaluation. If it assigns 80 percent confidence to a set of labels, roughly 80 percent would need to be correct for that confidence to be well calibrated on those cases. Do not use its confidence statements as a substitute for comparison with the human reference.
Account for labeling errors in the analysis
In the table above, the human reference identifies unmet need in 20 of 200 excerpts, or 10 percent. The LLM identifies 15, or 7.5 percent. Despite high overall agreement, the LLM labels give a different estimate of prevalence. Errors can also distort comparisons and regression estimates, especially when the likelihood of an error differs between groups or depends on other variables in the analysis.
Ask the agent to examine the errors that matter for the planned analysis and propose how to account for them. Methods such as design-based supervised learning combine a sample of human-reviewed observations with the larger set of LLM-labeled observations to estimate research quantities and their uncertainty. Their validity depends on the sampling and estimation procedure; a single accuracy score is not enough to make the correction.5
Agree on the approach before the full labeling run, including how observations will be selected for human review. Have the agent preserve those selections, the human and LLM labels, and the code for the proposed adjustment or sensitivity analysis. It should show how accounting for errors changes the estimates and intervals. When the evaluation material does not support a credible correction, additional human review or a narrower research claim may be needed.
Check sensitivity to prompts and LLMs
Research conclusions should not depend on an arbitrary wording choice. Ask the agent to vary reasonable features of the labeling method while keeping the research question and rubric fixed: equivalent prompt wording, the order of examples or material, the LLM or version, and generation settings. Specify these comparisons before examining their results. Changing what a category includes changes the measurement itself; resolve that definition separately.
Have the agent repeat requests with unchanged settings as well, so ordinary run-to-run variation can be distinguished from differences between configurations. For the interview example, compare which excerpts change labels and whether those changes alter the estimated proportion with unmet need or differences between participant groups. Where human reference labels are available, also compare error rates. A change that affects few labels could still matter if those labels concern a small group central to the research question.
The relevant variations depend on the task. When an LLM compares two written responses, reversing their order can reveal a preference for whichever appears first. Research on judging chatbot responses has documented position and verbosity biases, motivating checks of ordering and response length when they could affect the research task.6
Use development material when these comparisons will guide selection of a prompt or LLM. Prespecified sensitivity analyses can also show how final research estimates vary across configurations; do not select whichever gives the preferred substantive finding. If test results lead to changes in the labeling method, evaluate the revised method on new held-out material. When no defensible human reference exists, sensitivity analysis can still reveal unstable results, but stable outputs alone do not establish accurate or valid measurements.
Using the agreed comparisons, keep the research question and category definitions fixed while varying prompt wording, example order, LLM choice, or generation settings. Repeat requests with unchanged settings to assess ordinary run-to-run variation. Report which labels change and how those changes affect the research estimates, including relevant group comparisons. Compare errors where human reference labels are available. Use development material when the comparisons will guide method selection; keep prespecified analyses of final estimates separate. Save every configuration and response. Do not select a method because it produces a preferred research finding, and do not interpret stable labels as evidence of accuracy.
Evaluate simulated respondents and teams
LLMs can also generate responses as simulated participants, allowing researchers to explore proposed questions or interactions before organizing a human study. For example, an agent could arrange several LLM sessions to represent members of a small team, supply each with a role and task, and record their exchanges and decisions. These responses are generated behavior, rather than measurements collected from people. Unlike the simulations in Chapter 14: Test the analysis against known truth, their behavior does not follow a generating mechanism whose relevant relationships the researcher has explicitly set.
Define what the simulation needs to reproduce, then ask the agent to compare its outputs with relevant empirical results. For a team study, these might include differences in performance under particular working conditions, with estimates from comparable studies or a meta-analysis. Compare effect sizes and variation, as well as the direction of a difference. Keep some empirical comparisons out of prompt development, and do not supply the target results in the simulated participants’ instructions.
Agreement with one finding does not establish that simulated participants represent the target population or will respond realistically to a new intervention. A study found that LLM-generated survey responses could have averages similar to those from human respondents, while showing less variation and different associations between variables.7 Use the comparison to state which aspects of the simulation have empirical support and which remain exploratory. More generated responses can make a simulated estimate more precise without making it a better estimate of human behavior.
Run the procedure across many cases
Once the labeling procedure is ready, ask the agent to apply it to the full collection using the saved code. It should first run a few cases through every stage to check that requests, returned labels, and failures are recorded correctly. This smoke test checks execution; the held-out evaluation establishes measurement performance. Separate the collection of LLM responses from their analysis so a revised figure does not require labeling the interviews again.
A headless agent8 follows supplied instructions without an interactive conversation. The agent in the app can prepare a file containing the prompt and input locations, then launch fresh headless sessions from a script. This extends the approach used for independent implementations in Chapter 15: Check the agent with independent implementations. You do not need to operate a terminal yourself.
Keep each run’s context separate
Processing every case in one long conversation lets earlier cases influence later responses. Ask the agent to start a fresh headless session for each case, supply the same fixed instructions and its assigned input, and save its response separately. For interview labeling, each session receives the rubric and its assigned excerpt. The agent can repeat these calls unattended without carrying conversation history from one case to the next.
A fresh headless run can still load global user instructions, memory, files, plugins, or connected services. Ask the agent to check which of these each run can access and configure it to match the research protocol. It should identify any files or services whose access it could not verify. Fresh conversations alone establish neither access isolation nor statistical independence between the resulting outputs.9
Prepare one fresh headless session per case, using the supplied research protocol and fixed instructions. Give each session its assigned input without conversation history from other cases. Configure and check which files, tools, and services it can access; report any access you cannot verify. Run a few cases through requesting and saving responses before launching the full collection. Save each case’s inputs, raw response, failures, retries, LLM identifiers, and settings. Keep response collection separate from analysis so the saved outputs can be analyzed again without new LLM calls.
A job queue can schedule these runs. It controls when they start; the session configuration determines what information and tools they can access.
Save the outputs and reproduce the analysis
Have the agent save the presented inputs, prompts, raw responses, and code that extracts research variables from them. Retain failed requests, retries, and human corrections alongside successful responses, with the available LLM version identifiers and generation settings. Saved prompts, responses, and logs may contain sensitive information from the original data. Ask the agent to keep these files in approved storage and exclude them from any materials prepared for sharing with people who are not authorized to see the data.
| Task | What the agent uses |
|---|---|
| Reproduce the reported analysis | Saved responses and the code that turns them into research variables and results. |
| Generate another set of responses | The saved inputs, prompt, LLM configuration, and software that submits requests. |
| Assess whether the measurements remain suitable | Human reference material, the evaluation procedure, and checks of changes in the research estimates. |
Rerunning the analysis from saved responses can reproduce the reported results without calling an LLM. Generating responses again involves a new collection run: even a fixed configuration may produce different responses across multiple runs, and an app’s LLM name may not identify an exact LLM version. Ask the agent to preserve each collection separately and report changes that affect the conclusions. Chapter 17: Run and adapt LLMs for your research explains how running an LLM yourself or training it further changes these choices.
Footnotes
Gilardi, Alizadeh, and Kubli (2023), “ChatGPT outperforms crowd workers for text-annotation tasks”, found useful performance on specific annotation tasks involving tweets. Those results motivate evaluating LLM annotation on a research task; they do not establish its accuracy on other material.↩︎
A token is a unit of text processed by an LLM. It may be a whole word, part of a word, or a punctuation mark.↩︎
The
scikit-learndocumentation on model evaluation explains why test data must remain separate from fitting and selecting a predictive model. The same separation applies when selecting prompts and LLMs rather than fitting a new predictive model.↩︎Overall agreement is accuracy. Sensitivity (recall) is the proportion of human-positive cases detected: here, 10 / 20 = 50 percent. Precision (positive predictive value) is the proportion of positive predictions that are correct: here, 10 / 15 = 66.7 percent. Other summaries include F1, which combines precision and recall, and balanced accuracy. Choose measures that reflect the errors important to the research question; the
scikit-learnoverview of classification metrics explains these alternatives.↩︎Egami, Hinck, Stewart, and Wei (2023), “Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models”, show how high annotation accuracy can coexist with biased estimates and invalid confidence intervals. Their method combines surrogate labels with human reference labels using a specified sampling design for human annotation.↩︎
Zheng et al. (2023), “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, evaluate comparisons of chatbot responses. Their findings motivate task-specific checks; they do not establish the size of a bias in another research task.↩︎
Bisbee et al. (2024), “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models”, compare generated survey responses with American National Election Studies data. Their findings concern the tested surveys and LLM configurations, rather than every possible use of simulated participants.↩︎
For setup of headless sessions and guidance on controlling each run’s inputs and tool access, see Chapter 16: Run agents from saved prompts in the companion guide Extending Your Agent’s Toolkit.↩︎
Anthropic’s programmatic
Claude Codedocumentation explains noninteractive calls and their configuration, including which ordinary instructions and tools they can inherit.↩︎