17. Run and adapt LLMs for your research
17. Run and adapt LLMs for your research
Running an LLM yourself can give you control over where research material is processed and preserve access to the version used in a study. Training an existing LLM on reviewed examples may improve a recurring task. These are separate choices: the first concerns where computation happens, while the second changes what the LLM has learned.
For the interview-labeling task in Chapter 16: Use LLM outputs as research data, data restrictions might require processing on approved local hardware. Persistent labeling errors might also justify testing whether further training improves performance. An unchanged LLM can run locally, and a locally run LLM can still receive instructions and examples with each prompt.
Choose where the LLM runs
An LLM’s weights are numerical values learned during training. An open-weight LLM makes those values available to download and use under specified terms. Open-source AI requires more than downloadable weights: the Open Source Initiative’s definition also covers rights to use, study, modify, and share, along with code and sufficient training-data information.1
“Hosting locally” can mean running an LLM on your everyday computer. A runtime is software that loads the LLM and executes requests; Ollama is one example. Ask the agent to connect a suitable runtime to the script or agent that will submit the research inputs. Larger LLMs or workloads may require a more powerful computer. Data restrictions may also require the work to run on approved institutional hardware.
Using a hosted LLM sends the interview text to the provider’s computers. A promise not to train on it does not settle retention or deletion policies; Chapter 3: Know where your data goes explains how to assess those arrangements. Synthetic records can help develop the labeling code, but the actual interview text must reach the LLM that labels it. Open weights used through someone else’s hosted service still send inputs to that provider.
With local processing, have the agent also check whether tools, connected services, or backups send research material elsewhere, and where logs and caches retain copies. Choosing local hardware controls the LLM’s location; the rest of the workflow still needs to match the study’s data requirements.
Choose an LLM for the task and hardware
Ask the agent to compare candidate models, taking into account the research workload and available computing resources. Include the types of inputs: a text-only LLM cannot directly interpret an audio recording or image. For labeling, the main question is how well each candidate applies the rubric. If an agent will use the LLM to operate tools, its support for tool requests also matters.
| Requirement | What the agent should establish |
|---|---|
| Research performance | Errors on development examples, including consequential mistakes and cases needing human review. Confirm the selected configuration on held-out cases. |
| Inputs and outputs | Support for the material and its length, and reliable structured output: responses with specified fields, such as an excerpt identifier and label. |
| Available hardware | Enough working memory for the LLM, the supplied context, and the intended number of simultaneous requests. |
| Time and cost | Processing time on a representative sample, expected total workload, and any computing charges. |
| Tool use | Compatibility with the intended agent and its tools, if the research task requires them. |
Weights occupy disk and working memory. Longer contexts and simultaneous requests need additional memory. Quantization2 reduces memory requirements by storing numerical values at lower precision. Ask the agent to evaluate the actual quantized version it would use, since changing precision can change outputs. Institutional research computing, such as FAS Research Computing’s cluster, can help when the workload exceeds local resources.
The following table identifies recognizable families and example releases, with release-specific licenses checked in November 2026. It provides names to investigate rather than a ranking. A B in a parameter count means billions of parameters; the download size and working memory also depend on how those values are stored and used.
| Family | Developer | Example release | License for that release |
|---|---|---|---|
| Qwen | Alibaba | Qwen3-4B | Apache 2.0 |
| Gemma | Google DeepMind | Gemma 4 E4B | Apache 2.0 |
| Mistral | Mistral AI | Ministral 3 8B Instruct | Apache 2.0 |
| Llama | Meta | Llama 3.3 70B Instruct | Custom Llama 3.3 license |
| DeepSeek | DeepSeek | DeepSeek-V3.2 | MIT |
| Kimi | Moonshot AI | Kimi K2 | Modified MIT |
| GLM | Z.ai | GLM-4.7 | MIT |
| MiniMax | MiniMax | MiniMax-M2.7 | Custom noncommercial terms; commercial use requires authorization |
The smaller Qwen, Gemma, and Ministral examples are candidates for a personal computer, subject to its memory and the workload. Full releases such as Kimi K2 and DeepSeek-V3.2 require substantially greater resources. Have the agent report the exact release, license, runtime, and quantization used in its comparison. A license name alone does not establish that the system meets the open-source definition.
Chapter 17: Run and adapt an LLM on your own hardware in the companion guide Extending Your Agent’s Toolkit covers installation, hardware checks, and connecting a compatible LLM to an agent.
Decide whether to supply context or fine-tune
Start by giving an existing LLM a clear rubric and representative examples. In-context learning uses examples supplied with the prompt without updating weights. For tasks requiring information from documents, retrieval selects relevant passages to include in that context.3 The agent can arrange either approach without training the LLM.
Fine-tuning trains an existing LLM on additional data. For interview labeling, supervised training pairs excerpts with reviewed labels so training can adjust the LLM toward the required responses. It can use many examples in successive groups, called training batches, without putting them all in one prompt. The resulting LLM still receives each new excerpt and the instructions needed to process it.
| Research need | Starting approach |
|---|---|
| Answer from current documents with source support | Supply passages directly or through retrieval. |
| Apply a limited or changing rubric | Prompt an existing LLM with selected examples. |
| Improve a recurring task whose errors remain consequential | Test fine-tuning after developing a strong prompt, if suitable training and evaluation data are available. |
Fine-tuning may improve labeling accuracy, adherence to the output format, or performance with shorter prompts.4 It adds training and maintenance work, especially when the rubric changes. A fine-tuned LLM can still receive retrieved passages when it needs information from documents.
Understand LoRA and adapters
LoRA, or Low-Rank Adaptation, is a fine-tuning method. Training software keeps the original, or base, weights fixed and learns compact adjustments to selected parts of the LLM. Those adjustments are saved as an adapter, used with the matching base LLM or merged into a new copy.5
For example, Qwen names an LLM family, MLX LM and Hugging Face PEFT provide training software, and LoRA describes the method used to learn the adjustments. Unlike LoRA, full fine-tuning updates all of the LLM’s original weights and generally requires more training memory.
QLoRA combines low-rank adaptation with a quantized base LLM to reduce training memory further.6 Whether a configuration fits the available hardware still depends on the release, context length, and training settings. The companion toolkit’s adaptation section explains how to have the agent set up training tools.
Check whether fine-tuning helps
For the interview study, the question is whether fine-tuning improves labeling enough to justify preparing training data and maintaining an adapted LLM. Ask the agent to compare it with a baseline: the original LLM using the best prompt developed for the task.
Training examples supply the material used to learn the adjustments; development examples support choosing candidate versions and settings. Once those choices are fixed, the agent should compare the original and adapted LLMs on the same untouched test cases, without supplying their human labels to either LLM. Chapter 16: Use LLM outputs as research data explains how to separate the material and evaluate the labels.
The comparison should show whether fine-tuning reduces errors important to the study and how those changes affect the research estimates. Remaining human review and the costs of preparing examples, training, and generating responses also matter when deciding whether to adopt the adapted LLM. Better agreement with human labels does not resolve a poorly defined research category.
Save what is needed to use the LLM again
The files needed to run an adapted LLM differ from those needed to repeat its training. Ask the agent to preserve both sets and keep earlier configurations and evaluation results available for comparison.
| To do this later | What needs preserving |
|---|---|
| Run the adapted LLM | The matching base LLM and adapter—or fully updated weights—plus the software and settings used to run them. |
| Repeat the training | The original base LLM, reviewed data and assignments to training/development/test sets, plus training code and settings. |
Training can leave sensitive information in learned weights, adapters, or logs.7 These files need approved storage and appropriate access restrictions, just as the original records do. The agent should link the selected configuration, evaluation criteria, and results from SPECIFICATION.md, and record approved changes and their reasons in METHODOLOGY.md.
Preserving an LLM does not replace saving the responses actually used in the study. Chapter 16: Use LLM outputs as research data explains how those responses support reproducing the analysis without generating them again.
Footnotes
Open Source AI Definition 1.0, Open Source Initiative. The definition requires parameters, code, and information sufficient to understand the training data; it does not require publishing every raw training record.↩︎
Quantization represents an LLM’s numerical weights using fewer bits—for example, 8 rather than 32 per value—reducing the memory needed to store and run it. See Hugging Face’s explanation.↩︎
Ovadia et al. (2024), “Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs”, found retrieval more effective than unsupervised fine-tuning for the knowledge-intensive tasks tested. Training on unlabeled text differs from supervised training on reviewed excerpt/label pairs.↩︎
Liu et al. (2022), “Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning”, found accuracy and prediction-time computation benefits for T-Few with IA³ under their comparisons. This was not a general demonstration that LoRA outperforms a well-prompted current LLM on any research task.↩︎
Hu et al. (2021), “LoRA: Low-Rank Adaptation of Large Language Models”, explains the method and reductions in trainable parameters and training memory.↩︎
Dettmers et al. (2023), “QLoRA: Efficient Finetuning of Quantized LLMs”. Its hardware demonstrations concern specified training configurations, not every release and workload.↩︎
Zeng et al. (2024), “Exploring Memorization in Fine-tuned Language Models”, found that memorization differed across fine-tuning tasks. Keeping training local does not remove the need to protect the resulting weights and related artifacts.↩︎