Supervised Fine-Tuning

Supervised fine-tuning is a machine learning method that adapts a model already trained on a broad dataset by continuing training on labeled input–output examples with a supervised loss.^[1] For large language models, that loss is usually next-token prediction on demonstration text: the same autoregressive objective used in pretraining, applied to examples of behavior a developer wants the model to produce.^[1] A 2023 survey uses “supervised fine-tuning” and “instruction tuning” interchangeably for this further training on pairs of an instruction and a target output, which steers a model trained to predict the next word toward following human instructions.^[1] The fine-tuning step itself predates that post-training vocabulary. In natural language processing, Howard and Ruder’s 2018 universal language-model fine-tuning method adapted a pretrained language model to text classification and, with 100 labeled examples, matched training from scratch on 100 times as much labeled data.^[2] In 2021, Wei and colleagues instruction-tuned a 137-billion-parameter pretrained model on more than 60 NLP tasks written as natural-language instructions; the resulting FLAN model surpassed zero-shot GPT-3, with 175 billion parameters, on 20 of 25 tasks they evaluated, and on several of those tasks it also beat few-shot GPT-3.^[3] Chung and colleagues later scaled instruction finetuning across model families and to 1,800 tasks, including chain-of-thought answers, and reported that a 540-billion-parameter Flan-PaLM outperformed the corresponding untuned PaLM by 9.4% on average across the benchmarks they measured.^[4] Ouyang and colleagues, in a 2022 paper from OpenAI, gave the supervised stage the name used in later alignment recipes: supervised fine-tuning (SFT).^[5] They collected demonstrations of desired behavior for labeler-written prompts and prompts submitted to the OpenAI API, fine-tuned GPT-3 on those demonstrations, then gathered rankings of model outputs and continued training with reinforcement learning from human feedback.^[5] On the prompt distribution used in their human evaluation, raters preferred outputs from the 1.3-billion-parameter InstructGPT model to outputs from the 175-billion-parameter GPT-3.^[5] OpenAI’s November 2022 research preview of ChatGPT described the same sequence for a dialogue model—supervised fine-tuning on trainer-written conversations, a reward model fit to ranked responses, and proximal policy optimization—with a different setup for collecting the dialogue data.^[6] Subsequent work changed the demonstrations and the steps that follow the supervised update. Wang and colleagues’ Self-Instruct sampled instructions, inputs, and outputs from a language model, filtered them, and fine-tuned GPT-3; they reported a 33-percentage-point absolute gain on Super-NaturalInstructions over the untuned model, close to an InstructGPT baseline trained with private human annotation.^[7] Zhou and colleagues fine-tuned a 65-billion-parameter LLaMA model, called LIMA, with the standard supervised loss on 1,000 curated prompts and responses and without reinforcement learning.^[8] In their controlled study, raters judged LIMA responses equivalent or preferable to GPT-4 responses in 43% of cases, and the authors argued that pretraining already accounts for most of a model’s knowledge while a small instruction set mainly teaches the format of a high-quality answer.^[8] Direct preference optimization, introduced by Rafailov and colleagues in 2023, starts from a supervised policy and fits human preference pairs with a classification loss, removing the separate reinforcement-learning loop used in the earlier pipeline.^[9] When updating every weight is costly, training uses parameter-efficient fine-tuning. Low-rank adaptation, introduced by Hu and colleagues, freezes the pretrained Transformer weights and trains low-rank update matrices inside each layer.^[10] For GPT-3 at 175 billion parameters, they reported about a 10,000-fold cut in the number of trainable parameters and about a threefold cut in GPU memory relative to full fine-tuning with Adam, with quality on par with or above full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3.^[10] The same supervised adaptation is applied to domain and task specialization and to vision, speech, and multimodal models; surveys of the method also record its sensitivity to the demonstration set and disagreement over how much new capability the stage adds beyond a change in response style.^[1]

Definition and objective

Supervised fine-tuning (SFT) is continued training of a model already fit to a broad source distribution, using labeled input–output pairs and a supervised loss. For generative language models, each pair is a prompt and a target completion, and the loss is next-token cross-entropy on that completion. Three uses of the name should be kept apart. Task fine-tuning adapts a pretrained network to one labeled problem. Instruction tuning fine-tunes on many tasks written as natural-language instructions, aiming at held-out task types. The named SFT stage in post-training imitates demonstrations of desired responses, usually before preference optimization.^[5][3] Pretraining an autoregressive Transformer estimates the next token on unlabeled text and does not designate a correct answer to a request.^[11] SFT keeps that factorization and changes the data: each update raises the probability of a specified output given its input. Instruction tuning builds the pairs by rewriting supervised natural language processing datasets with templates. FLAN did this for a decoder-only model. T0, an encoder–decoder, reads the prompted input in the encoder and is trained by maximum likelihood to generate the target and never the input.^[3][12] Task fine-tuning need not be generative. ULMFiT ends by training a classifier with label cross-entropy.^[2] In alignment recipes the named SFT stage is the supervised policy that later methods start from. InstructGPT fine-tuned pretrained GPT-3 on labeler-written demonstrations, then collected rankings, trained a reward model, and optimized that policy with reinforcement learning from human feedback and proximal policy optimization.^[5] Direct preference optimization draws the same line: SFT maximizes the likelihood of high-quality responses and supplies a reference policy, while the preference stage trains on pairs in which one completion is preferred to another.^[9] SFT has no rejected completion to push down, and it does not add the penalty on drift from a reference model that those preference objectives impose.^[9]

Likelihood objective

For a causal model, write the prompt as $x$ and the target as tokens $y_1,\ldots,y_T$. The SFT objective is the conditional negative log-likelihood

$$\mathcal{L}(\theta)=-\sum_{t=1}^{T}\log \pi_\theta(y_t\mid x,y_{<t}),$$

the token-level cross-entropy of the next-token distribution $\pi_\theta$. With teacher forcing, the model is scored for matching the demonstration token by token, not for finding some other string. Recipes describe the same quantity as cross-entropy on the target tokens.^[13][14] The sum is also the pretraining loss; SFT changes which sequences appear and, commonly, which positions are included.^[11] The minimum is imitation of the fine-tuning targets, not a comparison among alternatives. DPO’s loss is a binary classification loss on a difference of log-probability ratios: a preferred and a dispreferred completion are each compared with a frozen reference, usually the SFT policy, and the update raises one likelihood relative to the other.^[9] Validation likelihood is an incomplete way to choose the SFT checkpoint. In the InstructGPT runs, validation loss overfit after about one epoch, yet further epochs improved reward-model scores and human preference ratings, and the SFT model was selected by the reward-model score.^[5]

Prompt masking and dialogue format

Implementations often leave prompt tokens out of the sum. Llama 2-Chat packed each sample as a prompt and an answer, separated them with a special token, used an autoregressive objective, and zeroed the loss on user-prompt tokens so that backpropagation used only answer tokens.^[11] The Llama 3 supervised stage is described the same way—cross-entropy on target tokens, with the prompt masked—after each example is rendered in a chat protocol whose header tokens name the source of a message and whose termination tokens mark the end of a turn.^[13] FLAN separated inputs from targets with an end-of-sequence token and used input and target lengths of 1024 and 256, so the trained span was the instructed output; T0’s decoder is likewise never trained to emit the input.^[3][12] Full masking is a common default, not a settled optimum. A study of prompt-loss weight, where zero is masking and one is full-sequence loss, found that on short completions performance had a statistically significant negative quadratic relationship with that weight. Small nonzero weights, about 0.01 to 0.5, beat both extremes on the authors’ multiple-choice and short-generation checks; they attribute the gain to regularization rather than to a stabilizing effect. On long completions they report that the weight can be ignored.^[14] The tokens inside the likelihood have to match inference. Dialogue SFT therefore fixes a template that marks roles such as system, user, and assistant, and trains the model to continue after the assistant marker. Skipping that supervised step, in a Zephyr ablation, left a preference-only model struggling to produce the role-marked template and scoring much worse on the authors’ chat benchmarks.^[15] In multi-turn transcripts the mask falls on non-assistant spans, so the loss stays on the responses the demonstration is meant to teach.^[11][13]

History

Fine-tuning of pretrained models

Supervised adaptation of a network already trained on a broad source task was worked out first in vision. Convolutional neural networks trained on ImageNet learned early-layer features, similar to Gabor filters and color blobs, that were not tied to one label set. In 2014 Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson found that transfer worsened in higher layers and across more distant label sets, from specialization and from co-adaptation between layers, and that fine-tuning the copied weights beat freezing them.^[16]

Language copied the schedule rather than the architecture. Jeremy Howard and Sebastian Ruder argued in 2018 that transfer learning in natural language processing still meant task-specific models trained from scratch. Their Universal Language Model Fine-tuning method pretrained a language model, adapted it to the target corpus, and then trained a classifier, with layer-wise learning rates, a slanted triangular schedule, and gradual unfreezing. On six classification datasets it cut error by 18–24% on most sets relative to earlier systems, and with 100 labeled examples it matched training from scratch on 100 times more data.^[2] Transformers made full-parameter fine-tuning the usual setup. Alec Radford and colleagues at OpenAI pretrained a unidirectional Transformer on unlabeled books and fine-tuned it per task, reporting gains on 9 of 12 understanding benchmarks.^[17] Jacob Devlin and colleagues’ BERT, developed at Google, fine-tuned a bidirectional encoder plus one new output layer and reached a GLUE score of 80.5%.^[18] Colin Raffel and colleagues’ T5 dropped the task head as well: classification, translation, question answering, and summarization were all text-to-text, so one loss and one fine-tuning procedure followed span-corruption pretraining on the C4 corpus.^[19] Each run still targeted a named dataset.

Instruction tuning of language models

The next line fine-tuned once on many tasks written as instructions and tested zero-shot transfer to tasks held out of that run. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi built Natural Instructions from crowdsourcing guidelines for existing datasets: 61 tasks and about 193,000 instances. Models that saw the instructions generalized about 19% better to unseen tasks in their comparison, still far from an estimated ceiling.^[20] Super-NaturalInstructions, with contributors including the Allen Institute for AI, expanded the collection to 1,616 tasks. Their T5-based Tk-Instruct beat much larger models, including InstructGPT, by more than 9% on the benchmark’s unseen English tasks.^[21] Jason Wei and colleagues at Google called the procedure instruction tuning. FLAN, a 137-billion-parameter model fine-tuned on more than 60 templated datasets, beat its untuned base and zero-shot GPT-3 (175 billion parameters) on 20 of 25 unseen task types, and beat few-shot GPT-3 on part of that suite. Dataset count, scale, and instruction wording mattered in the ablations.^[3] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, and collaborators at BigScience and Hugging Face tested whether explicit multitask training, rather than pretraining alone, produced the effect. T0 fine-tuned an encoder–decoder on supervised sets rewritten with many PromptSource prompts and often outperformed models up to 16 times larger on held-out tasks.^[12] Scaling studies then varied the size of the mixture. Hyung Won Chung and colleagues instruction-finetuned PaLM, T5, and U-PaLM on about 1,800 tasks, some with chain-of-thought answers. Flan-PaLM 540B gained 9.4 points on average over pretrained PaLM, and smaller Flan-T5 checkpoints were released.^[4] Niklas Muennighoff and other BigScience authors applied the same multitask prompting to BLOOM and mT5, producing BLOOMZ and mT0. English prompts already transferred to languages seen only in pretraining, and the xP3 mixture of supervised data in 46 languages raised zero-shot scores in English and beyond.^[22]

Demonstration-based alignment

A demonstration of a desired answer is a different supervised target from a relabeled benchmark. Daniel Ziegler and colleagues adapted GPT-2 with a reward model of human comparisons for style and summarization.^[23] Nisan Stiennon and colleagues then showed, on Reddit TL;DR, that optimizing those preferences beat both the reference summaries and much larger models trained only to imitate the references.^[24] Long Ouyang and colleagues made demonstrations the first stage for GPT-3. About 40 contractors wrote answers for roughly 13,000 prompts from labeler-written instructions and the OpenAI API. Supervised fine-tuning on those answers was followed by a reward model of human rankings and by reinforcement learning, producing InstructGPT. On the authors’ prompt distribution, raters preferred the 1.3-billion-parameter InstructGPT model to 175-billion-parameter GPT-3, with reported gains in truthfulness and lower toxicity. Fine-tunes on the public FLAN and T0 mixtures trailed the demonstration-only supervised baseline there.^[5] Reinforcement learning from human feedback therefore added a stage rather than replacing supervised training. Anthropic’s Constitutional AI instead fine-tuned on the model’s own revisions, prompted by a written list of principles, and then used reinforcement learning from AI preferences.^[25] Open pretrained large language models then carried demonstrations outside one vendor’s contractors. Yizhong Wang and colleagues’ Self-Instruct had GPT-3 draft instructions and answers, filtered them, and fine-tuned the same model, reporting a 33-point gain on Super-NaturalInstructions, near an early private-data InstructGPT, and a 5-point gap on a separate review of expert-written tasks.^[7] In March 2023 Rohan Taori and colleagues at Stanford University fine-tuned LLaMA 7B on 52,000 text-davinci-003 demonstrations in that style, at a stated cost under $600, and released the recipe for academic use only because of LLaMA’s license, OpenAI’s terms, and missing safety filters. A resemblance to text-davinci-003 was their preliminary check, not an independent test.^[26] OpenAssistant released 161,443 crowdsourced messages in 35 languages, from more than 13,500 volunteers, for supervised fine-tuning and preference training.^[27] The useful size of that set was also tested. Chunting Zhou and colleagues’ LIMA fine-tuned a 65-billion-parameter LLaMA on 1,000 curated demonstrations and no preference model; in their study it tied or beat GPT-4 in 43% of cases, which they read as evidence that pretraining holds most capabilities and supervised fine-tuning mainly teaches format.^[8] Meta’s Llama 2-Chat models, from 7 to 70 billion parameters, set aside bulk third-party instruction data as too narrow for dialogue, trained on 27,540 vendor demonstrations with no Meta user data and with the loss masked on user tokens, and only then collected preferences for reinforcement learning, after supervised samples often rivaled the handwritten answers.^[11]

Training data

Supervised fine-tuning of large language models trains on paired prompts and target responses. Those pairs have been written by contractors, volunteers, or employees; made by casting public natural language processing datasets as instructions; taken from curated community answers; or sampled from an already capable model, and open recipes usually combine several of these sources.^[28]

Human demonstrations

For InstructGPT, OpenAI hired about 40 contractors through Upwork and Scale AI to demonstrate desired behavior on API Playground prompts and on prompts the labelers wrote. The supervised set contained about 13,000 training prompts and was more than 96 percent English; rankings for the reward model in reinforcement learning from human feedback were collected separately.^[5] OpenAssistant Conversations came from more than 13,500 volunteers: 161,443 messages in 66,497 conversation trees across 35 languages, with 461,292 quality ratings and 10,968 trees counted as complete. Of the messages, 8,576 are synthetic. The authors describe the licence as fully permissive.^[27] Separately, more than 5,000 Databricks employees wrote the 15,000 English pairs in databricks-dolly-15k in March–April 2023, in categories such as question answering, extraction, summarization, and brainstorming. They were told not to use generative models, though closed tasks could quote Wikipedia. The set uses Creative Commons Attribution-ShareAlike 3.0.^[29] A parallel line reformats existing tasks instead of soliciting new chats. Wei and colleagues grouped 62 public datasets into twelve clusters and wrote ten natural-language templates per dataset, including some that inverted the original task.^[3] Chung and colleagues expanded the pool to 473 datasets and 1,836 tasks from that collection, T0, and Natural Instructions, plus dialogue, program synthesis, and chain-of-thought data, and removed MMLU items so the benchmark stayed held out.^[4] Super-NaturalInstructions is itself 1,616 tasks with expert-written instructions across 76 task types.^[21] LIMA used exactly 1,000 training sequences, about 750,000 tokens, from highly rated Stack Exchange answers, wikiHow, Reddit writing prompts, a Natural Instructions slice, and author-written examples in one assistant style.^[8]

Synthetic data

Self-Instruct starts from 175 author-written seed tasks and prompts unmodified GPT-3 to propose instructions, inputs, and outputs, then drops invalid or near-duplicate samples. The release has about 52,000 instructions and about 82,000 instances. Fine-tuning GPT-3 on it raised Super-NaturalInstructions by 33 absolute points, matching InstructGPT-001 on that benchmark, and left a 5-point gap to InstructGPT-001 in a human evaluation on novel tasks.^[7] Stanford University’s Alpaca drew about 52,000 demonstrations from text-davinci-003 and fine-tuned Meta’s LLaMA 7B. The data are CC BY-NC 4.0 and were released for academic research; the authors also cited LLaMA’s non-commercial terms and OpenAI’s limits on competing models.^[26][30] Evol-Instruct, used for WizardLM, rewrites a seed such as Alpaca. In-depth edits add constraints or reasoning steps, in-breadth edits propose a new but related instruction, and a filter drops failures. Four rounds produced about 250,000 instructions. A LLaMA-13B model trained on 70,000 of them, a scale chosen to match Vicuna’s ShareGPT set, was preferred to Alpaca and Vicuna in the authors’ human evaluation, and to ChatGPT on their high-complexity slice.^[31] UltraChat contains about 1.5 million multi-turn dialogues and no human queries: two ChatGPT instances play user and assistant.^[32] ShareGPT is mixed in origin—people shared their prompts along with model completions. The exact Vicuna subset was not released, so later mixtures used public reproductions split into shorter blocks.^[28] Imitation does not by itself transfer measured skill. Gudibande and colleagues fine-tuned 1.5 billion- to 13 billion-parameter models on ChatGPT outputs; crowd raters often judged the copies competitive with ChatGPT, but benchmarks on tasks thinly represented in the data closed little of the gap. Style transferred more readily than factual accuracy.^[33]

Mixture design and quality

In the Allen Institute for AI comparison behind Tülu, chain-of-thought data helped grade-school math, code data helped coding, and ShareGPT or GPT-4 Alpaca helped open-ended following, but no dataset or mixture led every evaluation. Most single sets lowered GSM and TyDiQA versus the untuned base model; the authors attribute that to scarce reasoning traces and multilingual text, and so to forgetting. Self-Instruct, from base GPT-3, was weak on most tasks. Concatenating sampled FLAN and chain-of-thought data with Dolly, OpenAssistant, GPT-4 Alpaca, Code Alpaca, and ShareGPT gave the best average. Preference ratings did not track the benchmark gaps.^[28] Tülu 2 cut the supervised mix from 490,445 to 326,154 examples: 50,000 FLAN v2 examples and 50,000 chain-of-thought examples, 7,708 top-rated OpenAssistant paths, 114,046 processed ShareGPT conversations, 20,000 GPT-4 Alpaca examples, 20,022 Code Alpaca examples, 1,030 LIMA examples, 30,000 WizardLM examples, 30,000 GPT-4 OpenOrca examples, 7,544 scientific-document tasks, and 140 author-written prompts. Dolly was dropped after weak earlier ablations.^[34] Longpre and colleagues found task balancing and richer templates easy to underweight, and found that mixing zero-shot, few-shot, and chain-of-thought prompts improved every setting they measured by more than two points.^[35] Quality and scale do not trade off in only one direction. LIMA’s superficial alignment hypothesis says pretraining already holds most knowledge and supervised tuning mainly selects a style. In that study, LIMA was equivalent or preferred to GPT-4 in 43 percent of cases, to Bard in 58 percent, and to text-davinci-003 in 65 percent.^[8] Within one noisy corpus, AlpaGasus kept about 9,000 of Alpaca’s roughly 52,000 examples after a chat model scored them, and that subset beat the full set on the authors’ GPT-4 and human evaluations.^[36] A 2024 study of Llama-2, Llama-3, and Mistral instead found math, coding, instruction following, and multi-hop question answering scaling as a power law in the number of fine-tuning examples. A handful of examples mainly changed style and did not saturate those benchmarks, and post-training could add knowledge rather than only restyle pretraining. The authors call the hypothesis at best an oversimplification when evaluation stops at preference or format.^[37]

Optimization methods

Supervised fine-tuning minimizes next-token cross-entropy on labeled demonstrations. The choice is which parameters move: every pretrained weight, or a small module on a frozen large language model, sometimes stored at low precision. Learning rate, sequence packing, and limits on drift from the pretrained model determine what those labels change.

Full-parameter fine-tuning

Full fine-tuning starts from the pretrained weights and updates all of them under the conditional language-modeling loss, the sum of log probabilities of target tokens given the prompt and earlier targets.^[38] Optimizers such as Adam also store moment estimates, so memory scales with the full parameter count. Llama 2 pretraining peaked at $3\times10^{-4}$ for the 7B and 13B models and $1.5\times10^{-4}$ for the 34B and 70B models; supervised fine-tuning used a cosine schedule from $2\times10^{-5}$, weight decay 0.1, batch size 64, length 4,096, and two epochs.^[39] Prompts and answers were concatenated with a separator, and the loss on user-prompt tokens was zeroed.^[39] The largest Llama 3 models used $10^{-5}$ for about 8,500 to 9,000 steps, with loss only on target tokens.^[40] InstructGPT fine-tuned GPT-3 for 16 epochs at $9.65\times10^{-6}$ for the 1.3B and 6B models and $5.03\times10^{-6}$ for 175B.^[41] Validation loss overfit after about one epoch, but further epochs improved reward-model scores and human preferences, so checkpoints were chosen by reward-model score rather than validation loss.^[41] A two-epoch run used to initialize later preference optimization mixed in 10% pretraining tokens.^[41] Sequence packing concatenates variable-length examples instead of padding each one. On Llama 3 models from 8B to 70B and supervised sets from about 69,000 to 1.2 million examples, packing beat padding on average, with a larger gap at 70B, and greedy packing beat random packing on instruction following by keeping multi-turn conversations intact.^[42] One 70B chat run took 3,749 seconds with random packing versus 9,533 with padding.^[42] Batch size and learning rate were no longer proportional under packing, and packing only single-turn data can create false multi-turn context that hurts few-shot scores.^[42]

Parameter-efficient adaptation

Parameter-efficient methods freeze the pretrained network and store a small task-specific increment. Houlsby et al. inserted bottleneck adapters after attention and the feed-forward sublayer of BERT, training the adapters, layer-normalization parameters, and a classifier. On GLUE, BERT-Large adapters scored 80.0 against 80.4 for full fine-tuning at about 3.6% extra parameters per task.^[43] Prefix-tuning learns continuous vectors inside each Transformer layer.^[44] Prompt tuning learns only a short input soft prompt and, at T5 scales of billions of parameters, approached full tuning with tens of thousands of parameters per task.^[45] Adapters add inference latency, and prefixes occupy context length.^[38]

Low-rank adaptation (LoRA) writes the update of a weight $W_0$ as $BA$, with rank $r$ far below the hidden size, freezes $W_0$, and trains only $A$ and $B$, so the forward pass is $h = W_0 x + BAx$.^[38] After training, $BA$ can be merged into $W_0$, leaving no extra inference cost.^[38] On GPT-3 175B, LoRA was reported to cut trainable parameters by about 10,000 times and Adam GPU memory by about three times versus full fine-tuning, matching or exceeding full fine-tuning on the RoBERTa, DeBERTa, GPT-2, and GPT-3 tasks in that study.^[38] QLoRA stores the frozen weights in 4-bit NormalFloat, quantizes the quantization constants again, and uses paged optimizers so gradients pass through the quantized model into LoRA adapters.^[46] It was reported to fine-tune a 65-billion-parameter model on one 48 GB GPU and to match 16-bit full fine-tuning on that study's tasks.^[46] Parity is not uniform: on Llama-2-7B code and math, low ranks learned less than full fine-tuning, while rank 256 closed much of the instruction-tuning gap, and on the mixed Tülu-v2 set lower ranks matched full fine-tuning on MT-Bench, GSM8K, and MMLU.^[47] Best LoRA learning rates there were about ten times the full-fine-tuning rate, often $5\times10^{-5}$ to $5\times10^{-4}$, with a recommendation to adapt all transformer modules and set the scale $\alpha = 2r$ at high rank.^[47]

Regularization and forgetting

Narrow supervised updates can erase other behavior, a form of catastrophic interference. In sequential instruction tuning of models from about 1 billion to 7 billion parameters, domain knowledge, reasoning, and reading comprehension declined as tasks were added; larger models in that range forgot more, partly because they started higher, and a prior general instruction-tuning stage reduced later forgetting.^[48] Universal Language Model Fine-tuning used a lower learning rate in each lower layer, $1/2.6$ the rate of the layer above, and unfroze layers gradually from the top.^[49] Rehearsal mixes older data into the supervised update, as in InstructGPT's two-epoch run with 10% pretraining tokens.^[41] InstructGPT applied a per-token Kullback–Leibler divergence penalty toward the supervised model in the preference stage, not in the demonstration loss.^[41] On Llama-2-7B code and math instruction tuning, LoRA forgot less than full fine-tuning, weight decay, or attention dropout, while full fine-tuning's weight changes were roughly 10 to 100 times higher rank than typical LoRA settings, so the updates that learned most on a distant domain also moved the base model furthest.^[47]

Role in model alignment

Post-training pipelines

Supervised fine-tuning is one stage in several post-training recipes for changing how a pretrained language model responds, not one laboratory’s fixed sequence. Early OpenAI work started from a pretrained model, fit a reward model to human comparisons, and optimized a policy with proximal policy optimization under a penalty on Kullback–Leibler divergence from that initialization.^[23] On summarization, Stiennon et al. made the supervised stage explicit: the model was fine-tuned to imitate reference summaries, comparisons trained a reward model, and reinforcement learning from human feedback optimized that reward while staying close to the supervised policy. Raters preferred those models both to the human references and to much larger supervised models. Reference likelihood and ROUGE were treated as rough proxies for quality.^[50] Ouyang et al. scaled a similar three-step procedure to a broad instruction distribution, producing InstructGPT from GPT-3. About 13,000 labeler demonstrations trained a supervised policy; rankings trained a reward model; and PPO then updated the supervised policy against that reward, with a per-token KL penalty from the supervised model. Comparison collection and reinforcement learning can be repeated as the policy moves.^[41] On held-out prompts, preferences rose in steps from a few-shot prompt, to supervised fine-tuning, to PPO. The 175-billion-parameter InstructGPT model was preferred to GPT-3 85 ± 3 percent of the time and to few-shot GPT-3 71 ± 4 percent of the time, while the paper’s main figure already places the 175-billion-parameter supervised model above plain and prompted GPT-3.^[41] The authors describe this as alignment to the stated preferences of their labelers and researchers, not to any broader notion of “human values.”^[41] A heavy demonstration fine-tune was not universal. In Anthropic’s helpful-and-harmless work, Bai et al. wrote that fine-tuning occurred “purely through RL” after context distillation of a helpful, harmless, and honest prompt, which they called much more like prompting than supervised training on demonstrations. Preference models provided the reward, policies and preference models were refreshed about weekly, and some data came from rejection sampling. Reward tracked approximately the square root of KL divergence from the initial policy.^[51] Supervised updates later returned as a way to absorb preferences before or instead of full reinforcement learning. Constitutional AI fine-tunes on a model’s critiques and revisions against written principles, then trains a preference model on model-written comparisons and optimizes with reinforcement learning from AI feedback.^[52] For Llama 2-Chat, Meta AI supervised-fine-tuned on 27,540 vendor annotations, favoring quality over volume, then iterated rejection-sampling fine-tuning, in which the highest-reward sample becomes a new supervised target, adding PPO in later rounds. Rejection sampling ran on the 70-billion-parameter model, and smaller models were fine-tuned on its selected outputs.^[53] Direct preference optimization drops the explicit reward model and the on-policy loop but keeps the same KL-constrained objective: Rafailov et al. use the supervised model as both reference and initialization, or, if it is unavailable, a maximum-likelihood fit on the preferred completions alone.^[54]

Scope of behavioral change

The same papers do not show that supervised fine-tuning installs a general set of values. In the InstructGPT evaluations it was a large preference step from GPT-3, and PPO a further step, on one group of labelers.^[41] The full InstructGPT models invented unsupported content about half as often as GPT-3 on closed-domain tasks (21 percent versus 41 percent) and were more often truthful and informative on TruthfulQA, with only small toxicity reductions and no clear gain on the reported bias probes.^[41] Their supervised baseline was the least toxic model in one rating but also the least preferred and least continuous, which the authors suggest may reflect short or degenerate answers.^[41] Labelers agreed on comparisons only about 73 percent of the time, and researchers wrote the instructions those labelers followed.^[41] On summarization, preference optimization beat imitation of reference summaries, so copying demonstrations was not the same as matching rater preferences.^[50] Whether the shift is mostly format is disputed. Zhou et al. stated a “superficial alignment hypothesis”: knowledge and capabilities are learned almost entirely in pretraining, while alignment teaches which subdistribution of formats to use with users. A 65-billion-parameter Llama model, LIMA, trained with ordinary supervised loss on 1,000 curated examples and no preference modeling was equivalent or strictly preferred to GPT-4 in 43 percent of cases in their study, to Bard in 58 percent, and to a human-feedback-trained Davinci003 in 65 percent.^[55] The paper also reports that curation was hard to scale, that an unlucky sample or adversarial prompt often produced a weak answer, and that with only 13 safety examples the model was safe on about 80 percent of 30 sensitive prompts but complied more often when harmful intent was implicit.^[55] Raghavendra, Nath, and Hendryx call the hypothesis at best an oversimplification. On Llama and Mistral models, a handful of supervised examples mainly fixed style on math and multihop questions without saturating scores. Performance instead followed a power law in the number of examples for mathematical reasoning, coding, instruction following, and multihop reasoning, and it tracked reasoning errors more than format. Post-training also improved use of knowledge beyond the pretraining corpus, and preference judgments could favor stylistically aligned but incorrect answers.^[56] On an arithmetic card game and a navigation task, in text and vision, Chu et al. found that outcome-based reinforcement learning generalized to unseen rules and images, while supervised fine-tuning tended to memorize and failed out of distribution. The supervised stage still mattered as a format stabilizer that made a later reinforcement-learning signal usable.^[57] The evidence supports supervised fine-tuning as a shift toward demonstrated formats and, with enough task data, higher scores on covered tasks, while transfer, adversarial robustness, and the added effect of preference optimization remain contested.

Evaluation

Automatic benchmarks

Supervised fine-tuning of a large language model is commonly scored on held-out suites with checkable answers. Wei et al. grouped more than 60 public NLP datasets into task clusters, instruction-tuned a 137-billion-parameter model while withholding a whole cluster, and tested zero-shot transfer to that unseen type. The model beat zero-shot GPT-3 on 20 of 25 datasets.^[3] Chung et al. scaled the mixture to 1.8K tasks and reported MMLU, BIG-Bench Hard, TyDiQA, and MGSM, with direct answers and chain-of-thought prompts. Flan-PaLM 540B gained 9.4 points on average over untuned PaLM 540B and reached 75.2% on five-shot MMLU. The reported checkpoint was selected from scores on those held-out tasks, so the number is not a fully untouched test.^[4] Closed-ended exams do not measure whether a free-form reply is useful. Zheng et al. found that MMLU and HELM can look similar for a base model and a conversation-tuned derivative even when people prefer the tuned model.^[58] Ouyang et al. treated public NLP sets as a side check. On TruthfulQA, InstructGPT gave truthful and informative answers about twice as often as GPT-3, and on closed-domain tasks it invented unsupported facts less often (21% versus 41%). Tuning with reinforcement learning from human feedback then lowered scores relative to GPT-3 on SQuAD, DROP, HellaSwag, and translation—an "alignment tax" mostly reduced by mixing in pre-training likelihood. On API prompts, labelers preferred the demonstration-trained model to copies tuned only on FLAN and T0: against that supervised baseline those variants won 29.8% and 26.8% of comparisons, while preference-tuned InstructGPT won 73.4%.^[5] IFEval checks constraints rather than facts. About 500 prompts combine 25 verifiable instructions, such as word counts, keywords, JSON, and fixed endings, and programs score prompt-level and instruction-level accuracy under strict and loose rules. The loose rule can mark a miss as a hit, and the benchmark does not grade whether the content is useful.^[59] Yang et al. showed that n-gram decontamination misses paraphrases, that fine-tuning a 13-billion-parameter model on rephrased MMLU, GSM8K, and HumanEval items can reach scores reported for GPT-4, and that both synthetic instruction data and pre-training collections such as RedPajama already contain benchmark text.^[60]

Preference judgments

Open-ended quality is measured by comparing answers. InstructGPT's primary metric was pairwise preference on held-out API prompts: the 175-billion-parameter model beat same-size GPT-3 85±3% of the time and few-shot GPT-3 71±4% of the time. Labelers excluded from data collection preferred it at a similar rate, but the study did not cover users who disagree about the right behavior.^[5] Zhou et al. used the same pairwise setup, with a tie option, for LIMA, a 65-billion-parameter Llama model trained with supervised loss on 1,000 demonstrations. On 300 prompts, crowd workers rated LIMA equivalent or better than GPT-4 in 43% of cases, Bard in 58%, and an RLHF-tuned model in 65%. Agreement, with ties discounted, was 82% between crowd workers and 78–79% between workers and GPT-4. The comparison still depends on the raters and on a single sample per prompt.^[8] Zheng et al. made the comparison reusable. MT-Bench has 80 two-turn questions; Chatbot Arena collects blind votes from people chatting with two anonymous models. A GPT-4 judge agreed with expert and crowd votes more than 80% of the time, about the human–human rate, yet showed clear biases. After a position swap it was consistent on near-duplicate answers only 65% of the time and favored the first answer 30% of the time. Padding an answer with a repeated list fooled GPT-3.5 and Claude-v1 on 91.3% of items and GPT-4 on 8.7%. Evidence that a judge prefers its own outputs was inconclusive. On math, the judge endorsed a wrong solution it could have solved alone; a reference answer cut that failure rate from 14 of 20 to 3 of 20.^[58] AlpacaEval uses an LLM judge against a fixed baseline. Dubois et al. found a length bias, and estimating the preference at equal length raised Spearman correlation with Chatbot Arena from 0.94 to 0.98.^[61]

Limitations and criticism

Overfitting and capability loss

In the InstructGPT pipeline, OpenAI fine-tuned GPT-3 for 16 epochs on labeler demonstrations and found that validation loss overfit after about one epoch; further epochs still raised reward-model scores and human preference ratings, so the checkpoint was chosen by reward score rather than by that loss. ^[5] A regression the paper called an "alignment tax" appeared during later reinforcement learning from human feedback, not in that supervised stage: against GPT-3, performance fell on SQuAD, DROP, HellaSwag, and WMT 2015 French-to-English translation. ^[5] Mixing policy updates with likelihood updates on the pretraining distribution, called PPO-ptx, reduced those drops without lowering labeler preference scores. ^[5] Catastrophic forgetting has been measured when instruction tuning continues across tasks. Luo et al. reported losses of domain knowledge, reasoning, and reading comprehension in large language models from about 1 billion to 7 billion parameters, with relatively more severe forgetting as scale increased inside that range, and milder forgetting for decoder-only BLOOMZ models than for encoder-decoder mT0 models. ^[62] Zhou et al. stated a "superficial alignment hypothesis": capabilities are learned almost entirely in pretraining, while alignment mainly selects response formats. ^[8] Their LIMA model, a 65-billion-parameter Llama system trained with a supervised loss on 1,000 curated pairs and no preference model, was rated equivalent or preferable to GPT-4 in 43% of a controlled comparison. ^[8] Gudibande et al. found a similar split in imitation: crowdworkers rated about 70% of outputs from open models fine-tuned on ChatGPT transcripts as equal to or better than ChatGPT, but Natural Questions, MMLU, and HumanEval closed little of the gap, which the authors attributed to copying style, including a confident tone, rather than factuality. ^[33]

Behavioral failure modes

In closed-book question answering, Gekhman et al. found that supervised examples stating unknown facts were learned much more slowly than examples consistent with prior knowledge and, once learned, increased hallucination on that prior knowledge roughly linearly, which they read as evidence that fine-tuning uses pretraining knowledge more reliably than it installs new facts. ^[63] Sharma et al. found sycophancy—answers bent toward a user's stated view—already present at the start of reinforcement learning for Anthropic's Claude 2, evidence in their view that pretraining and supervised fine-tuning contribute, and reported that some forms increased during reinforcement learning. ^[64] The preference model sometimes ranked sycophantic answers above more truthful ones, so optimizing against it could trade accuracy for agreement. ^[64] Röttger et al. introduced XSTest, 250 safe prompts that share wording or topics with harm and 200 unsafe contrasts, to measure "exaggerated safety": refusal of benign prompts after training aimed at both helpfulness and harmlessness. ^[65] Knowledge distillation onto a stronger model's sampled answers can copy the teacher's errors as well as its manner; Gudibande et al. treated high subjective ratings alongside flat factual benchmarks as a reason not to take those transcripts as a substitute for a stronger base model. ^[33]

Labor, copyright, and licensing

In 2023, Time reported that OpenAI contracted Sama for workers in Nairobi, Kenya, to label sexual-abuse, hate-speech, and violence text for a toxicity detector later used with ChatGPT—post-training safety labor, not a practice unique to supervised demonstrations. ^[66] Using payslips and contracts, Time reported take-home pay of about $1.32 to $2 an hour on work worth about $200,000, and workers described mental distress with rare counseling. ^[66] Sama disputed the quota and said after-tax pay could be about $1.46 to $3.74 an hour; OpenAI confirmed the safety use, the contract ended early in 2022, and OpenAI said about $150,000 was paid. ^[66] A 2023 Washington Post investigation described contractors on Scale AI's Remotasks platform in the Philippines who annotated data for artificial-intelligence clients and reported delayed, reduced, or canceled pay. ^[67] In 2024, Wired reported a letter from 97 African workers, most of them Kenyan, telling the U.S. president that practices at Meta Platforms, OpenAI, and Scale AI "amount to modern day slavery," including graphic review often under $2 an hour and a Remotasks ban in Kenya, Nigeria, and Pakistan that the letter said left wages unpaid; the companies had not immediately commented. ^[68] Scale AI told Rest of World that the cutoff followed enhanced security protocols and that missing notice was an administrative error. ^[69] Terms of use also limit whose model outputs may become supervised targets. Stanford University's Alpaca model was fine-tuned on 52,000 demonstrations from OpenAI's text-davinci-003 and was barred from commercial use because the Llama license was non-commercial, because those API terms prohibited competing models, and because the authors judged its safety measures inadequate. ^[26] Databricks later released Dolly 2.0, fine-tuned from an EleutherAI Pythia model on about 15,000 employee-written pairs under a Creative Commons Attribution-ShareAlike 3.0 license, so that training would not depend on ChatGPT outputs. ^[29] Anthropic's usage policy prohibits using Claude outputs to train a model, including by scraping or distillation, without prior authorization. ^[70] Google's terms prohibit using AI-generated content from its services to develop machine-learning models. ^[71] License fields on the collections themselves are often incomplete. The Data Provenance Initiative traced more than 1,800 text datasets in popular instruction and alignment collections and reported that more than 70% of licenses on GitHub and Hugging Face were unspecified, with reannotation frequently disagreeing with the original authors. ^[72] Sets classed as closed to commercial use disproportionately covered lower-resource languages, creative tasks, and newer synthetic text. ^[72]

Applications

Specialized language models

Supervised fine-tuning specializes general large language models, but “fine-tuning” in a code paper is not always training on labeled pairs. Code Llama continues Llama 2 training on code; its 7B, 13B, and 34B Instruct variants then train on about 5 billion further tokens of dialogue, mixing Llama 2 supervised and rejection-sampled conversations with programming problems whose solutions are kept only if they pass execution.^[73] StarCoder’s Python variant is continued causal language modeling of StarCoderBase on about 35 billion Python tokens, not instruction–response pairs.^[74] Magicoder does use such pairs: it fine-tunes code models, including Code Llama–Python 7B, for two epochs on about 75,000 synthetic instruction–solution examples from OSS-Instruct, which seeds a generator with open-source snippets.^[75] Mathematical specialization often labels a chain-of-thought trace as well as the answer. Rejection-sampling fine-tuning draws many reasoning paths from a supervised model, drops those with wrong answers, and fine-tunes on the rest; pooling samples from several models moved a LLaMA-7B GSM8K score from 35.9% under ordinary supervised fine-tuning to 49.3%.^[76] MetaMath builds MetaMathQA by rewriting questions forward and backward and attaching rejection-sampled paths, then fine-tunes Llama 2; the 7B model is reported at 66.5% on GSM8K and 19.8% on MATH.^[77] STaR repeats a similar filter—writing rationales, retrying failures with the correct answer as a hint, and fine-tuning the original checkpoint only on rationales that yield the right answer.^[78] Domain adaptation usually adds a supervised stage after continued pretraining. MEDITRON extends Llama 2 pretraining with PubMed articles, abstracts, and clinical guidelines, then fine-tunes on medical question-answering splits with the loss restricted to response tokens; the authors report further gains from that step and advise against clinical use without more adaptation.^[79] SaulLM-7B continues pretraining from Mistral 7B on more than 30 billion tokens of English legal text, then instruction-tunes on supervised dialogues that mix generic coding, mathematics, and chat with legal question answering and summarization.^[80]

Other modalities

The older use is supervised transfer learning in vision: a network trained with a classification loss on ImageNet is copied onto a new labeled task and updated. Yosinski and colleagues found that fine-tuning transferred layers of a convolutional neural network improved generalization over training from scratch, even though higher layers are more task-specific than early filters.^[16] Kornblith, Shlens, and Le fine-tuned 16 ImageNet architectures on 12 datasets and reported a 0.96 correlation between ImageNet top-1 accuracy and fine-tuned accuracy, with full fine-tuning ahead of frozen features on most pairs.^[81] In speech, Whisper is an encoder–decoder trained to predict transcripts and related targets from about 680,000 hours of audio–text pairs; Radford and colleagues emphasized zero-shot speech recognition without dataset-specific fine-tuning.^[82] Later work still adapts those checkpoints on smaller labeled sets. Liu, Yang, and Qu compared five strategies on seven low-resource FLEURS languages and found that each reduced error relative to the untuned model, with full and parameter-efficient updates differing in accuracy and in weights changed.^[83] The same answer-only loss is used for image-conditioned dialogue. LLaVA links a frozen CLIP vision encoder to Vicuna with a learned projection and, holding the vision encoder fixed, fine-tunes the projection and language model on 158,000 image–instruction samples—conversations, descriptions, and reasoning items—written by a text-only model from captions and boxes, with an autoregressive loss on assistant tokens.^[84] A separate run turns ScienceQA items into single-turn visual instructions and fine-tunes on that split.^[84]