AI Model Distillation Explained: How Smaller Models Learn From Larger Models

Last updated: July 22, 2026 · Research cutoff: July 22, 2026

When Moonshot AI introduced Kimi K3 in July 2026, the model’s performance, price, and planned open-weight release immediately revived a sharp question: had a smaller or more efficient system learned from a closed frontier model such as Claude Fable 5? Anthropic had already accused Moonshot of a large, earlier campaign to extract Claude capabilities. Online anecdotes then encouraged a much more specific story—that K3 had been distilled from Fable 5.

The public evidence does not establish that specific claim. Anthropic’s documented allegation predates both Fable 5 and Kimi K3, Moonshot has not disclosed Claude-derived training data, and a model identifying itself as another model is not reliable evidence of provenance. But the controversy is useful because it exposes how loosely the word distillation is used. It can describe routine model compression, licensed synthetic-data generation, internal product engineering, black-box behavioral imitation, or an alleged extraction attack. Those are technically related but ethically and legally different.

The important distinction is not whether one model learned from another. Almost every modern AI system learns from other systems somewhere in its pipeline. The harder question is how, at what scale, and under what permissions.

This is Kingy.ai’s technical field guide to that harder question. Readers looking for a shorter conceptual overview can begin with our introductory guide to AI distillation. Here we go further into the training objectives, dataset pipeline, reasoning-trace debate, transfer limits, economics, security, and evidence standards that separate a useful explanation from a slogan.

1. Direct definition

AI model distillation is supervised transfer from a teacher model to a student model. Instead of learning only from human-provided labels or raw data, the student also learns from information the teacher produces. In classical knowledge distillation, that information is often a probability distribution. In modern LLM distillation, it may be a set of generated answers, complete sequences, critiques, preferences, reasoning-like traces, or tool calls.

A compact mathematical version looks like this:

Distillation loss = α · T² · KL(p_teacher^(T) || p_student^(T)) + (1 − α) · standard label loss

Here, both p_teacher^(T) and p_student^(T) are probability distributions softened at temperature T; α is the weight on the distillation term, while 1 − α weights the ground-truth label term. In normal language: one part of training rewards the student for agreeing with the teacher’s full ranking of possibilities; another part rewards it for matching known correct answers. A higher temperature makes low-probability alternatives more visible. The student can then learn not only that “cat” is correct, but that “fox” is more plausible than “truck.”

That formulation comes from the classical classifier setting popularized by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. LLM practice is broader. When the teacher is a hosted API and raw logits are unavailable, developers commonly train on sampled text. That is better described as sequence-level, behavioral, or black-box distillation than as pure logit matching.

2. Model distillation in plain English

Suppose a frontier model is excellent at handling customer questions, but each request is expensive and slow. A company collects representative support cases, asks the large model to classify and answer them, checks the answers, and uses the approved examples to train a smaller model. The smaller model will not become the frontier model in miniature. It may, however, become very good at the portion of the job represented in its training data.

Distillation therefore compresses a behavioral function, not a file. The student is learning a mapping from inputs to useful outputs under a chosen distribution. If the training set covers refund questions, the student may learn refunds. If it lacks angry customers, multilingual requests, fraud cases, or ambiguous policies, the student does not receive those abilities by osmosis.

This explains both the appeal and the limits. A student can be remarkably capable inside a well-measured lane and brittle outside it.

3. The teacher-student analogy

The analogy works if we use it carefully. A teacher does not pour a substance called knowledge into an empty student. The teacher creates signals: examples, corrections, rankings, demonstrations, critiques, and practice problems. The student changes through training on those signals.

In AI:

• The teacher is the model or ensemble producing the training signal.
• The student is the model whose weights are updated.
• The transfer set is the collection of inputs used to elicit or compare teacher behavior.
• The objective decides which aspects of the teacher the student is rewarded for matching.
• The evaluation set tests whether the student learned the job rather than memorized the lessons.

The teacher does not have to be larger. It can be an ensemble, a slower reasoning model, a tool-augmented system, a model with privileged data, or a collection of experts. What matters is that its outputs contain useful structure the student does not yet express reliably.

4. How the process works step by step

1. Define the deployment target. Specify the task, latency, cost, hardware, safety, privacy, context, and accuracy requirements. “Make it smaller” is not an evaluation plan.
2. Choose a teacher legally and technically fit for use. Check ownership, license terms, API terms, geography, privacy, and whether the desired signals are accessible.
3. Choose the student. Its base pretraining, architecture, tokenizer, parameter count, context window, modality support, and license set the ceiling for transfer.
4. Design the prompt or input distribution. Include normal traffic, rare cases, adversarial cases, different formats, and the situations where the student should escalate.
5. Collect teacher signals. Depending on access, these may be logits, labels, sequences, rankings, critiques, traces, tool calls, or synthetic tasks.
6. Filter, verify, and deduplicate. Remove wrong, unsafe, repetitive, contaminated, private, malformed, or unverifiable examples.
7. Train the student. Possible methods include cross-entropy on teacher sequences, KL divergence on distributions, representation matching, preference optimization, or a mixture.
8. Evaluate independently. Compare with ground truth and real tasks—not merely teacher agreement. A student can faithfully inherit a teacher’s mistake.
9. Deploy with routing and monitoring. Let difficult or high-risk requests fall back to a stronger model or a human.

For autoregressive models, a further distinction matters. Off-policy distillation trains on teacher-generated sequences. On-policy distillation lets the student generate its own sequences and then obtains teacher feedback on the states the student actually visits. On-Policy Distillation of Language Models, which introduces generalized knowledge distillation (GKD), argues that the latter can reduce the mismatch between the polished teacher examples seen during training and the imperfect prefixes the student produces in deployment.

Researchers also disagree about the divergence to optimize. Classical forward KL, KL(teacher || student), encourages the student to cover the teacher’s probability mass but can spend capacity on low-probability modes. Reverse KL, KL(student || teacher), emphasizes outputs the student actually produces and can yield sharper behavior, but it can ignore valid modes. MiniLLM advocates reverse-KL-based distillation for generative language models, while GKD treats the divergence and sampling policy as design choices. There is no universally best direction: task diversity, student capacity and deployment distribution determine which failure matters more.

5. Main types of model distillation

White-box distillation

The developer owns or is authorized to inspect the teacher. Training can use logits, hidden states, attention maps, layer relationships, or intermediate features. White-box methods can transfer more information per example, but student and teacher architectures may need alignment layers or carefully chosen matching points.

Black-box distillation

The teacher is queried like a service. The developer sees prompts and outputs, perhaps with rankings or log probabilities if the API provides them. This is common in LLM product work because closed models do not expose internal activations. It is also where distillation overlaps most strongly with contract restrictions and model-extraction security.

Sequence-level distillation

Kim and Rush’s 2016 machine-translation paper trained students on complete sequences decoded by a teacher. Their reported translation student ran about ten times faster with little performance loss in that particular recurrent-model setting. The historical result is important, but it is not a promise that every modern LLM student will preserve capability at the same ratio.

Synthetic-data distillation

A teacher generates tasks, examples, counterexamples, critiques, or structured outputs, and the student is fine-tuned on the filtered dataset. If the teacher is not supplying a comparative or privileged signal, some researchers would call this synthetic-data training rather than classical knowledge distillation. The boundary is functional, not ceremonial.

6. Distillation versus fine-tuning

Fine-tuning means updating a model’s weights using training examples. Distillation describes where an important part of those training targets came from: another model.

A model can be fine-tuned on human-written examples with no teacher. A distilled student is often implemented through supervised fine-tuning on teacher-generated examples. The concepts overlap, but they answer different questions:

• Fine-tuning asks: how are we adapting this model?
• Distillation asks: what teacher signal are we transferring?

Instruction tuning is not automatically distillation either. FLAN-style instruction tuning used a broad task mixture to improve zero-shot generalization. Self-Instruct then showed how a model could generate and filter tens of thousands of instructions for later fine-tuning. When those examples come from a stronger teacher and are meant to transfer its behavior, the workflow becomes a form of behavioral distillation.

For a deployment-oriented look at base-model selection, adapters and fine-tuning, see Kingy.ai’s guide to owning and fine-tuning an open-model stack.

7. Distillation versus quantization

Quantization stores weights or activations at lower numerical precision. A 16-bit value might be represented with 8, 4, or fewer bits. This usually reduces memory and can improve speed, although quality may fall.

Quantization normally keeps the same basic model and approximates its numbers. Distillation trains a separate student. Teams often combine them: distill a 70-billion-parameter teacher into an 8-billion-parameter student, then quantize the student for a laptop or edge device.

For hardware planning, see Kingy.ai’s local AI models guide, which explains model formats, memory ceilings and practical quantization levels.

8. Distillation versus pruning

Pruning removes weights, neurons, attention heads, experts, channels, or other structures judged unnecessary. It reduces capacity in an existing network. Distillation builds or retrains a student to reproduce selected behavior.

Pruning can be unstructured, removing individual weights, or structured, removing blocks that hardware can skip efficiently. A pruned model may then be fine-tuned or distilled to recover lost performance. Again, the techniques are complementary rather than mutually exclusive.

9. Distillation versus model extraction

Distillation is a broad training method. Model extraction is a security term for learning a useful approximation of a target model through access to its behavior or interface, often without the owner’s cooperation.

The algorithms may overlap. Authorization and intent are what change the framing. A company compressing its own model is distilling. A researcher testing extraction defenses under permission may use the same techniques. A competitor using fake accounts and proxies to reconstruct proprietary capability is a different security and contract problem.

Research has progressively demonstrated stronger forms of extraction. Tramèr and colleagues showed in 2016 that prediction APIs could reveal enough behavior to reproduce several model classes. Orekondy and colleagues’ 2019 “Knockoff Nets” work trained image models from input-prediction pairs. In 2024, Carlini and colleagues recovered a production language model’s embedding projection layer up to mathematical symmetries through API queries. These are related risks, but behavioral imitation, parameter recovery and training-data extraction are not the same attack.

10. Distillation versus copying weights

Model weights are the learned parameter files produced by training. Stealing or copying them gives the recipient the teacher artifact itself, subject to architecture and runtime compatibility.

Black-box distillation does not copy those weights. It trains new student weights from observed signals. The student can use a different architecture, tokenizer, parameter count, data mixture and alignment process. Similar behavior is therefore not proof of shared weights.

This distinction matters for both journalism and law. “The model copied Claude’s weights” is a much stronger factual claim than “the model may have trained on Claude outputs.” Evidence for one does not establish the other.

11. Why reasoning traces matter

Final answers are thin supervision. If the correct answer is 42, the student learns that 42 should follow a prompt. A worked solution can add decomposition, intermediate checks, tool choices, error recovery and the boundary between plausible and valid reasoning.

That richer signal is why reasoning distillation became central to math, code and agent training. In Distilling Step-by-Step, teacher-generated rationales were used as an auxiliary target. A task-specific 770-million-parameter T5 student outperformed a few-shot 540-billion-parameter PaLM baseline on one reported benchmark while using less labeled data. The comparison is real but narrow: the student was specialized, the teacher was general-purpose and few-shot prompted, and the result does not mean the student became more generally capable.

DeepSeek-R1 later made the pattern visible at LLM scale. Its report describes 800,000 curated samples used to supervised-fine-tune smaller Qwen- and Llama-family models. Several students performed strongly on targeted math and coding benchmarks. They did not surpass DeepSeek-R1 overall.

There is also a conceptual trap. A written chain of thought is not necessarily a transcript of the model’s causal computation. Research by Turpin and colleagues showed that explanations can rationalize answers influenced by hidden biases. A trace can still be useful training data, but developers should treat it as a candidate solution to verify, not a window into the teacher’s mind.

12. Can final answers alone transfer intelligence?

They can transfer useful behavior, especially when tasks are narrow and answers are informative. A student can learn tone, formatting, common classifications, code idioms and standard procedures from final outputs.

But final answers discard information. They do not reveal the teacher’s uncertainty, rejected alternatives, intermediate subgoals, tool failures, or why one solution is safer than another. Multiple very different reasoning processes can produce the same answer.

This helps explain the findings of The False Promise of Imitating Proprietary LLMs. Across base models from 1.5 to 13 billion parameters and imitation corpora from roughly 0.3 million to 150 million tokens, human raters observed improved style and instruction-following, but targeted evaluations found little closure of capability gaps where the imitation data did not cover the tested skill. Copying the surface is easier than copying broad competence.

Better transfer can come from softer probability targets, multiple sampled solutions, teacher critiques, verifiers, preference pairs, counterexamples, tool trajectories and on-policy feedback. None eliminates the need for a capable base student.

13. How synthetic datasets are created

Synthetic-data distillation is a production pipeline, not a single prompt.

1. Seed tasks: start with real prompts, domain documents, human-authored tasks or a small set of demonstrations.
2. Task expansion: ask a teacher to create variations across difficulty, language, persona, format and edge case.
3. Response generation: sample one or more answers, solutions, critiques, labels or tool trajectories.
4. Independent checking: run unit tests, calculators, symbolic solvers, retrieval, rule checks, other models or humans.
5. Selection: keep examples that pass correctness, diversity, safety, provenance and format thresholds.
6. Balancing: control task mix so easy, frequent or stylistically similar examples do not swamp the tail.
7. Student training: fine-tune in stages, often with curriculum ordering or preference optimization.
8. Held-out evaluation: test on real prompts and data that were never shown to the teacher-generation pipeline.

Self-Instruct offers a useful historical scale point: it began with 175 human-written seed tasks, then generated and filtered 52,445 instructions and 82,439 instances. Microsoft’s phi-1 work combined selected real code data with one billion tokens of GPT-3.5-generated textbooks and exercises. Neither paper implies that arbitrary synthetic text is a substitute for broad, high-quality pretraining.

A narrower research example appears in Kingy.ai’s summary of training evaluators with synthetic preference data. For the broader model-building lifecycle, our review of a modern LLM training playbook provides useful context.

14. How outputs are filtered and scored

The dataset filter is often more important than the generator. Common gates include:

• Correctness: exact answers, executable tests, factual retrieval, calculators, solvers or expert review.
• Format validity: parseable JSON, compilable code, schema compliance and complete tool arguments.
• Safety: policy compliance, privacy screening, secret removal and harmful-capability controls.
• Provenance: teacher, model version, prompt template, date, temperature, source documents and license or permission basis.
• Diversity: semantic deduplication, task balance, difficulty balance and reduction of repetitive phrasing.
• Consistency: agreement across repeated samples, critics, judges or deterministic checks.
• Value: remove trivial examples the student already solves and ambiguous examples with no stable target.

Rejection sampling generates several candidates and keeps only those that pass a reward model, verifier, test suite or human rubric. This can improve average quality while narrowing diversity, so selection thresholds should be monitored. A verifier can also be wrong or exploitable; “passed the judge” is not equivalent to “correct.”

Model-collapse research adds another warning. Shumailov and colleagues showed that recursively replacing real data with generated data can erase distribution tails and create drift. That does not mean all synthetic data is harmful. It means teams should preserve high-quality real anchors, provenance and diversity rather than letting models repeatedly train on untraceable descendants of their own output.

15. What capabilities transfer well

Distillation is strongest when the target behavior is observable, well represented and easy to verify:

• classification, routing and extraction with stable labels;
• structured formats such as JSON, SQL templates and function-call schemas;
• tone, policy phrasing and repetitive support workflows;
• coding tasks with comprehensive tests;
• math problems with exact or independently checkable answers;
• tool selection and argument construction within a bounded system;
• domain behavior already supported by the student’s pretraining;
• narrow latency-sensitive jobs where hard cases can escalate.

The teacher can also reduce label ambiguity. Its probability distribution may show that two classes are related even when the training set supplies a single hard label.

16. What capabilities do not transfer easily

• Broad general intelligence: a finite imitation corpus samples only a small part of a frontier model’s behavior.
• Rare knowledge and distribution tails: generation and filtering tend to overproduce common, clean examples.
• Robust reasoning outside the sampled curriculum: the student may memorize solution shapes rather than acquire reusable algorithms.
• Calibration: matching answers does not guarantee matching uncertainty or knowing when to abstain.
• Safety boundaries: a student can inherit capability without the surrounding monitoring, product controls and refusal behavior.
• Long-horizon agency: small errors compound across many tool calls, files and decisions.
• Multimodal perception: text-only teacher outputs cannot supply raw visual or audio representations the student architecture lacks.
• Fresh facts: distillation bakes behavior into weights. Retrieval is usually better for information that changes.

A weak student architecture is another hard limit. More teacher data cannot make a model represent computations it lacks the capacity or context to execute reliably.

17. Can students outperform teachers?

Yes, on a defined benchmark or deployment task. That does not mean the student is generally smarter.

A student can beat the measured teacher because it is specialized, sees more examples from the target distribution, combines teacher targets with ground truth, benefits from regularization, starts from strong independent pretraining, uses a better decoder, or receives more test-time compute. The teacher may also be evaluated zero-shot while the student is repeatedly trained on the task.

The fair statement is: a distilled student can outperform its teacher under specific evaluation conditions. Always name the task, teacher configuration, student training, data, decoding, compute and metric. “The student surpassed the teacher” without those details invites a false general conclusion.

18. Real-world examples

Simple hypothetical example: customer support

A large language model labels 100,000 historical support tickets, drafts answers from current policy documents and marks escalation triggers. Humans review high-risk categories and a statistically meaningful sample. A smaller student learns intent classification, routine answers and when to route billing, legal, safety or angry-customer cases. The student does not replace the knowledge base or the escalation team. Its value is cheap, fast handling of the common path.

Intermediate hypothetical example: coding assistant

A strong coding teacher receives repository tasks and generates several patches for each. The pipeline runs unit tests, integration tests, static analysis and security checks. It rejects broken patches, keeps multiple valid strategies and records the teacher’s critiques. A smaller coding assistant trains on the verified prompt-patch-test trajectories. Evaluation uses private repositories and tasks not present in generation. The tests, not the teacher’s confidence, decide whether code is correct.

Advanced hypothetical example: post-training a mixture-of-experts model

A pretrained sparse mixture-of-experts student already possesses broad language, code and multimodal representations. Post-training combines four teacher-generated streams: verified reasoning solutions, chosen/rejected preference pairs, tool-use trajectories with observations, and adversarial safety examples. Rejection sampling uses tests, retrieval, policy rules and domain experts. Training begins with supervised examples, continues with preference optimization and then adds on-policy feedback for states the student itself generates. Router balance, activated-expert capacity and long-context failures are evaluated separately.

This example is hypothetical. It describes a plausible advanced recipe, not a claim about Kimi K3 or any other named model.

Published case studies

• Model Compression (2006): Buciluǎ, Caruana and Niculescu-Mizil used pseudo-data labeled by an ensemble to train a compact model, predating the modern “knowledge distillation” label.
• Hinton, Vinyals and Dean (2015): formalized the now-standard soft-target and temperature framing.
• Sequence-Level Knowledge Distillation (2016): trained translation students on full teacher-decoded sequences.
• Distilling Step-by-Step (2023): used rationales as auxiliary supervision for smaller task-specific students.
• DeepSeek-R1 distill models (2025): showed how a large curated reasoning dataset could improve smaller open model families on targeted benchmarks.

19. Cost and infrastructure implications

Distillation has two bills. The first is the one-time or periodic cost of generating data, filtering it, training the student and running evaluations. The second is ongoing inference. A project makes economic sense when the inference savings, latency gains, privacy benefits or deployment reach justify the training and maintenance cost.

High-volume narrow workflows are the clearest case. If a frontier API call costs only cents but runs millions of times, the recurring bill can dominate. A smaller self-hosted model can reduce per-request cost, but it introduces GPU utilization, serving, monitoring, model updates, security and staffing costs. “Cheaper tokens” do not automatically mean a cheaper system. Kingy.ai’s token-budgeting and model-selection guide shows how to compare inference choices at the workflow level.

Moonshot’s Kimi K3 illustrates a different scale lesson. Its own deployment guidance recommends supernode configurations with at least 64 accelerators despite sparse expert activation. A model can be efficient relative to its total parameter count and still require substantial infrastructure. For broader deployment economics, see Kingy.ai’s analysis of open-weight models and AI capital spending and our Kimi K3 open-weight economics deep dive.

20. Security risks

For a model provider, a high-volume API can become a behavioral side channel. Attackers can use many accounts, varied prompts and repeated sampling to map valuable capabilities. They may target code, tool use, safety boundaries, vision, refusal behavior or reasoning-like explanations.

Anthropic says it observed more than 3.4 million Claude exchanges associated with Moonshot across hundreds of fraudulent accounts, and that later traffic tried to reconstruct reasoning traces. Those are Anthropic’s allegations and attribution conclusions; the company has not published the underlying logs for independent audit.

Defenses include identity verification, rate limits, cross-account coordination detection, behavioral fingerprints, canary prompts, abuse monitoring, output controls and contractual enforcement. These measures create a difficult product trade-off: restrictions that frustrate extraction can also reduce legitimate research and customer utility.

For the student operator, risk runs in the opposite direction. A distilled model may preserve capability without the teacher’s safeguards, monitoring, retrieval, routing, or product-level controls. Safety must be trained and evaluated as its own capability.

21. Legal questions

Distillation is not inherently unlawful. The legal risk turns on what was collected, how access was obtained, what permissions applied, whether protected expression or trade secrets were taken, and whether security controls or regional restrictions were evaded.

In many commercial disputes, the first claim may be breach of contract. Anthropic’s commercial terms restrict using its services to build a competing product or train a competing model without approval. Output ownership and permitted use are separate questions: a provider can assign rights it has in an output while still contractually restricting competitive training.

Copyright, trade-secret, anti-circumvention and computer-access claims require additional facts. Capability, ideas, methods, facts and style are not automatically protected expression. Copying weights, memorized passages, confidential traces or proprietary code is different from learning a functional pattern. The U.S. trade-secret definition in 18 U.S.C. §1839 excludes reverse engineering and independent derivation from “improper means,” while deception, breached duties or electronic espionage can support a claim. A terms violation alone is not automatically a computer crime: the Computer Fraud and Abuse Act is codified at 18 U.S.C. §1030, and the U.S. Supreme Court interpreted its “exceeds authorized access” language narrowly in Van Buren v. United States.

This is general information, not legal advice. A separate Kingy.ai legal guide will examine jurisdictions and fact patterns in more depth.

22. Open-weight and closed-model implications

Open weights change access, not every permission. A downloadable model can allow researchers to use logits and hidden states, inspect architecture, run white-box experiments and deploy locally. Its license may still restrict use, redistribution, naming, acceptable use or downstream derivatives. “Open-weight” is not synonymous with public domain or OSI-approved open source.

Closed models concentrate the best observable signals behind an API. That makes output terms, rate limits and detection central. It also encourages black-box research because developers cannot inspect internal representations.

Distillation makes open-weight ecosystems strategically important. A community can use a capable open teacher to produce smaller models for languages, devices and domains the original lab did not prioritize. It can also magnify provenance problems if synthetic datasets circulate without clear teacher, license or filtering records.

Kingy.ai’s guide to open-source and open-weight AI models explains the licensing and deployment distinction, while our AI sovereignty guide covers the hardware and control trade-offs.

23. What the Kimi K3 controversy demonstrates

Three dates prevent the story from collapsing into a claim the public evidence cannot support:

• February 23, 2026: Anthropic published its allegation that Moonshot had generated more than 3.4 million Claude exchanges targeting agentic reasoning, coding, data analysis, computer use, vision and later reasoning traces.
• June 9, 2026: Anthropic introduced Claude Fable 5.
• July 16, 2026: Moonshot introduced Kimi K3 and said full weights and a technical report would follow.

The February allegation cannot by itself prove a Fable 5-to-K3 pipeline because Fable 5 had not been released. Later anecdotes and claims may justify questions, but benchmark proximity, style similarity and self-identification are not provenance evidence. Models can inherit names from training data, synthetic conversations, prompt contamination or hallucination.

Moonshot’s own launch post says K3 is a 2.8-trillion-parameter sparse mixture-of-experts model with native vision, a one-million-token context window, 16 of 896 experts activated, and quantization-aware training from the supervised-fine-tuning stage. It also says K3 trails Fable 5 and GPT-5.6 Sol overall in Moonshot’s evaluation suite. None of those statements discloses whether Claude outputs were used in K3 training.

The controversy demonstrates five durable lessons:

1. Behavior is evidence of capability, not provenance.
2. Authorized distillation and alleged extraction attacks can use similar learning methods.
3. Closed-model providers increasingly treat output access as a security boundary.
4. Open-weight releases raise the stakes because a successful student can be redistributed and studied.
5. Technical reporting needs chronologies and uncertainty labels, not theft metaphors standing in for proof.

For Kingy.ai’s direct model coverage, see our independent Kimi K3 benchmark analysis and Claude Fable 5’s benchmark and safeguard profile. Those pages document capabilities and positioning; neither proves training provenance.

24. Frequently asked questions

What is AI model distillation?

AI model distillation is training a student model using signals produced by a teacher model. The signals can be probabilities, answers, generated examples, preferences, hidden features, reasoning-like traces or tool-use trajectories.

What is a teacher model?

A teacher model produces the training signal. It is often larger or more capable, but it can also be an ensemble, a tool-augmented system, a model with privileged data or a slower specialist.

What is a student model?

A student is the model whose weights are updated to learn selected teacher behavior. It may use a different architecture and is usually optimized for cost, speed, size or specialization.

Is LLM distillation the same as fine-tuning?

No. Fine-tuning is a method for adapting weights or adapters. Distillation describes teacher-to-student transfer. Many LLM distillation projects use supervised fine-tuning on teacher-generated examples.

Is distillation the same as quantization?

No. Quantization lowers numerical precision in a model. Distillation trains a separate student. A distilled student can later be quantized.

What is black-box distillation?

Black-box distillation uses observable teacher inputs and outputs without access to weights or internal activations. Hosted LLM APIs are common black-box teachers.

What is white-box distillation?

White-box distillation has authorized access to internal signals such as logits, hidden states or attention patterns. It can transfer more information per example than final outputs alone.

How much training data is needed?

There is no universal number. Requirements depend on task breadth, student pretraining and capacity, teacher quality, data diversity, filtering, objective and evaluation distribution. Published projects range from tens of thousands of instructions to hundreds of thousands of curated traces, while broad imitation can consume far more data and still miss capabilities.

Can a student model become better than its teacher?

It can outperform the measured teacher on a narrow task or benchmark, especially when specialized and trained with ground truth plus teacher signals. That does not imply greater general capability.

Does distillation copy weights?

Not in ordinary black-box or response-based distillation. It creates new student weights. Directly copying the teacher’s parameter files is weight copying or theft, not distillation.

Is model distillation legal?

Often, but not automatically. Ownership, licenses, service terms, access methods, protected expression, trade secrets, privacy, jurisdiction and evasion of controls can change the answer.

Can final answers transfer reasoning?

They can transfer some task behavior, but they omit uncertainty, intermediate checks and rejected paths. Verified worked solutions, multiple candidates, critiques and on-policy feedback can provide richer supervision.

Why is distillation strategically important?

It can turn expensive frontier capability into cheaper specialized systems, improve local and edge deployment, reduce dependence on APIs, and spread capabilities across open ecosystems. It also creates model-extraction and safety risks.

25. Glossary

Black-box distillation

Training from observable inputs and outputs without teacher internals.

Chain of thought

A written sequence of intermediate reasoning-like steps; useful as supervision but not guaranteed to reveal internal computation.

Dark knowledge

Information in a teacher’s relative probabilities over wrong or less likely alternatives.

Distillation loss

The training objective that rewards student agreement with teacher signals, often combined with ground-truth loss.

Feature distillation

Matching intermediate representations rather than only outputs.

KL divergence

A measure of difference between probability distributions, commonly used in distillation objectives.

Logits

Raw model scores before conversion into probabilities.

Model extraction

Reproducing target behavior or recovering model properties through access or queries, often studied as a security problem.

On-policy distillation

Teacher feedback on sequences or states produced by the student itself.

Post-training

Training after broad pretraining, including instruction tuning, preference optimization, safety tuning and distillation.

Rejection sampling

Generating multiple candidates and retaining only those that pass a verifier or quality threshold.

Sequence-level distillation

Training on complete sequences generated by a teacher.

Soft label

A probability distribution over alternatives rather than one hard correct class.

Student model

The model updated to learn from teacher signals.

Synthetic data

Training examples generated or transformed by a model or program rather than collected directly from the target environment.

Teacher model

The model or system producing the privileged training signal.

Temperature

A parameter that smooths or sharpens a probability distribution; higher distillation temperatures reveal more relative structure.

White-box distillation

Authorized training with access to internal teacher signals such as logits or hidden states.

26. Sources and methodology

This guide prioritizes original papers, official technical reports, provider terms and primary legal materials. Vendor allegations and vendor-reported benchmarks are labeled as such. Quantitative claims are tied to the configuration reported by the source rather than generalized to all models.

1. Buciluǎ, Caruana and Niculescu-Mizil, Model Compression, KDD 2006.
2. Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network, 2015.
3. Kim and Rush, Sequence-Level Knowledge Distillation, EMNLP 2016.
4. Wei and colleagues, Finetuned Language Models Are Zero-Shot Learners, ICLR 2022.
5. Wang and colleagues, Self-Instruct, ACL 2023.
6. Gunasekar and colleagues, Textbooks Are All You Need, 2023.
7. Hsieh and colleagues, Distilling Step-by-Step, ACL Findings 2023.
8. Gudibande and colleagues, The False Promise of Imitating Proprietary LLMs, 2023.
9. Agarwal and colleagues, On-Policy Distillation of Language Models, ICLR 2024.
10. Gu and colleagues, MiniLLM: Knowledge Distillation of Large Language Models, ICLR 2024.
11. Turpin and colleagues, Language Models Don’t Always Say What They Think, NeurIPS 2023.
12. Shumailov and colleagues, AI models collapse when trained on recursively generated data, Nature 2024.
13. DeepSeek AI, DeepSeek-R1 technical report and official repository, 2025.
14. Tramèr and colleagues, Stealing Machine Learning Models via Prediction APIs, USENIX Security 2016.
15. Orekondy, Schiele and Fritz, Knockoff Nets, CVPR 2019.
16. Carlini and colleagues, Stealing Part of a Production Language Model, ICML 2024.
17. Anthropic, Detecting and preventing distillation attacks, February 2026.
18. Anthropic, Claude Fable 5 and Claude Mythos 5 and the system card, June 2026.
19. Moonshot AI, Kimi K3: Open Frontier Intelligence, July 2026.
20. Anthropic, Commercial Terms of Service, checked July 22, 2026.
21. U.S. Copyright Office, Copyright and Artificial Intelligence, Part 3, 2025.
22. U.S. Code, 18 U.S.C. §1839 (trade-secret definitions) and 18 U.S.C. §1030 (Computer Fraud and Abuse Act).
23. U.S. Supreme Court, Van Buren v. United States, 593 U.S. 374 (2021).

Editorial standard: A model output, benchmark result or public allegation can suggest a question. Training provenance requires stronger evidence: disclosed data, technical documentation, logs, contractual records, verifiable artifacts or an independently auditable chain. This guide does not treat resemblance as proof.