Embarrassingly Simple Self-Distillation Improves Code Generation
55.3%. That's how much Qwen3-30B improved on code generation using only its own outputs—no teacher model, no RL, no verifier needed.
Why it matters
Apple's simple self-distillation technique unlocks a cheap, scalable path to model improvement that doesn't require external supervision or reinforcement learning infrastructure—directly relevant to cost-conscious AI teams building production code systems.
The key facts
6 to knowQwen3-30B-Instruct improved from 42.4% to 55.3% pass@1 on LiveCodeBench v6
Method: sample outputs at configured temperature/truncation, then fine-tune with standard SFT
Gains concentrated on harder problems
Generalizes across Qwen and Llama models (4B, 8B, 30B scales)
No verifier, teacher model, or reinforcement learning required
Published by Apple Machine Learning Research
Go to the source
Apple Machine Learningmachinelearning.apple.com
Publisher excerpt: Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation…