Pre-training interventions, ex post facto

TL;DR

Instead of running synthetic document fine-tuning (SDF) on the post-trained model, run it on the base model and add the weight difference to the post-trained model. The model picks up the new beliefs while staying much more coherent, and you can train once and reuse the update across post-trained checkpoints.

Grafting (bottom) separates SDF from post-training: learn an additive weight update on the base model and serve it on the post-trained model. This avoids the cost of true midtraining (top) and the damage from doing SDF on an instruct-tuned model (native, middle).

Links

• 📄 Paper (arXiv)
• ✨ Interactive blog post (explore the results)
• ✍️ LessWrong post
• 🧵 Thread on X
• 💻 Code