Definition

Knowledge distillation is when a large, powerful model (called the teacher) is used to train a smaller, simpler model (the student). The goal is to make the small model almost as smart as the big one—without needing as much compute power or memory.

Example

“A tiny chatbot trained using outputs from GPT-4 is an example of knowledge distillation.”

How It’s Used in AI

This technique helps bring advanced model capabilities to devices with limited resources. It's widely used in mobile AI, embedded systems, and for making model compression more effective.

Brief History

Introduced by Geoffrey Hinton in 2015, knowledge distillation became a standard tool for building efficient models like DistilBERT and MobileBERT. It’s a key part of many AI deployment pipelines today.

Key Tools or Models

• DistilBERT, TinyBERT, MiniLM

• Frameworks: PyTorch, TensorFlow, and Hugging Face Transformers

• Used in both NLP and vision models

Pro Tip

Distillation isn’t perfect—student models may miss rare knowledge or subtle logic. Use it for general-purpose tasks, not highly specialized reasoning.