Distillation can make models smaller, faster, and cheaper. The difficult part is deciding what the student cannot afford to forget.