200427 LightPAFF.md

https://arxiv.org/abs/2004.12817

LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning (Kaitao Song, Hao Sun, Xu Tan, Tao Qin, Jianfeng Lu, Hongzhi Liu, Tie-Yan Liu)

#distillation #pretraining #finetuning