https://arxiv.org/abs/2007.07834
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou)
cross-lingual lm. mlm + translation lm에 moco를 사용해 문장 단위의 constrastive learnin objective를 결합. #nlp #pretraining #cross_lingual