Pretrained vocabulary from transformers is sometimes not saved in our Vocabulary object

[MaksymDel]

We only add tokens from pretrained vocab from transformers lib if the tokenizer has vocab key inside.

allennlp/allennlp/data/token_indexers/pretrained_transformer_indexer.py

Line 62 in 88fe007

if not self._added_to_vocabulary and hasattr(self.tokenizer, "vocab"):

This is not the case for all models in the lib and I did not find a unified get_vocab method in huggingface code.
Anyways, the tokens will still be correctly indexed. It is just that we will not have tokens in our vocabulary object with all the consequences.

In #3453 I will add support for RoBERTa and XLM that have encoder keys instead of vocab key, but I do not know about other models.

[matt-gardner]

See also #3097 on issues about when vocabularies are actually written to disk. This is a separate issue, though, right?

[MaksymDel]

Yes, I think they are separate. This one is about correctly getting BERT vocab into our Vocabulary object while #3097 is about saving the vocab to disk.

[MaksymDel]

The newest version of transformers now includes get_vocab method added to tokenizers, which can be used to retrieve all the vocabulary from the tokenizers. Using it should fix this issue.

Edit: it is not blocked anymore since #4018