Vocabulary mismatch when using pre-trained transformers

[nicola-decao]

Default OOV and unknown tokens are hardcoded as default in the vocabulary class:

allennlp/allennlp/data/vocabulary.py

Line 226 in 9a6962f

self._padding_token = DEFAULT_PADDING_TOKEN

and

allennlp/allennlp/data/vocabulary.py

Line 227 in 9a6962f

self._oov_token = DEFAULT_OOV_TOKEN

so when using any pre-trained token embedder where OOV and unknown tokens are defined differently (e.g., OOV is [UNK] instead of @@UNKNOWN@@) it breaks.

I would add padding_token and oov_token as parameters of the Vocabulary class inizialiser but I am not sure if this will add inconsistencies.

[matt-gardner]

I think this would be ok. It might also help with #3097. PR welcome! Just be sure that it doesn't break any tests, and that you add sufficient tests for the new functionality.

[nicola-decao]

I am gonna look into this and make a PR in the weekend.