Add `"add_prefix_space": true,`; this allows for much stronger token-level performance (e.g. NER, ColBERT)
#10
by
tomaarsen
HF staff
- opened
Hello!
Pull Request overview
- Add
"add_prefix_space": true,to the tokenizer config
Details
This allows for much stronger token-level performance (e.g. NER, ColBERT), because otherwise each token will not be prepended by a space, while our model is trained with data where each token is prepended by a space.
We will need to explain that users can set add_prefix_space to False in the model card somewhere.
cc
@bclavie
@bwarner
@NohTow
could one of you take care of that?
P.s. feel free to hold off on merging for now, this PR can also be used to run some tests first (with revision="refs/pr/...").
Note that you need to use transformers after https://github.com/huggingface/transformers/pull/35593 was merged.
- Tom Aarsen
bclavie
changed pull request status to
merged