[ERROR] Tokenizer and TokenizerFast ???

[1512262]

🐛 Bug

Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): 'bert-base-multilingual-cased'

The problem arises when using:

• [ x] the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• [ x] my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. from transformers import *
2. tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-cased')
3. tokenizer.decode(tokenizer.encode('mở bài lạc trôi')) --> wrong

but:

1. from transformers import *
2. tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
3. tokenizer.decode(tokenizer.encode('mở bài lạc trôi')) --> true

Expected behavior

the decode sentence after encoding and decoding using TokenizerFast should be true

Environment info

• transformers version:
• Platform: Pytorch and TF
• Python version: 3.6
• PyTorch version (GPU?): GPU
• Tensorflow version (GPU?): 2.2
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

[n1t0]

This is related to #2917. In the slow tokenizers, when do_lower_case=False we don't strip accents, while we do it when do_lower_case=True. In the fast tokenizers, this is controlled by the strip_accents option, which is True here.

@thomwolf How do you think we should fix this?

[thomwolf]

Yes let's do it @n1t0 and stick to the official bert tokenizer behavior in the fast tokenizers as well.