T5Tokenizer: decode does not show special tokens

[jsrozner]

Environment info

• transformers version: 3.4.0
• Platform: macOS-10.15.7-x86_64-i386-64bit
• Python version: 3.8.5
• PyTorch version (GPU?): 1.7.0 (False)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: N/a
• Using distributed or parallel set-up in script? N/a

Who can help

examples/seq2seq: @sshleifer
-->

Information

Model I am using (Bert, XLNet ...): T5Tokenizer

The problem arises when using:

• the official example scripts: (give details below)
• [X ] my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• [ X] my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

from transformers import T5Tokenizer
input = "word  <pad> <unk> </s> </s>"
t5tokenizer = T5Tokenizer.from_pretrained('t5-small')
tokenized = t5tokenizer.batch_encode_plus([input], max_length=10, padding="longest", return_tensors="pt").input_ids
print(t5tokenizer.batch_decode(tokenized, skip_special_tokens=False, clean_up_tokenization_spaces=False))

IDs output: _word <pad> <unk> </s> </s>
decode output: word ⁇

Expected behavior

The tokens should be shown in the decoded output, but everything except for the unknown token is dropped (no pad or EOS).
convert_ids_to_tokens followed by convert_tokens_to_string also drops the tokens.

[sshleifer]

T5: @patrickvonplaten I think you need to set _additional_special_tokens.

[sshleifer]

@jsrozner want to try to fix?

[patrickvonplaten]

This is a duplicate of #5142 and will be fixed with the PR linked below. Thanks for reporting it - seems like multiple people were running into this issue!!!