Error running inference using HuggingFace spaces: AttributeError: 'Wav2Vec2CTCTokenizer' object has no attribute 'unique_no_split_tokens'

[andergisomon]

🐛

I ran inferences using HuggingFace spaces and it worked without issue until recently when wav2vec2 outputs an error.

To Reproduce

I used HuggingFace AutoProcessor to load the wav2vec2 model:

LANG = "dtp" #Change to tih for Timugon Murut or iba for Iban
model_id = "facebook/mms-1b-all"  
processor = AutoProcessor.from_pretrained(model_id)
model = Wav2Vec2ForCTC.from_pretrained(model_id).to("cpu")
processor.tokenizer.set_target_lang(LANG)

It failed on the last line, where the terminal outputs:

Traceback (most recent call last):
  File "/home/user/app/app.py", line 24, in <module>
    processor.tokenizer.set_target_lang(LANG)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/models/wav2vec2/tokenization_wav2vec2.py", line 237, in set_target_lang
    self.unique_no_split_tokens.append(token)
AttributeError: 'Wav2Vec2CTCTokenizer' object has no attribute 'unique_no_split_tokens'

Expected behavior

It should throw no errors and run absolutely fine.

Environment

• fairseq Version: 0.12.2
• PyTorch Version: 2.0.1
• OS: Linux
• How you installed fairseq: pip
• Python version: 3.10

[andergisomon]

This might not be related to wav2vec at all. Refer here: huggingface/transformers#26349

[andergisomon]

Simply downgrade your transformers version. I think this issue came from 4.33.2. Downgrading to 4.30.2 fixed it.