Check local files if (RepoNotFound, GatedRepo, HTTPError) while downloading files

[jiamings]

In certain cases, we might assume that the model files exist (like in a docker) but huggingface token is not set, and the user only needs to run the model locally. This works for diffusers with default arguments, but in transformers, a RepositoryNotFoundError is raised if not_local_files_only is true, but user is not logged into huggingface.

Example code:

from transformers import T5EncoderModel, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("DeepFloyd/IF-I-XL-v1.0", subfolder="tokenizer")
# tokenizer = T5Tokenizer.from_pretrained("DeepFloyd/IF-I-XL-v1.0", subfolder="tokenizer", local_files_only=True)

prompt = "something"
text_inputs = tokenizer(prompt, padding="max_length", max_length=77, truncation=True, add_special_tokens=True, return_tensors="pt")

print(text_inputs)

The fix is to check if there is a token or not. If there is a token, then there might be authentication issues. If not, then it might be expected behavior from the user that a local file is being used (like in production).

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[Wauplin]

Hi @jiamings, thanks for asking about this and opening a PR for it! Actually your proposal is perfectly valid as it can also be useful for setups where a cache directory is shared among different users that might not have access to the repo on the Hub.

I hope it is fine with you, I completed your PR to add support for more use cases I had in mind:

• private repos (i.e. repo not found), even if token is passed
• gated repos
• error 500 or 504 => unlikely case if the Hub is down but files are already cached
  I also added a test for it.

cc @julien-c @LysandreJik can you review the logic to confirm it's good for you?

NOTE: failing tests are unrelated