failed to download model snapshot in tmp_folder on Windows

[mayankjobanputra]

Describe the bug

I tried to use tmp_path provided by pytest and

and I am getting the following error.

tmp_path = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_farm_reader_load_hf_local0')

@pytest.mark.integration
def test_farm_reader_load_hf_local(tmp_path):
# Test Case: 2. HuggingFace downloaded (local load)

hf_model = "hf-internal-testing/tiny-random-RobertaForQuestionAnswering"
local_model_path = str(Path.joinpath(tmp_path, "locally_saved_hf"))

model_path = snapshot_download(repo_id=hf_model, revision="main", cache_dir=local_model_path)
test\nodes\test_reader.py:290:

c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\huggingface_hub\utils_validators.py:120: in _inner_fn
return fn(*args, **kwargs)
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\huggingface_hub_snapshot_download.py:235: in snapshot_download
thread_map(
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\tqdm\contrib\concurrent.py:69: in thread_map
return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\tqdm\contrib\concurrent.py:51: in _executor_map
return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\tqdm\std.py:1178: in iter
for obj in iterable:
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\concurrent\futures_base.py:619: in result_iterator
yield fs.pop().result()
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\concurrent\futures_base.py:437: in result
return self.__get_result()
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\concurrent\futures_base.py:389: in __get_result
raise self._exception
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\concurrent\futures\thread.py:57: in run
result = self.fn(*self.args, **self.kwargs)
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\huggingface_hub_snapshot_download.py:212: in _inner_hf_hub_download
return hf_hub_download(
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\huggingface_hub\utils_validators.py:120: in _inner_fn
return fn(*args, **kwargs)
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\huggingface_hub\file_download.py:1316: in hf_hub_download
_create_relative_symlink(blob_path, pointer_path, new_blob=True)
c:\hostedtoolcache\windows\python\3.8.10\x64\lib\site-packages\huggingface_hub\file_download.py:850: in _create_relative_symlink
os.path.dirname(os.path.commonpath([os.path.realpath(src), os.path.realpath(dst)]))

paths = ('\?\C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_farm_reader_load_hf_local...ting--tiny-random-RobertaForQuestionAnswering\snapshots\eea4a81306891746bac9e7715f805a2d9dbf4be7\pytorch_model.bin')

def commonpath(paths):
"""Given a sequence of path names, returns the longest common sub-path."""

if not paths:
    raise ValueError('commonpath() arg is an empty sequence')

paths = tuple(map(os.fspath, paths))
if isinstance(paths[0], bytes):
    sep = b'\\'
    altsep = b'/'
    curdir = b'.'
else:
    sep = '\\'
    altsep = '/'
    curdir = '.'

try:
    drivesplits = [splitdrive(p.replace(altsep, sep).lower()) for p in paths]
    split_paths = [p.split(sep) for d, p in drivesplits]

    try:
        isabs, = set(p[:1] == sep for d, p in drivesplits)
    except ValueError:
        raise ValueError("Can't mix absolute and relative paths") from None

    # Check that all drive letters or UNC paths match. The check is made only
    # now otherwise type errors for mixing strings and bytes would not be
    # caught.
    if len(set(d for d, p in drivesplits)) != 1:
      raise ValueError("Paths don't have the same drive")

E ValueError: Paths don't have the same drive

Reproduction

Use a window machine, install huggingface_hub and run the following code using pytest.

def test_farm_reader_load_hf_local(tmp_path):
  hf_model = "hf-internal-testing/tiny-random-RobertaForQuestionAnswering"
  local_model_path = str(Path.joinpath(tmp_path, "locally_saved_hf"))
  model_path = snapshot_download(repo_id=hf_model, revision="main", cache_dir=local_model_path)

Logs

No response

System info

- huggingface_hub version: 0.13.1
- Platform: Linux-4.15.0-197-generic-x86_64-with-glibc2.27
- Python version: 3.9.14
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Has saved token ?: True
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 1.12.1
- Jinja2: 3.1.2
- Graphviz: N/A
- Pydot: N/A
- Pillow: 9.2.0
- hf_transfer: N/A
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /home/monk/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /home/monk/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/monk/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False

[Wauplin]

Hi @mayankjobanputra , thanks for reporting and sorry for the delay. We introduced a bug in the last release of huggingface_hub. We fixed it yesterday and made a patch release (v0.13.2). Could you update your huggingface_hub package, re-try and tell me if you still have the issue. Thank you in advance.

[Wauplin]

I'm closing this issue as we haven't got feedback for 2 weeks. Problem has probably being fixed by v0.13.2 and v0.13.3 patches. If that's not the case, please let us know by opening this issue.

[mayankjobanputra]

@Wauplin super sorry for not being able to get back to you on this. I think I tested it and it worked, but I will still confirm before the eod. Thanks for a very quick fix. I am super grateful :)

[Wauplin]

No worries, I was just doing some cleaning :) If you can confirm, that would be perfect. Otherwise, don't worry about it.
Glad if it unlocked you quickly :)

[mayankjobanputra]

@Wauplin Sorry for taking so long but I am still getting the same error on Windows with 0.13.3 and 0.13.4 both :(

[mayankjobanputra]

Here is the code sample in case you want to try it: https://github.com/deepset-ai/haystack/pull/4607/files

[Wauplin]

@mayankjobanputra I pushed a fix in #1437. Could you try it please? :)

[mayankjobanputra]

Sure @Wauplin :) Sorry for troubling you 😬

[Wauplin]

No worries, it's always good to report issues :)

[Wauplin]

Closed by #1437.