Error in are_symlinks_supported causing hf_hub_download to stall indefinitely on Windows

[brycepg]

Describe the bug

While trying to use bark.cpp I encountered an error while running download_weights.py, which calls hf_hub_download using huggingface_hub=='0.19.4':

         hf_hub_download(repo_id=repo_id, filename=filename, local_dir=out_dir)

I made a small tweak in in huggingface_hub which fixed my issue. I changed https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/file_download.py#L918

 _support_symlinks = are_symlinks_supported(os.path.dirname(commonpath))

to

  _support_symlinks = are_symlinks_supported((commonpath))

commonpath was "C:\Bryce" which caused the following error. After the code change above it became "C:\Users\Bryce".

Without this code change I recieved:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 368, in mkdtemp
    _os.mkdir(file, 0o700)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\tmpqsiyuu3o'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Bryce\bark.cpp\download_weights.py", line 38, in <module>
    hf_hub_download(repo_id=repo_id, filename=filename, local_dir=out_dir)
  File "C:\Users\Bryce\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\huggingface_hub\utils\_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Bryce\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\huggingface_hub\file_download.py", line 1410, in hf_hub_download
    return _to_local_dir(blob_path, local_dir, relative_filename, use_symlinks=local_dir_use_symlinks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Bryce\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\huggingface_hub\file_download.py", line 1723, in _to_local_dir
    _create_symlink(real_blob_path, local_dir_filepath, new_blob=False)
  File "C:\Users\Bryce\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\huggingface_hub\file_download.py", line 924, in _create_symlink
    _support_symlinks = are_symlinks_supported(os.path.dirname(commonpath))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Bryce\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\huggingface_hub\file_download.py", line 121, in are_symlinks_supported
    with SoftTemporaryDirectory(dir=cache_dir) as tmpdir:
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\Bryce\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\huggingface_hub\utils\_fixes.py", line 55, in SoftTemporaryDirectory
    tmpdir = tempfile.TemporaryDirectory(prefix=prefix, suffix=suffix, dir=dir, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 854, in __init__
    self.name = mkdtemp(suffix, prefix, dir)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 374, in mkdtemp
    if (_os.name == 'nt' and _os.path.isdir(dir) and
                             ^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 42, in isdir
KeyboardInterrupt

Reproduction

1. Clone bark.cpp on a windows machine to C:\Users\<user>\bark.cpp
2. cd into bark.cpp and run pip install -r requirements.txt
3. run download_weights: python3 download_weights.py --download-dir ./models

After the first model is downloaded, download_weights will stall indefinitely

System info

- huggingface_hub version: 0.19.4
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.10
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: C:\Users\Bryce\.cache\huggingface\token
- Has saved token ?: False
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.1.1
- Jinja2: 3.1.2
- Graphviz: N/A
- Pydot: N/A
- Pillow: 10.0.0
- hf_transfer: N/A
- gradio: 3.41.2
- tensorboard: N/A
- numpy: 1.24.4
- pydantic: 2.3.0
- aiohttp: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: C:\Users\Bryce\.cache\huggingface\hub
- HF_ASSETS_CACHE: C:\Users\Bryce\.cache\huggingface\assets
- HF_TOKEN_PATH: C:\Users\Bryce\.cache\huggingface\token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

[Wauplin]

Thanks @brycepg for reporting!

I confirm that I have been able to reproduce on a Windows machine on AWS. The fix you are suggesting seems legit but I'll first want to make a few more tests. I can't remember why we implemented with a os.path.dirname but it had to be for a good reason 🤔 . My only supposition is that the commonpath might be a file instead of a directory in some cases (but I can't see why?) and then we take the parent dir to ensure we are testing on a directory.

[brycepg]

yw @Wauplin !

I'm not sure either since it appears that commonpath always removes the file name:

In [8]: os.path.commonpath(["foo/bar.txt", "foo/bar.json"])
Out[8]: 'foo'

In [9]: os.path.commonpath(["bar.txt", "baz.txt"])
Out[9]: ''

original commit: b38dd9c (it was youu!!!! :) )

Related to #1385. See CI logs in dependent library.

If we try to create a symlink between 2 volumes, common_path will fail with ValueError("Paths don't have the same drive"). Let's not create symlinks in such a case on Windows (still ok on Linux/MacOS).

cc @mayankjobanputra can you please try to run your test suite with this fix (pip install git+https://github.com/huggingface/huggingface_hub@1385-fix-symlinks-on-different-volumes) and report here if it definitely fixed your issue? Thanks in advance.

[Wauplin]

I opened a PR to fix it: #1852. And I remember why we had the os.path.dirname. As explained in the PR, the implementation was initially thought to be used only from within the cache directory. Anyway, it was for a tiny optimization so let's remove it.

[Wauplin]

Hey @brycepg, PR #1852 has been merged which should fix your issue. The fix will be available in next release but you can already use it if you install huggingface_hub from source.. Please let me know how it goes!