v0.14.1: patch release

Fixed an issue reported in diffusers impacting users downloading files from outside of the Hub. Expected download size now takes into account potential compression in the HTTP requests.

• Fix consistency check when downloading a file by @Wauplin in #1449

Full Changelog: v0.14.0...v0.14.1

v0.14.0: Filesystem API, Webhook Server, upload improvements, keep-alive connections, and more

HfFileSystem: interact with the Hub through the Filesystem API

We introduce HfFileSystem, a pythonic filesystem interface compatible with fsspec. Built on top of HfApi, it offers typical filesystem operations like cp, mv, ls, du, glob, get_file and put_file.

>>> from huggingface_hub import HfFileSystem
>>> fs = HfFileSystem()

# List all files in a directory
>>> fs.ls("datasets/myself/my-dataset/data", detail=False)
['datasets/myself/my-dataset/data/train.csv', 'datasets/myself/my-dataset/data/test.csv']

>>> train_data = fs.read_text("datasets/myself/my-dataset/data/train.csv")

Its biggest advantage is to provide ready-to-use integrations with popular libraries like Pandas, DuckDB and Zarr.

import pandas as pd

# Read a remote CSV file into a dataframe
df = pd.read_csv("hf://datasets/my-username/my-dataset-repo/train.csv")

# Write a dataframe to a remote CSV file
df.to_csv("hf://datasets/my-username/my-dataset-repo/test.csv")

For a more detailed overview, please have a look to this guide.

• Transfer the hffs code to hfh by @mariosasko in #1420
• Hffs misc improvements by @mariosasko in #1433

Webhook Server

WebhooksServer allows to implement, debug and deploy webhook endpoints on the Hub without any overhead. Creating a new endpoint is as easy as decorating a Python function.

# app.py
from huggingface_hub import webhook_endpoint, WebhookPayload

@webhook_endpoint
async def trigger_training(payload: WebhookPayload) -> None:
    if payload.repo.type == "dataset" and payload.event.action == "update":
        # Trigger a training job if a dataset is updated
        ...

For more details, check out this twitter thread or the documentation guide.

Note that this feature is experimental which means the API/behavior might change without prior notice. A warning is displayed to the user when using it. As it is experimental, we would love to get feedback!

• [Feat] Webhook server by @Wauplin in #1410

Some upload QOL improvements

Faster upload with hf_transfer

Integration with a Rust-based library to upload large files in chunks and concurrently. Expect x3 speed-up if your bandwidth allows it!

• feat: add hf_transfer upload by @McPatate in #1395

Upload in multiple commits

Uploading large folders at once might be annoying if any error happens while committing (e.g. a connection error occurs). It is now possible to upload a folder in multiple (smaller) commits. If a commit fails, you can re-run the script and resume the upload. Commits are pushed to a dedicated PR. Once completed, the PR is merged to the main branch resulting in a single commit in your git history.

upload_folder(
    folder_path="local/checkpoints",
    repo_id="username/my-dataset",
    repo_type="dataset",
    multi_commits=True, # resumable multi-upload
    multi_commits_verbose=True,
)

Note that this feature is also experimental, meaning its behavior might be updated in the future.

• New endpoint: create_commits_on_pr by @Wauplin in #1375

Upload validation

Some more pre-validation done before committing files to the Hub. The .git folder is ignored in upload_folder (if any) + fail early in case of invalid paths.

• Fix path_in_repo validation when committing files by @Wauplin in #1382
• Raise issue if trying to upload .git/ folder + ignore .git/ folder in upload_folder by @Wauplin in #1408

Keep-alive connections between requests

Internal update to reuse the same HTTP session across huggingface_hub. The goal is to keep the connection open when doing multiple calls to the Hub which ultimately saves a lot of time. For instance, updating metadata in a README became 40% faster while listing all models from the Hub is 60% faster. This has no impact for atomic calls (e.g. 1 standalone GET call).

• Keep-alive connection between requests by @Wauplin in #1394
• Accept backend_factory to configure Sessions by @Wauplin in #1442

Custom sleep time for Spaces

It is now possible to programmatically set a custom sleep time on your upgraded Space. After X seconds of inactivity, your Space will go to sleep to save you some $$$.

from huggingface_hub import set_space_sleep_time

# Put your Space to sleep after 1h of inactivity
set_space_sleep_time(repo_id=repo_id, sleep_time=3600)

• [Feat] Add sleep_time for Spaces by @Wauplin in #1438

Breaking change

• fsspec has been added as a main dependency. It's a lightweight Python library required for HfFileSystem.

No other breaking change expected in this release.

Bugfixes & small improvements

File-related

A lot of effort has been invested in making huggingface_hub's cache system more robust especially when working with symlinks on Windows. Hope everything's fixed by now.

• Fix relative symlinks in cache by @Wauplin in #1390
• Hotfix - use relative symlinks whenever possible by @Wauplin in #1399
• [hot-fix] Malicious repo can overwrite any file on disk by @Wauplin in #1429
• Fix symlinks on different volumes on Windows by @Wauplin in #1437
• [FIX] bug "Invalid cross-device link" error when using snapshot_download to local_dir with no symlink by @thaiminhpv in #1439
• Raise after download if file size is not consistent by @Wauplin in # 1403

ETag-related

After a server-side configuration issue, we made huggingface_hub more robust when getting Hub's Etags to be more future-proof.

• Update file_download.py by @Wauplin in #1406
• 🧹 Use HUGGINGFACE_HEADER_X_LINKED_ETAG const by @julien-c in #1405
• Normalize both possible variants of the Etag to remove potentially invalid path elements by @dwforbes in #1428

Documentation-related

• Docs about how to hide progress bars by @Wauplin in #1416
• [docs] Update docstring for repo_id in push_to_hub by @tomaarsen in #1436

Misc

• Prepare for 0.14 by @Wauplin in #1381
• Add force_download to snapshot_download by @Wauplin in #1391
• Model card template: Move model usage instructions out of Bias section by @NimaBoscarino in #1400
• typo by @Wauplin (direct commit on main)
• Log as warning when waiting for ongoing commands by @Wauplin in #1415
• Fix: notebook_login() does not update UI on Databricks by @fwetdb in #1414
• Passing the headers to hf_transfer download. by @Narsil in #1444

Internal stuff

• Fix CI by @Wauplin in #1392
• PR should not fail if codecov is bad by @Wauplin (direct commit on main)
• remove cov check in PR by @Wauplin (direct commit on main)
• Fix restart space test by @Wauplin (direct commit on main)
• fix move repo test by @Wauplin (direct commit on main)

Security patch v0.13.4

Security patch to fix a vulnerability in huggingface_hub. In some cases, downloading a file with hf_hub_download or snapshot_download could lead to overwriting any file on a Windows machine. With this fix, only files in the cache directory (or a user-defined directory) can be updated/overwritten.

• Malicious repo can overwrite any file on disk #429 @Wauplin

Full Changelog: v0.13.3...v0.13.4

Patch release v0.13.3

Patch to fix symlinks in the cache directory. Relative paths are used by default whenever possible. Absolute paths are used only on Windows when creating a symlink betweenh 2 paths that are not on the same volume. This hot-fix reverts the logic to what it was in huggingface_hub<=0.12 given the issues that have being reported after the 0.13.2 release (#1398, huggingface/diffusers#2729 and huggingface/transformers#22228)

Hotfix - use relative symlinks whenever possible #1399 @Wauplin

Full Changelog: v0.13.2...v0.13.3

Patch release v0.13.2

Patch to fix symlinks in the cache directory. All symlinks are now absolute paths.

• Fix relative symlinks in cache #1390 @Wauplin

Full Changelog: v0.13.1...v0.13.2

Patch release v0.13.1

Patch to fix upload_folder when passing path_in_repo=".". That was a breaking change compared to 0.12.1. Also added more validation around the path_in_repo attribute to improve UX.

• Fix path_in_repo validation when committing files by @Wauplin in #1382

Full Changelog: v0.13.0...v0.13.1

v0.13.0: download files to a specific folder, documentation, duplicate spaces, and more

Download files to a specific folder

It is now possible to download files from the Hub and move them to a specific folder!

Two behaviors are possible: either create symlinks or move the files from the cache. This can be controlled with the local_dir_use_symlinks input parameter. The default -and recommended- value is "auto" which will duplicate small files to ease user experience (no symlinks when editing a file) and create symlinks for big files (save disk usage).

from huggingface_hub import snapshot_download
# or "from huggingface_hub import hf_hub_download"

# Download and cache files + duplicate small files (<5MB) to "my-folder" + add symlinks for big files
snapshot_download(repo_id, local_dir="my-folder")

# Download and cache files + add symlinks in "my-folder"
snapshot_download(repo_id, local_dir="my-folder", local_dir_use_symlinks=True)

# Duplicate files already existing in cache and/or download missing files directly to "my-folder"
snapshot_download(repo_id, local_dir="my-folder", local_dir_use_symlinks=False)

• Download file to specific destination by @Wauplin in #1360

Documentation

Efforts to improve documentation have continued. The guides overview has been refactored to display which topics are covered (repository, upload, download, search, inference, community tab, cache, model cards, space management and integration).

Upload / Download files

The repository, upload and download guides have been revisited to showcase the different possibilities to manage a repository and upload/download files to/from it. The focus has been explicitly put on the HTTP endpoints rather than the git cli.

• Refactor guides section + promote HTTP over GIT by @Wauplin in #1338

Integrate a library

A new guide has been added on how to integrate any ML framework with the Hub. It explains what is meant by that and how to do it. Here is the summary table to remember:

• Write Mixins/Integration guide by @Wauplin in #1362

Other

• Add repo_type to repo_info docstring by @albertvillanova in #1347

New endpoints + QOL improvements

Duplicate a Space

It's now possible to duplicate a Space programmatically!

>>> from huggingface_hub import duplicate_space

# Duplicate a Space to your account
>>> duplicate_space("multimodalart/dreambooth-training")
RepoUrl('https://huggingface.co/spaces/nateraw/dreambooth-training',...)

• ✨ add duplicate_space by @nateraw in #1353

delete_patterns in upload_folder

New input parameter delete_patterns for the upload_folder method. It allows to delete some remote files before pushing a folder to the Hub, in a single commit. Useful when you don't exactly know which files have already been pushed. Here is an example to upload log files while deleting existing logs on the Hub:

api.upload_folder(
    folder_path="/path/to/local/folder/logs",
    repo_id="username/trained-model",
    path_in_repo="experiment/logs/",
    allow_patterns="*.txt", # Upload all local text files
    delete_patterns="*.txt", # Delete all remote text files before
)

• Add delete_patterns option to upload_folder by @Wauplin in #1370

List repo history

Get the repo history (i.e. all the commits) for a given revision.

# Get initial commit on a repo
>>> from huggingface_hub import list_repo_commits
>>> initial_commit = list_repo_commits("gpt2")[-1]

# Initial commit is always a system commit containing the `.gitattributes` file.
>>> initial_commit
GitCommitInfo(
    commit_id='9b865efde13a30c13e0a33e536cf3e4a5a9d71d8',
    authors=['system'],
    created_at=datetime.datetime(2019, 2, 18, 10, 36, 15, tzinfo=datetime.timezone.utc),
    title='initial commit',
    message='',
    formatted_title=None,
    formatted_message=None
)

• Add list_repo_commits to list git history of a repo by @Wauplin in #1331

Accept token in huggingface-cli login

--token and --add-to-git-credential option have been added to login directly from the CLI using an environment variable. Useful to login in a Github CI script for example.

huggingface-cli login --token $HUGGINGFACE_TOKEN --add-to-git-credential

• Add token and git credentials to login cli command by @silvanocerza in #1372
• token in CLI login docs by @Wauplin (direct commit on main)

Telemetry helper

Helper for external libraries to track usage of specific features of their package. Telemetry can be globally disabled by the user using HF_HUB_DISABLE_TELEMETRY.

from huggingface_hub.utils import send_telemetry

send_telemetry("gradio/local_link", library_name="gradio", library_version="3.22.1")

• Add utility to send telemetry by @Wauplin in #1344

Breaking change

When loading a model card with an invalid model_index in the metadata, an error is explicitly raised. Previous behavior was to trigger a warning and ignore the model_index. This was problematic as it could lead to a loss of information. Fixing this is a breaking change but impact should be limited as the server is already rejecting invalid model cards. An optional ignore_metadata_errors argument (default to False) can be used to load the card with only a warning.

• Explicit raise on invalid model_index + add ignore_metadata_errors option by @Wauplin in #1377

Bugfixes & small improvements

Model cards, datasets cards and space cards

A few improvements in repo cards: expose RepoCard as top-level, dict-like methods for RepoCardData object (#1354), updated template and improved type annotation for metadata.

• Updating MC headings by @EziOzoani in #1367
• Switch datasets type in ModelCard to a list of datasets by @davanstrien in #1356
• Expose RepoCard at top level + few qol improvements by @Wauplin in #1354
• Explicit raise on invalid model_index + add ignore_metadata_errors option by @Wauplin in #1377

Misc

• Fix contrib CI for timm by @Wauplin in #1346
• 🚑 hotfix isHf test user permissions bug by @nateraw in #1357
• Remove unused private methods by @Wauplin in #1359
• Updates types for RepoCards fields by @davanstrien in #1358
• some typos by @Wauplin (direct commit on main)
• [WIP] make repr formatting narrower by @davanstrien in #903
• Add ReprMixin to Repofile by @Wauplin (direct commit on main)
• Fix ReprMixin for python3.7 + test by @Wauplin in #1380
• fix create_commit on lowercased repo_id + add regression test by @Wauplin in #1376
• FIX create_repo with exists_ok but no permission by @Wauplin in #1364

Patch release v0.12.1

Hot-fix to remove authorization header when following redirection (using cached_download). Fix was already implemented for hf_hub_download but we forgot about this one. Has only a consequence when downloading LFS files from Spaces. Problem arose since a server-side change on how files are served. See #1345.

Full Changelog: v0.12.0...v0.12.1

v0.12.0: Spaces on steroids 🚀, extended HfApi, documentation, modelcards and more

Spaces on steroids 🚀

Spaces support has been substantially enhanced. You can now:

• Request hardware for your Space !
• Configure your Space secrets
• Get runtime information about your Space

# Assign hardware when creating the Space
api.create_repo(repo_id=repo_id, repo_type="space", space_sdk="gradio", space_hardware="cpu-upgrade")

# Configure some secrets
api.add_space_secret(repo_id=repo_id, key="HF_TOKEN", value="hf_api_***")

# Request hardware on the fly
api.request_space_hardware(repo_id=repo_id, hardware="t4-medium")

# Get Space runtime (state, hardware, sdk,...)
api.get_space_runtime(repo_id=repo_id)

Visit the docs for more details.

• Manage space hardware and secrets programmatically by @Wauplin in #1243
• Mock Space tests requiring GPUs by @Wauplin in #1263
• Remove get_space_secrets endpoint by @Wauplin in #1264

And in bonus: Spaces now support Dockerfile natively !

api.create_repo(repo_id=repo_id, repo_type="space", space_sdk="docker")

Check out the docs for more details.

New features and improvements for HfApi

List all branches and tags from a repo

It's now possible to list branches/tags from a repo, getting exact ref and target_commit.
More details in the docs.

>>> api.list_repo_refs("bigcode/the-stack", repo_type='dataset')
GitRefs(
    branches=[
        GitRefInfo(name='main', ref='refs/heads/main', target_commit='18edc1591d9ce72aa82f56c4431b3c969b210ae3'),
        GitRefInfo(name='v1.1.a1', ref='refs/heads/v1.1.a1', target_commit='f9826b862d1567f3822d3d25649b0d6d22ace714')],
    converts=[],
    tags=[
        GitRefInfo(name='v1.0', ref='refs/tags/v1.0', target_commit='c37a8cd1e382064d8aced5e05543c5f7753834da')
    ]
)

• List branches and tags from a repo by @Wauplin in #1276

Manage your "favorite repos" list

New endpoints to like a repo, unlike it and list_liked_repos.

• Add method to list liked repos by @Wauplin in #1254
• Fix list_liked_repos (only public likes are returned) by @Wauplin in #1273
• Fix like test by @Wauplin (direct commit on main)

Get the "real" repo_id when creating a repo

When using create_repo, one can provide a simple repo_name without specifying a namespace (example: "my-cool-model" instead of "Wauplin/my-cool-model"). This was annoying for as one could easily know if a namespace has been added to the provided repo_id. To facilitate this, the return value of create_repo is now an instance of RepoUrl which contains information like the endpoint, namespace, repo_id and repo_type.

• Extend create_repo return value by @Wauplin in #1268

Create branch from any revision

By default, new branches start from main HEAD. It's now possible to specify any branch, tag or commit id to start from.

• Create branch from given revision by @Wauplin in #1293

Modelcards

Modelcards module is getting some adjustments to better integrate with the Hub. The scope of this work is larger than "just" huggingface_hub and resulted in the launch of the HF Model Card Guidebook to help operationalize model cards in the ML community.

• Clarify what "More Resources" is by breaking it into its parts. by @meg-huggingface in #1238
• Adding "summary" variables by @meg-huggingface in #1248
• datasetcard_template: I think linking to a GH user does not make sense anymore now that dataset repos are fully on the Hub by @julien-c in #1257
• Push empty model card and dataset card to the Hub by @Wauplin in #1261
• [Dataset | Model card] When pushing to template repos, work on actual raw contents by @julien-c in #1282
• Update REGEX_YAML_BLOCK by @julien-c in #1285
• Make model use more visible by @osanseviero in #1286

Documentation updates

Quite some effort has been put into the documentation in the past few weeks:

• a new landing page
• a more detailed installation page
• a new page explaining the difference between HTTP vs GIT approaches
• revisited "Searching the Hub" page
• small updates in the existing docs to add some examples and consistency

• Revisit documentation part 1: divio-like landing page, installation page, configuration page by @Wauplin in #1233
• Updates in docs by @Wauplin in #1235
• Doc page: git vs http by @Wauplin in #1245
• Minor suggestions post #1245 by @osanseviero in #1249
• fix broken urls in docs by @Wauplin in #1253
• [Minor doc tweak] *currently... by @julien-c in #1256
• Update Repository docstring by @Wauplin in #1297
• add link to create_repo by @Wauplin in #1299
• Document .no_exist folder by @Wauplin in #1308
• Revisit "search the hub" page (+HfApi reference page) by @Wauplin in #1303

Test suite for dependent libraries

huggingface_hub is getting more and more mature but you might still have some friction if you are maintainer of a library depending on huggingface_hub. To help detect breaking changes that would affect third-party libraries, we built a framework to run simple end-to-end tests in our CI. This is still preliminary work but the hope is make hfh ecosystem more and more robust over time. Check out our README for more details.

• [RFC] Contrib test suite + tests for timm and sentence_transformers by @Wauplin in #1200

Faster download

Goal is to download files faster. First step has been to increase the chunk size by which the files are uploaded. Second step has been to add an optional Rust extension. This is not officially documented for now as we are internally assessing its benefits and limits. Installing and activating hf_transfer is purely optional.

• Enabling hf_transfer use. by @Narsil in #1272
• Increase the chunk size for faster download by @Narsil in #1267

Breaking changes

• Repository "clone_from" feature do not create the remote repository if it doesn't exist on the Hub. Please use create_repo first before cloning it locally. The private init argument has also been removed as it was not needed anymore.
• allow_regex and ignore_regex have been removed from snapshot_download in favor allow_patterns and ignore_patterns.
• git-based implementation of push_to_hub_fastai has been removed in favor of the HTTP-based approach. Same for ModelHubMixin, PyTorchModelHubMixin, push_to_hub_keras and KerasModelHubMixin.
• create_repo is now forced to use keyword-arguments. Same for metadata_eval_result.

• Remove deprecated code in v0.12 by @Wauplin in #1196

QOL improvements

Not really some features, not really some fixes. But definitely a quality of life improvement for users 🙂

• Progress bar when uploading LFS files by @Wauplin in #1266
• tell user how to paste token on Windows by @fristhon in #1289
• fixes 1290 - check for existing token during interpreter login by @lonnen in #1291
• Store token in configurable HF_HOME (in HfFolder) by @Wauplin in #1236
• Compute sha by chunks when uploading by @Wauplin in #1296

Internal

CI runs on Windows !

• Add windows machine for CI by @Wauplin in #1112

Misc

• Remove obsolete comment by @severo in #1265
• fix: authorization header should never forwarded by @rtrompier in #1269
• refacto Offline repository tests by @Wauplin (direct commit on main)
• Revert "refacto Offline repository tests" by @Wauplin (direct commit on main)
• 1278 More robust temporary directory by @Wauplin in #1279
• Refacto Repository tests by @Wauplin in #1277
• skip flake8 B028 by @Wauplin (direct commit on main)

Bugfixes & small improvements

• Fix metadata_update for verified evaluations by @lewtun in #1214
• Accept pathlib.Path in upload_file by @Wauplin in #1247
• Skip failing tests related to DatasetFilter by @Wauplin in #1251
• fix tasks ids test by @Wauplin (direct commit on main)
• Fix InferenceAPI on image task by @Wauplin in #1270
• Update "move_repo" doc url + fix Bigcode test in CI by @Wauplin in #1292
• Fix test_comment_discussion by @Wauplin in #1295
• Handle hf:// urls + raise ValueError if repo...

Patch release v0.11.1

Hot-fix to fix permission issues when downloading with hf_hub_download or snapshot_download. For more details, see #1220, #1141 and #1215.

Full changelog: v0.11.0...v0.11.1