CLI interface for uploading files

[Wauplin]

Related to #1105 (cli to download files) and especially #1105 (comment).

It would be great to have a CLI command to upload files, using HTTP-only methods. This is not a plan to write a wrapper around git CLI but rather around upload_file and upload_folder utilities.

Usage

>>> huggingface-cli upload --help
usage: huggingface-cli upload REPO_ID [PATH] [PATH_IN_REPO] [--help] [--repo-type REPO_TYPE] [--revision REVISION] [--token TOKEN] [--allow-patterns ALLEW_PATTERNS] [--ignore-patterns IGNORE_PATTERNS] [--create-pr] [--every]

Upload file

Explicit path in repo

# Upload `path/to/file.json` as `data.json` on the Hub
>>> huggingface-cli upload Wauplin/my-cool-model path/to/file.json data.json

Implicit path_in_repo computed based on relative path to current pwd

# Upload `path/to/file.json` as `path/to/file.json` on the Hub
>>> huggingface-cli upload Wauplin/my-cool-model path/to/file.json

Upload folder

Explicit path_in_repo

# Upload `path/to/folder` to `data/` on the Hub
>>> huggingface-cli upload Wauplin/my-cool-model path/to/folder data/

Implicit path_in_repo based on relative path to current pwd.
Implicit path_in_repo must be computed as POSIX path (even on Windows)

# Upload `path/to/folder` to `path/to/folder/` on the Hub
>>> huggingface-cli upload Wauplin/my-cool-model path/to/folder

Implicit folder

# Upload pwd to `/` on the Hub
>>> huggingface-cli upload Wauplin/my-cool-model .

Upload folder with filters

# Upload only safetensors files
>>> huggingface-cli upload Wauplin/my-cool-model . --allow-patterns=*.safetensors

And also --ignore-patterns.

Specify repo_type and revision

# Upload `config.json` to a PR in a dataset
>>> huggingface-cli upload Wauplin/my-cool-dataset config.json --repo-type=dataset --revision=refs/pr/1

Create a PR

# Update `config.json` to a PR
>>> huggingface-cli upload Wauplin/my-cool-dataset config.json --repo-type=dataset --create-pr

---

All of the above cases can be dealt with upload_file and upload_folder. CLI must determine which one to use based on if PATH is a file or a folder.

Potential issues to gracefully handle:

• repo_id does not exist: if so, there is an existing huggingface-cli repo create to suggest
• --token is not passed and huggingface-cli login has not being run
• PATH does not exists
• --allow-patterns / --ignore-patterns is used with a file path (doesn't make sense)

CLI command can be implemented in several PRs, with some features added later. Implementation and tests must solely be based on upload_file and upload_folder (logic must not be re-implemented)

[Wauplin]

Idea for a future PR: add a --every option to run a CommitScheduler instead of a single commit.

[Wauplin]

Closing thanks to #1618 from @martinbrose ! 🚀 Will make a release pretty soon together with the download command (see #1105 (comment)).

# Upload file
>>> huggingface-cli upload my-cool-model ./my-cool-model.safetensors

# Upload filtered directory (example: tensorboard logs except for the last run)
>>> huggingface-cli upload my-cool-model ./model/training /logs --include "*.tfevents.*" --exclude "*20230905*"

# Upload with token
>>> huggingface-cli upload Wauplin/my-cool-model --token=hf_****

# Schedule commits every 30 minutes
huggingface-cli upload Wauplin/my-cool-model --every=30