Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Root methods instead of HfApi #388

Merged
merged 5 commits into from
Oct 6, 2021
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 5 additions & 12 deletions docs/hub/adding-a-library.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,18 +104,11 @@ This will also add a tag to your model so users can quickly identify models from

## Upload files to the Hub

You might also want to provide a method for creating model repositories and uploading files to the Hub directly from your library. The `HfApi` class has two methods to assist you with creating repositories and uploading files:
You might also want to provide a method for creating model repositories and uploading files to the Hub directly from your library. The `huggingface_hub` library offers two methods to assist you with creating repositories and uploading files:

- `create_repo` creates a repository on the Hub.
- `upload_file` directly uploads files to a repository on the Hub.

Begin by instantiating the `HfApi` class:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
```

You will also need to retrieve your Hugging Face API token to create a repository and upload files to it. Retrieve your token with the `HfFolder` method:

```python
Expand All @@ -128,9 +121,8 @@ You will also need to retrieve your Hugging Face API token to create a repositor
The `create_repo` method creates a repository on the Hub. Use the `name` parameter to provide a name for your repository:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> api.create_repo(token=token, name="test-model")
>>> from huggingface_hub import create_repo
>>> create_repo(token=token, name="test-model")
'https://huggingface.co/lysandre/test-model'
```

Expand All @@ -148,7 +140,8 @@ The `upload_file` method uploads files to the Hub. This method requires the foll
For example:

```python
>>> api.upload_file(
>>> from huggingface_hub import upload_file
>>> upload_file(
... token,
... path_or_fileobj="/home/lysandre/dummy-test/README.md",
... path_in_repo="README.md",
Expand Down
2 changes: 1 addition & 1 deletion docs/hub/endpoints.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ We have open endpoints that you can use to retrieve information from the Hub as

## Endpoints table

| Endpoint | Description | `huggingface_hub` `HfApi` method | Payload |
| Endpoint | Description | `huggingface_hub` root methods | Payload |
| -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| /api/models <br/> GET | Get information from all models in the Hub. You can specify additional parameters to have more specific results. <br/>- `filter`: Filter based on tags, such as `text-classification` or `spacy`.<br/>- `sort`: Property to use when sorting. <br/>- `direction`: Direction in which to sort. <br/>- `limit`: Limit the number of models fetched. <br/>- `full`: Whether to fetch most model data, such as all tags, the files, etc. <br/>- `config`: Whether to also fetch the repo config. | `list_models()` | ```params= { "filter":"filter", "full":"full", "sort": "sort", "direction": "direction", "limit": "limit", "config": "config" }``` |
| /api/models/{repo_id} <br/> /api/models/{repo_id}/revision/{revision} <br/> GET | Get all information for a specific model. | `model_info(repo_id, revision)` | ```headers = { "authorization" : "Bearer $token" }``` |
Expand Down
56 changes: 25 additions & 31 deletions docs/hub/how-to-upstream.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ title: How to create repositories and upload files to the Hub

*Upstream* utilities allow you to publish files to the Hub from your library. This guide will show you how to:

* Use the `HfApi` class to manage a repository.
* Use the repository-management methods available in the `huggingface_hub` package.
* Use the `Repository` class to handle files and version control a repository with Git-like commands.

## `HfApi`
## `huggingface_hub` repository-management methods

The `HfApi` class is a high-level class that wraps around HTTP requests. There are many valuable tasks you can accomplish with the `HfApi` class, including:
The `huggingface_hub` package offers high-level methods that wraps around HTTP requests. There are many valuable tasks you can accomplish with it, including:

- List and filter models and datasets.
- Inspect model or dataset metadata.
Expand All @@ -27,51 +27,48 @@ You can view all the available filters on the left of the [model Hub](http://hf.
![/docs/assets/hub/hub_filters.png](/docs/assets/hub/hub_filters.png)

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> from huggingface_hub import list_models

# List all models.
>>> api.list_models()
>>> list_models()

# List only text classification models.
>>> api.list_models(filter="text-classification")
>>> list_models(filter="text-classification")

# List only Russian models compatible with PyTorch.
>>> api.list_models(filter=("languages:ru", "pytorch"))
>>> list_models(filter=("languages:ru", "pytorch"))

# List only the models trained on the "common_voice" dataset.
>>> api.list_models(filter="dataset:common_voice")
>>> list_models(filter="dataset:common_voice")

# List only the models from the spaCy library.
>>> api.list_models(filter="spacy")
>>> list_models(filter="spacy")
```

Explore available public datasets with `list_datasets`:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> from huggingface_hub import list_datasets

# List only text classification datasets.
>>> api.list_datasets(filter="task_categories:text-classification")
>>> list_datasets(filter="task_categories:text-classification")

# List only datasets in Russian for language modeling.
>>> api.list_datasets(filter=("languages:ru", "task_ids:language-modeling"))
>>> list_datasets(filter=("languages:ru", "task_ids:language-modeling"))
```

### Inspect model or dataset metadata

Get important information about a model or dataset as shown below:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> from huggingface_hub import model_info, dataset_info

# Get metadata of a single model.
>>> api.model_info("distilbert-base-uncased")
>>> model_info("distilbert-base-uncased")

# Get metadata of a single dataset.
>>> api.dataset_info("glue")
>>> dataset_info("glue")
```

### Create a repository
Expand All @@ -87,39 +84,36 @@ Create a repository with `create_repo` and give it a name with the `name` parame
2. Use `create_repo` to create your repository:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> api.create_repo(token=token, name="test-model")
>>> from huggingface_hub import create_repo
>>> create_repo(token=token, name="test-model")
'https://huggingface.co/lysandre/test-model'
```

### Delete a repository

Delete a repository with `delete_repo`. Make sure you are certain you want to delete a repository because this is an irreversible process!

Pass your token and the full repository ID to `delete_repo`. The full repository ID looks like `{username_or_org}/{repo_name}`, and you can retrieve it with `HfAPI().get_full_repo_name()` as shown below:
Pass your token and the full repository ID to `delete_repo`. The full repository ID looks like `{username_or_org}/{repo_name}`, and you can retrieve it with `get_full_repo_name()` as shown below:

```python
>>> from huggingface_hub import HfApi
>>> name = HfAPI().get_full_repo_name(repo_name)
>>> api = HfApi()
>>> api.delete_repo(token=token, name=name)
>>> from huggingface_hub import get_full_repo_name, delete_repo
>>> name = get_full_repo_name(repo_name)
>>> delete_repo(token=token, name=name)
```

Delete a dataset repository by adding the `repo_type` parameter:

```python
>>> api.delete_repo(token=token, name=REPO_NAME, repo_type="dataset")
>>> delete_repo(token=token, name=REPO_NAME, repo_type="dataset")
```

### Change repository visibility

A repository can be public or private. A private repository is only visible to you or members of the organization in which the repository is located. Change a repository to private as shown in the following:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> api.update_repo_visibility(token=token, name=REPO_NAME, private=True)
>>> from huggingface_hub import update_repo_visibility
>>> update_repo_visibility(token=token, name=REPO_NAME, private=True)
```

## `Repository`
Expand All @@ -143,7 +137,7 @@ The `clone_from` parameter clones a repository from a Hugging Face model ID to a
Easily combine the `clone_from` parameter with `create_repo` to create and clone a repository:

```python
>>> repo_url = HfApi().create_repo(name="repo_name")
>>> repo_url = create_repo(name="repo_name")
>>> repo = Repository(local_dir="repo_local_path", clone_from=repo_url)
```

Expand Down
17 changes: 5 additions & 12 deletions docs/hub/tutorial-add-library.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,18 +104,11 @@ This will also add a tag to your model so users can quickly identify models from

## Upload files to the Hub

You might also want to provide a method for creating model repositories and uploading files to the Hub directly from your library. The `HfApi` class has two methods to assist you with creating repositories and uploading files:
You might also want to provide a method for creating model repositories and uploading files to the Hub directly from your library. The `huggingface_hub` package has two methods to assist you with creating repositories and uploading files:

- `create_repo` creates a repository on the Hub.
- `upload_file` directly uploads files to a repository on the Hub.

Begin by instantiating the `HfApi` class:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
```

You will also need to retrieve your Hugging Face API token to create a repository and upload files to it. Retrieve your token with the `HfFolder` method:

```python
Expand All @@ -128,9 +121,8 @@ You will also need to retrieve your Hugging Face API token to create a repositor
The `create_repo` method creates a repository on the Hub. Use the `name` parameter to provide a name for your repository:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> api.create_repo(token=token, name="test-model")
>>> from huggingface_hub import create_repo
>>> create_repo(token=token, name="test-model")
'https://huggingface.co/lysandre/test-model'
```

Expand All @@ -148,7 +140,8 @@ The `upload_file` method uploads files to the Hub. This method requires the foll
For example:

```python
>>> api.upload_file(
>>> from huggingface_hub import upload_file
>>> upload_file(
... token,
... path_or_fileobj="/home/lysandre/dummy-test/README.md",
... path_in_repo="README.md",
Expand Down
19 changes: 18 additions & 1 deletion src/huggingface_hub/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,24 @@
TF_WEIGHTS_NAME,
)
from .file_download import cached_download, hf_hub_download, hf_hub_url
from .hf_api import HfApi, HfFolder, repo_type_and_id_from_hf_id
from .hf_api import (
HfApi,
HfFolder,
create_repo,
dataset_info,
delete_repo,
get_full_repo_name,
list_datasets,
list_models,
list_repos_objs,
login,
logout,
model_info,
repo_type_and_id_from_hf_id,
update_repo_visibility,
upload_file,
whoami,
)
from .hub_mixin import ModelHubMixin, PyTorchModelHubMixin
from .inference_api import InferenceApi
from .keras_mixin import (
Expand Down
20 changes: 20 additions & 0 deletions src/huggingface_hub/hf_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -871,3 +871,23 @@ def delete_token(cls):
os.remove(cls.path_token)
except FileNotFoundError:
pass


api = HfApi()

login = api.login
logout = api.logout
whoami = api.whoami

list_models = api.list_models
model_info = api.model_info
list_repos_objs = api.list_repos_objs

list_datasets = api.list_datasets
dataset_info = api.dataset_info

create_repo = api.create_repo
delete_repo = api.delete_repo
update_repo_visibility = api.update_repo_visibility
upload_file = api.upload_file
get_full_repo_name = api.get_full_repo_name