Skip to content

Commit

Permalink
Root methods instead of HfApi (#388)
Browse files Browse the repository at this point in the history
* Updated API proposal

* Docs

* Update docs/hub/how-to-upstream.md

Co-authored-by: Steven Liu <[email protected]>

* Update docs/hub/how-to-upstream.md

Co-authored-by: Steven Liu <[email protected]>

Co-authored-by: Steven Liu <[email protected]>
  • Loading branch information
LysandreJik and stevhliu authored Oct 6, 2021
1 parent 5a0eb4d commit f5c8b25
Show file tree
Hide file tree
Showing 6 changed files with 73 additions and 59 deletions.
19 changes: 5 additions & 14 deletions docs/hub/adding-a-library.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,28 +104,18 @@ This will also add a tag to your model so users can quickly identify models from

## Upload files to the Hub

You might also want to provide a method for creating model repositories and uploading files to the Hub directly from your library. The `HfApi` class has two methods to assist you with creating repositories and uploading files:
You might also want to provide a method for creating model repositories and uploading files to the Hub directly from your library. The `huggingface_hub` library offers two methods to assist you with creating repositories and uploading files:

- `create_repo` creates a repository on the Hub.
- `upload_file` directly uploads files to a repository on the Hub.

Begin by instantiating the `HfApi` class:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
```

This class contains a few methods we're interested in: `create_repo` and `upload_file`.

### `create_repo`

The `create_repo` method creates a repository on the Hub. Use the `name` parameter to provide a name for your repository:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> api.create_repo("test-model")
>>> from huggingface_hub import create_repo
>>> create_repo(name="test-model")
'https://huggingface.co/lysandre/test-model'
```

Expand All @@ -142,7 +132,8 @@ The `upload_file` method uploads files to the Hub. This method requires the foll
For example:

```python
>>> api.upload_file(
>>> from huggingface_hub import upload_file
>>> upload_file(
... path_or_fileobj="/home/lysandre/dummy-test/README.md",
... path_in_repo="README.md",
... repo_id="lysandre/test-model"
Expand Down
2 changes: 1 addition & 1 deletion docs/hub/endpoints.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ We have open endpoints that you can use to retrieve information from the Hub as

## Endpoints table

| Endpoint | Description | `huggingface_hub` `HfApi` method | Payload |
| Endpoint | Description | `huggingface_hub` root methods | Payload |
| -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| /api/models <br/> GET | Get information from all models in the Hub. You can specify additional parameters to have more specific results. <br/>- `filter`: Filter based on tags, such as `text-classification` or `spacy`.<br/>- `sort`: Property to use when sorting. <br/>- `direction`: Direction in which to sort. <br/>- `limit`: Limit the number of models fetched. <br/>- `full`: Whether to fetch most model data, such as all tags, the files, etc. <br/>- `config`: Whether to also fetch the repo config. | `list_models()` | ```params= { "filter":"filter", "full":"full", "sort": "sort", "direction": "direction", "limit": "limit", "config": "config" }``` |
| /api/models/{repo_id} <br/> /api/models/{repo_id}/revision/{revision} <br/> GET | Get all information for a specific model. | `model_info(repo_id, revision)` | ```headers = { "authorization" : "Bearer $token" }``` |
Expand Down
56 changes: 25 additions & 31 deletions docs/hub/how-to-upstream.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ title: How to create repositories and upload files to the Hub

*Upstream* utilities allow you to publish files to the Hub from your library. This guide will show you how to:

* Use the `HfApi` class to manage a repository.
* Use the repository-management methods available in the `huggingface_hub` package.
* Use the `Repository` class to handle files and version control a repository with Git-like commands.

## `HfApi`
## `huggingface_hub` repository-management methods

The `HfApi` class is a high-level class that wraps around HTTP requests. There are many valuable tasks you can accomplish with the `HfApi` class, including:
The `huggingface_hub` package offers high-level methods that wraps around HTTP requests. There are many valuable tasks you can accomplish with it, including:

- List and filter models and datasets.
- Inspect model or dataset metadata.
Expand All @@ -27,90 +27,84 @@ You can view all the available filters on the left of the [model Hub](http://hf.
![/docs/assets/hub/hub_filters.png](/docs/assets/hub/hub_filters.png)

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> from huggingface_hub import list_models

# List all models.
>>> api.list_models()
>>> list_models()

# List only text classification models.
>>> api.list_models(filter="text-classification")
>>> list_models(filter="text-classification")

# List only Russian models compatible with PyTorch.
>>> api.list_models(filter=("languages:ru", "pytorch"))
>>> list_models(filter=("languages:ru", "pytorch"))

# List only the models trained on the "common_voice" dataset.
>>> api.list_models(filter="dataset:common_voice")
>>> list_models(filter="dataset:common_voice")

# List only the models from the spaCy library.
>>> api.list_models(filter="spacy")
>>> list_models(filter="spacy")
```

Explore available public datasets with `list_datasets`:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> from huggingface_hub import list_datasets

# List only text classification datasets.
>>> api.list_datasets(filter="task_categories:text-classification")
>>> list_datasets(filter="task_categories:text-classification")

# List only datasets in Russian for language modeling.
>>> api.list_datasets(filter=("languages:ru", "task_ids:language-modeling"))
>>> list_datasets(filter=("languages:ru", "task_ids:language-modeling"))
```

### Inspect model or dataset metadata

Get important information about a model or dataset as shown below:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> from huggingface_hub import model_info, dataset_info

# Get metadata of a single model.
>>> api.model_info("distilbert-base-uncased")
>>> model_info("distilbert-base-uncased")

# Get metadata of a single dataset.
>>> api.dataset_info("glue")
>>> dataset_info("glue")
```

### Create a repository

Create a repository with `create_repo` and give it a name with the `name` parameter.

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> api.create_repo("test-model")
>>> from huggingface_hub import create_repo
>>> create_repo("test-model")
'https://huggingface.co/lysandre/test-model'
```
### Delete a repository

Delete a repository with `delete_repo`. Make sure you are certain you want to delete a repository because this is an irreversible process!

Pass the full repository ID to `delete_repo`. The full repository ID looks like `{username_or_org}/{repo_name}`, and you can retrieve it with `HfAPI().get_full_repo_name()` as shown below:
Pass the full repository ID to `delete_repo`. The full repository ID looks like `{username_or_org}/{repo_name}`, and you can retrieve it with `get_full_repo_name()` as shown below:

```python
>>> from huggingface_hub import HfApi
>>> name = HfAPI().get_full_repo_name(repo_name)
>>> api = HfApi()
>>> api.delete_repo(name=name)
>>> from huggingface_hub import get_full_repo_name, delete_repo
>>> name = get_full_repo_name(repo_name)
>>> delete_repo(name=name)
```

Delete a dataset repository by adding the `repo_type` parameter:

```python
>>> api.delete_repo(name=REPO_NAME, repo_type="dataset")
>>> delete_repo(name=REPO_NAME, repo_type="dataset")
```

### Change repository visibility

A repository can be public or private. A private repository is only visible to you or members of the organization in which the repository is located. Change a repository to private as shown in the following:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> api.update_repo_visibility(name=REPO_NAME, private=True)
>>> from huggingface_hub import update_repo_visibility
>>> update_repo_visibility(name=REPO_NAME, private=True)
```

## `Repository`
Expand All @@ -134,7 +128,7 @@ The `clone_from` parameter clones a repository from a Hugging Face model ID to a
Easily combine the `clone_from` parameter with `create_repo` to create and clone a repository:

```python
>>> repo_url = HfApi().create_repo(name="repo_name")
>>> repo_url = create_repo(name="repo_name")
>>> repo = Repository(local_dir="repo_local_path", clone_from=repo_url)
```

Expand Down
16 changes: 4 additions & 12 deletions docs/hub/tutorial-add-library.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,26 +104,18 @@ This will also add a tag to your model so users can quickly identify models from

## Upload files to the Hub

You might also want to provide a method for creating model repositories and uploading files to the Hub directly from your library. The `HfApi` class has two methods to assist you with creating repositories and uploading files:
You might also want to provide a method for creating model repositories and uploading files to the Hub directly from your library. The `huggingface_hub` package has two methods to assist you with creating repositories and uploading files:

- `create_repo` creates a repository on the Hub.
- `upload_file` directly uploads files to a repository on the Hub.

Begin by instantiating the `HfApi` class:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
```

### `create_repo`

The `create_repo` method creates a repository on the Hub. Use the `name` parameter to provide a name for your repository:

```python
>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> api.create_repo("test-model")
>>> from huggingface_hub import create_repo
>>> create_repo(name="test-model")
'https://huggingface.co/lysandre/test-model'
```

Expand All @@ -140,7 +132,7 @@ The `upload_file` method uploads files to the Hub. This method requires the foll
For example:

```python
>>> api.upload_file(
>>> upload_file(
... path_or_fileobj="/home/lysandre/dummy-test/README.md",
... path_in_repo="README.md",
... repo_id="lysandre/test-model"
Expand Down
19 changes: 18 additions & 1 deletion src/huggingface_hub/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,24 @@
TF_WEIGHTS_NAME,
)
from .file_download import cached_download, hf_hub_download, hf_hub_url
from .hf_api import HfApi, HfFolder, repo_type_and_id_from_hf_id
from .hf_api import (
HfApi,
HfFolder,
create_repo,
dataset_info,
delete_repo,
get_full_repo_name,
list_datasets,
list_models,
list_repos_objs,
login,
logout,
model_info,
repo_type_and_id_from_hf_id,
update_repo_visibility,
upload_file,
whoami,
)
from .hub_mixin import ModelHubMixin, PyTorchModelHubMixin
from .inference_api import InferenceApi
from .keras_mixin import (
Expand Down
20 changes: 20 additions & 0 deletions src/huggingface_hub/hf_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -981,3 +981,23 @@ def delete_token(cls):
os.remove(cls.path_token)
except FileNotFoundError:
pass


api = HfApi()

login = api.login
logout = api.logout
whoami = api.whoami

list_models = api.list_models
model_info = api.model_info
list_repos_objs = api.list_repos_objs

list_datasets = api.list_datasets
dataset_info = api.dataset_info

create_repo = api.create_repo
delete_repo = api.delete_repo
update_repo_visibility = api.update_repo_visibility
upload_file = api.upload_file
get_full_repo_name = api.get_full_repo_name

0 comments on commit f5c8b25

Please sign in to comment.