[Feat] Webhook server

docs/source/_toctree.yml

@@ -66,3 +66,5 @@
       title: Repo Cards and Repo Card Data
     - local: package_reference/space_runtime
       title: Space runtime
+    - local: package_reference/webhooks_server
+      title: Webhooks server

docs/source/package_reference/environment_variables.mdx

@@ -115,6 +115,15 @@ to disable this warning.
 
 For more details, see [cache limitations](../guides/manage-cache#limitations).
 
+### HF_HUB_DISABLE_EXPERIMENTAL_WARNING
+
+Some features of `huggingface_hub` are experimental. This means you can use them but we do not guarantee they will be
+maintained in the future. In particular, we might update the API or behavior of such features without any deprecation
+cycle. A warning message is triggered when using an experimental feature to warn you about it. If you know what you are doing,
+you can set `HF_HUB_DISABLE_EXPERIMENTAL_WARNING=1` to disable the warning.
+
+If you are using an experimental feature, please let us know! Your feedback can help us design and improve it.
+
 ### HF_HUB_DISABLE_TELEMETRY
 
 By default, some data is collected by HF libraries (`transformers`, `datasets`, `gradio`,..) to monitor usage, debug issues and help prioritize features.

docs/source/package_reference/webhooks_server.mdx

@@ -0,0 +1,77 @@
+# Webhooks Server
+
+Webhooks are a foundation for MLOps-related features. They allow you to listen for new changes on specific repos or to
+all repos belonging to particular set of users/organizations (not just your repos, but any repo). If you want to know
+more about webhooks on the Huggingface Hub, you can read this generic [guide](https://huggingface.co/docs/hub/webhooks).
+
+[`WebhooksServer`] is a class to easily start a server that can receive webhooks from the Huggingface Hub. The goal
+is to ease setting up your own webhook endpoints, debug them locally and run them in a Space. 
+
+<Tip warning={true}>
+
+This is an experimental feature. This means that we are still working on improving the API. Breaking changes might be
+introduced in the future without prior notice. Make sure to pin the version of `huggingface_hub` in your requirements.
+
+</Tip>
+
+
+
+## Server
+
+The server is a [Gradio](https://gradio.app/) app. It has a UI to display instructions for you or your users and an API
+to listen to webhooks. Implementing a webhook endpoint is as simple as decorating a function. You can then debug it
+by redirecting Huggingface webhooks to your machine (using a gradio tunnel) before deploying it to a Space.
+
+### WebhooksServer
+
+[[autodoc]] huggingface_hub.WebhooksServer
+
+### @webhook_endpoint
+
+[[autodoc]] huggingface_hub.webhook_endpoint
+
+## Payload
+
+[`WebhookPayload`] is the main data structure that represents the payload you would expect from an HF webhook. This is
+a pydantic class which makes it very easy to use with FastAPI. If you pass it as a parameter to a webhook endpoint, it
+will be automatically validated and parsed as a Python object.
+
+For more information about webhooks payload, you can refer to [this guide](https://huggingface.co/docs/hub/webhooks#webhook-payloads).
+
+[[autodoc]] huggingface_hub.WebhookPayload
+
+### WebhookPayload
+
+[[autodoc]] huggingface_hub.WebhookPayload
+
+### WebhookPayloadComment
+
+[[autodoc]] huggingface_hub.WebhookPayloadComment
+
+### WebhookPayloadDiscussion
+
+[[autodoc]] huggingface_hub.WebhookPayloadDiscussion
+
+### WebhookPayloadDiscussionChanges
+
+[[autodoc]] huggingface_hub.WebhookPayloadDiscussionChanges
+
+### WebhookPayloadEvent
+
+[[autodoc]] huggingface_hub.WebhookPayloadEvent
+
+### WebhookPayloadMovedTo
+
+[[autodoc]] huggingface_hub.WebhookPayloadMovedTo
+
+### WebhookPayloadRepo
+
+[[autodoc]] huggingface_hub.WebhookPayloadRepo
+
+### WebhookPayloadUrl
+
+[[autodoc]] huggingface_hub.WebhookPayloadUrl
+
+### WebhookPayloadWebhook
+
+[[autodoc]] huggingface_hub.WebhookPayloadWebhook

setup.py

@@ -41,6 +41,7 @@ def get_version() -> str:
 
 extras["tensorflow"] = ["tensorflow", "pydot", "graphviz"]
 
+
 extras["testing"] = extras["cli"] + [
     "jedi",
     "Jinja2",
@@ -50,6 +51,7 @@ def get_version() -> str:
     "pytest-xdist",
     "soundfile",
     "Pillow",
+    "gradio",  # to test webhooks
 ]
 
 # Typing extra dependencies list is duplicated in `.pre-commit-config.yaml`

src/huggingface_hub/__init__.py

@@ -66,6 +66,21 @@
         "SpaceRuntime",
         "SpaceStage",
     ],
+    "_webhooks_payload": [
+        "WebhookPayload",
+        "WebhookPayloadComment",
+        "WebhookPayloadDiscussion",
+        "WebhookPayloadDiscussionChanges",
+        "WebhookPayloadEvent",
+        "WebhookPayloadMovedTo",
+        "WebhookPayloadRepo",
+        "WebhookPayloadUrl",
+        "WebhookPayloadWebhook",
+    ],
+    "_webhooks_server": [
+        "WebhooksServer",
+        "webhook_endpoint",
+    ],
     "community": [
         "Discussion",
         "DiscussionComment",
@@ -330,6 +345,21 @@ def __dir__():
         SpaceRuntime,  # noqa: F401
         SpaceStage,  # noqa: F401
     )
+    from ._webhooks_payload import (
+        WebhookPayload,  # noqa: F401
+        WebhookPayloadComment,  # noqa: F401
+        WebhookPayloadDiscussion,  # noqa: F401
+        WebhookPayloadDiscussionChanges,  # noqa: F401
+        WebhookPayloadEvent,  # noqa: F401
+        WebhookPayloadMovedTo,  # noqa: F401
+        WebhookPayloadRepo,  # noqa: F401
+        WebhookPayloadUrl,  # noqa: F401
+        WebhookPayloadWebhook,  # noqa: F401
+    )
+    from ._webhooks_server import (
+        WebhooksServer,  # noqa: F401
+        webhook_endpoint,  # noqa: F401
+    )
     from .community import (
         Discussion,  # noqa: F401
         DiscussionComment,  # noqa: F401

src/huggingface_hub/_webhooks_payload.py

@@ -0,0 +1,124 @@
+# coding=utf-8
+# Copyright 2023-present, the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Contains data structures to parse the webhooks payload."""
+from typing import List, Optional
+
+from .utils import is_gradio_available
+from .utils._typing import Literal
+
+
+if not is_gradio_available():
+    raise ImportError(
+        "You must have `gradio` installed to use `WebhooksServer`. Please run `pip install --upgrade gradio` first."
+    )
+
+from pydantic import BaseModel
+
+
+# This is an adaptation of the ReportV3 interface implemented in moon-landing. V0, V1 and V2 have been ignored as they
+# are not in used anymore. To keep in sync when format is updated in
+# https://github.com/huggingface/moon-landing/blob/main/server/lib/HFWebhooks.ts (internal link).
+
+
+WebhookEvent_T = Literal[
+    "create",
+    "delete",
+    "move",
+    "update",
+]
+RepoChangeEvent_T = Literal[
+    "add",
+    "move",
+    "remove",
+    "update",
+]
+RepoType_T = Literal[
+    "dataset",
+    "model",
+    "space",
+]
+DiscussionStatus_T = Literal[
+    "closed",
+    "draft",
+    "open",
+    "merged",
+]
+SupportedWebhookVersion = Literal[3]
+
+
+class ObjectId(BaseModel):
+    id: str
+
+
+class WebhookPayloadUrl(BaseModel):
+    web: str
+    api: Optional[str]
+
+
+class WebhookPayloadMovedTo(BaseModel):
+    name: str
+    owner: ObjectId
+
+
+class WebhookPayloadWebhook(ObjectId):
+    version: SupportedWebhookVersion
+
+
+class WebhookPayloadEvent(BaseModel):
+    action: WebhookEvent_T
+    scope: str
+
+
+class WebhookPayloadDiscussionChanges(BaseModel):
+    base: str
+    mergeCommitId: Optional[str]
+
+
+class WebhookPayloadComment(ObjectId):
+    author: ObjectId
+    hidden: bool
+    content: Optional[str]
+    url: WebhookPayloadUrl
+
+
+class WebhookPayloadDiscussion(ObjectId):
+    num: int
+    author: ObjectId
+    url: WebhookPayloadUrl
+    title: str
+    isPullRequest: bool
+    status: DiscussionStatus_T
+    changes: Optional[WebhookPayloadDiscussionChanges]
+    pinned: Optional[bool]
+
+
+class WebhookPayloadRepo(ObjectId):
+    owner: ObjectId
+    head_sha: Optional[str]
+    name: str
+    private: bool
+    subdomain: Optional[str]
+    tags: Optional[List[str]]
+    type: Literal["dataset", "model", "space"]
+    url: WebhookPayloadUrl
+
+
+class WebhookPayload(BaseModel):
+    event: WebhookPayloadEvent
+    repo: WebhookPayloadRepo
+    discussion: Optional[WebhookPayloadDiscussion]
+    comment: Optional[WebhookPayloadComment]
+    webhook: WebhookPayloadWebhook
+    movedTo: Optional[WebhookPayloadMovedTo]