[InferenceClient] Support proxy calls for 3rd party providers #2781
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
tests/test_inference_client.py
Outdated
| @@ -1186,3 +1186,203 @@ def test_chat_completion_error_in_stream(): | |||
| def test_resolve_chat_completion_url(model_url: str, expected_url: str): | |||
| url = _build_chat_completion_url(model_url) | |||
| assert url == expected_url | |||
|
|
|||
|
|
|||
| class TestHFInferenceProvider: | |||
There was a problem hiding this comment.
There might be some refactoring here as the tests are quite similar, but i opted to have separate test classes per provider to make adding tests for a new provider more straightforward.
There was a problem hiding this comment.
(likely to be revisited at some point yes but not shocked by having some redundancy in these tests)
| @@ -2806,7 +2806,7 @@ def visual_question_answering( | |||
| """ | |||
| provider_helper = get_provider_helper(self.provider, task="visual-question-answering") | |||
| request_parameters = provider_helper.prepare_request( | |||
| inputs=None, | |||
| inputs=image, | |||
There was a problem hiding this comment.
not sure how the test did not fail before 😅
tests/test_inference_client.py
Outdated
| @@ -1186,3 +1186,203 @@ def test_chat_completion_error_in_stream(): | |||
| def test_resolve_chat_completion_url(model_url: str, expected_url: str): | |||
| url = _build_chat_completion_url(model_url) | |||
| assert url == expected_url | |||
|
|
|||
|
|
|||
| class TestHFInferenceProvider: | |||
There was a problem hiding this comment.
(likely to be revisited at some point yes but not shocked by having some redundancy in these tests)
| @@ -2806,7 +2806,7 @@ def visual_question_answering( | |||
| """ | |||
| provider_helper = get_provider_helper(self.provider, task="visual-question-answering") | |||
| request_parameters = provider_helper.prepare_request( | |||
| inputs=None, | |||
| inputs=image, | |||
| "Routing the call through Hugging Face's infrastructure using your HF token, " | ||
| "and the usage will be billed directly to your Hugging Face account" |
There was a problem hiding this comment.
Is it actually the case? My understanding was that if a user sets their api_key in their HF account, the proxy will use it. So usage is billed on HF account only if using proxy + haven't set an api_key in their user settings. cc @SBrandeis for confirmation
There was a problem hiding this comment.
| "Routing the call through Hugging Face's infrastructure using your HF token, " | |
| "and the usage will be billed directly to your Hugging Face account" | |
| "Calling fal.ai provider through Hugging Face proxy." |
There was a problem hiding this comment.
ah yes maybe you're right
There was a problem hiding this comment.
My understanding was that if a user sets their api_key in their HF account, the proxy will use it.
correct (but this changed today, there was some back'n'forth)
Resolve #2780
This is a first draft of the implementation of proxied calls. For now, this is done directly in
prepare_requestmethod for each provider, as there might be some some particularities for some providers (e.g. we need to update headers for fal.ai when the call is proxied).