Partially override types of a module to supply only the needed incomplete types

[Zeckie]

TLDR: Provide a way to have stubs that don't contain everything in the package.

Currently, when creating stubs, you have to create stubs for everything under the top level (including __init__.py files so it treats folders as modules). It would be useful to be able to mark the stub (or a class / function within it) as incomplete, and give better errors if later attempting to use something not included in the stub.

eg. When using requests

example.py

import requests
from requests import Response

r : Response = requests.get('https://example.com/')
print (r.status_code)
print (r.text)

requests.pyi

class Response:
    text: str
    status_code: int

def get(url: str) -> Response: ...

So that stub is enough so basedmypy can check example.py, but doesn't include:

• other classes / modules (like requests.PreparedRequest and requests.api)
• all the fields of requests.Response
• all the optional parameters of requests.get

If the code then tried replacing requests.get with requests.Session().get, basedmypy currently complains:

example.py:4:16: error: Module has no attribute "Session"  [attr-defined]
    r : Response = requests.Session().get('https://example.com/')
                   ^

Would be useful to be able to mark it somehow as partial, so it will provide a message to say that the stub does not contain requests.Session.

The same behaviour currently occurs even when types-requests is installed (say for example we wanted to use those stubs, but fix one function or field in there that was incorrectly typed, or just contained Anys).

Example: requests.model.Response' contains this function, with lots of Any`s:

    def json(
        self,
        *,
        cls: type[JSONDecoder] | None = ...,
        object_hook: Callable[[dict[Any, Any]], Any] | None = ...,
        parse_float: Callable[[str], Any] | None = ...,
        parse_int: Callable[[str], Any] | None = ...,
        parse_constant: Callable[[str], Any] | None = ...,
        object_pairs_hook: Callable[[list[tuple[Any, Any]]], Any] | None = ...,
        **kwds: Any,
    ) -> Any: ...

[KotlinIsland]

You can currently provide partial stubs, and when using an member that isn't in the stub you get the error in the OP, are you suggesting that if the stub is annotated somehow as 'incomplete'/'partial' then you will instead receive a different error such as "using member from incomplete stub"?

say for example we wanted to use those stubs, but fix one function or field in there that was incorrectly typed

You could just vendor the entire stub and fix that def.

If you mean you want to do something like patching in/overriding specific types then that sounds like #17.

And avoiding the Anys in third-party stubs is covered by #148

[Zeckie]

... are you suggesting that if the stub is annotated somehow as 'incomplete'/'partial' then you will instead receive a different error such as "using member from incomplete stub"?

Yes, or something like "incomplete stub does not contain definition for requests.model.PreparedRequest"

You could just vendor the entire stub and fix that def.

Looks like we'd have to include the whole of types-requests, and then handle updating it when the official version is updated. I'd prefer not to have to do that to make one small fix.

[Zeckie]

And avoiding the Anys in third-party stubs is covered by #148

But that doesn't usually make them correct, it just gets rid of the Any related errors. If it is something that I know the type of, it would be good to be able to fix it.

Better example is cookies.pyi, where almost everything is untyped, so can't even tell which functions return.

[KotlinIsland]

Yes, or something like "incomplete stub does not contain definition for requests.model.PreparedRequest"

I don't like the sound of this, if you are removing all of the stub except for the parts you are currently using then you lose IDE completions.

#17 would enable patching specific fixes on top of existing types, which I think addresses the issue more robustly than this suggestion.

[Zeckie]

I don't like the sound of this, if you are removing all of the stub except for the parts you are currently using then you lose IDE completions.

I guess that could be worked around, eg. using a different file extension so any IDE that doesn't understand the partial stubs will ignore them. Presumably, IDE completion is able to be done without stubs.

But put another way, my reason for raising this issue was to have a way of limiting the scope (eg. specify that the stub is only for one class / module / function, and doesn't imply the existence or otherwise of other classes / modules / functions). This could be taken further to separate the typing from what exists.

[KotlinIsland]

I guess that could be worked around, eg. using a different file extension so any IDE that doesn't understand the partial stubs will ignore them. Presumably, IDE completion is able to be done without stubs.

IDE completion is normally based off a vendored typeshed (PyCharm vendors typeshed, and VSCode uses a vendored typeshed for pyright) then falling back to analyzing the modules. A new file extension would be a nightmare.

my reason for raising this issue was to have a way of limiting the scope

I think #17 covers this use case fairly comprehensively.

[Zeckie]

PEP 484 specifies how stub files can be incomplete, using

def __getattr__(name) -> Any: ...

Maybe there could be another value that could be used instead of Any to indicate that basedmypy should keep looking (at other stub / python files). This could be done in a way that other tools will fall back to treating the new value as Any.

PEP 561 specifies a way that stub packages can be partial (not include stubs for all modules). This seems to already be supported by [based]mypy.

[DetachHead]

some good points were raised on the typescript issue for the this feature:
microsoft/TypeScript#36146 (comment)
microsoft/TypeScript#36146 (comment)

the biggest issue imo being the concern that it would make users will be less likely to fix incorrect/incomplete types upstream

[DetachHead]

so basically, broken/incomplete stubs should be fixed upstream