Can't resolve generic type from return type of callable passed as (default) argument?

[posita]

UPDATE: Probably a dup of #3737 and #10504.

---

What I want to do is define a function that takes a callable that returns a generic as an argument where that generic influences the return type of the function. Something like this:

def doit(constructor: Callable[..., T]) -> List[T]:
    ...

I can't seem to figure out a way to get that done, though, especially when I am supplying a default argument for the callable. Here are my test cases:

from typing import Callable, List, Protocol, TypeVar

_T = TypeVar("_T")

# error: Incompatible default for argument "constructor" (default has type "Type[str]", argument has type "Callable[[int], _T]")
def generate(n: int, constructor: Callable[[int], _T] = str) -> List[_T]:
    return [constructor(i) for i in range(n)]

# error: Need type annotation for "res" (hint: "res: List[<type>] = ...")
res = generate(5)  # res should be of type List[str]
assert res == ["0", "1", "2", "3", "4"]
assert generate(5, float) == [0.0, 1.0, 2.0, 3.0, 4.0]

_T_co = TypeVar("_T_co", covariant=True)

class Constructor(Protocol[_T_co]):
    def __call__(self, n: int) -> _T_co:
        ...

def generate2(n: int, constructor: Constructor[_T] = str) -> List[_T]:
    return [constructor(i) for i in range(n)]

# error: Need type annotation for "res2" (hint: "res2: List[<type>] = ...")
res2 = generate2(5, float)  # res2 should be of type List[float]
assert res2 == [0.0, 1.0, 2.0, 3.0, 4.0]

Runtime:

% python --version
Python 3.9.6
% python test.py
% mypy --version
mypy 0.910
% mypy --config-file=/dev/null test.py
/dev/null: No [mypy] section in config file
test.py:6: error: Incompatible default for argument "constructor" (default has type "Type[str]", argument has type "Callable[[int], _T]")
test.py:10: error: Need type annotation for "res" (hint: "res: List[<type>] = ...")
test.py:24: error: Need type annotation for "res2" (hint: "res2: List[<type>] = ...")
Found 3 errors in 1 file (checked 1 source file)

I tried searching for a similar issue, but was unable to find what I was looking for.

[erictraut]

I think that your first sample (generate) should work. That looks like a bug in mypy. It works fine in pyright.

For your second sample (generate2 with the Constructor protocol), the problem is that you've specified that the __call__ method must have a parameter with the name n, but the constructors for str and float don't have a parameter with that name. If you change the __call__ method to indicate that it accepts position-only parameters, it will type check without errors in both mypy and pyright. The error message provided by mypy in this case is not very useful in diagnosing the problem.

    def __call__(self, n: int, /) -> _T_co:

[posita]

Thanks @erictraut! I can verify that your suggestion fixed my simple test case. Here's one closer to the real thing (note the generic in the more complex return type):

from fractions import Fraction
from typing import List, Protocol, Tuple, TypeVar

_T = TypeVar("_T")
_T_co = TypeVar("_T_co", covariant=True)

class _RationalP(Protocol[_T_co]):
    def __call__(self, numerator: int, denominator: int, /) -> _T_co:
        ...

def generate3(n: int, constructor: _RationalP[_T] = Fraction) -> List[Tuple[float, _T]]:
    return [(float(i), constructor(i, 2)) for i in range(n)]   # ^^^^^^^^^^^^^^^^^^**^^

# error: Need type annotation for "res3"
res3 = generate3(5)  # res3 should be of type List[Tuple[float, Fraction]]
res3_list = list(res3)
print(res3_list)
assert res3_list == [
    (0.0, Fraction(0, 1)),
    (1.0, Fraction(1, 2)),
    (2.0, Fraction(1, 1)),
    (3.0, Fraction(3, 2)),
    (4.0, Fraction(2, 1)),
]

Runtime:

% python --version
Python 3.9.6
% python test.py
[(0.0, Fraction(0, 1)), (1.0, Fraction(1, 2)), (2.0, Fraction(1, 1)), (3.0, Fraction(3, 2)), (4.0, Fraction(2, 1))]
% mypy --version
mypy 0.910
% mypy --config-file=/dev/null test.py
/dev/null: No [mypy] section in config file
test.py:15: error: Need type annotation for "res3"
Found 1 error in 1 file (checked 1 source file)

This doesn't work (at least with MyPy):

# …
_PairT = Tuple[float, _T]
# …
def generate3(n: int, constructor: _RationalP[_T] = Fraction) -> List[_PairT[_T]]:
    ...
# …

This gives a new error:

# …
class _PairT(Tuple[float, _T]):
    ...
# …
# error: Generic tuple types not supported
def generate3(n: int, constructor: _RationalP[_T] = Fraction) -> List[_PairT[_T]]:
    ...
# …

[erictraut]

That should type check without errors. Pyright is fine with it.

If you explicitly pass Fraction as a second argument to generate3, mypy is happy with it. So this appears to be a bug in mypy's handling of default parameter values.

[posita]

Whoo boy.

from fractions import Fraction
from typing import List, Protocol, Tuple, TypeVar, overload

_T = TypeVar("_T")
_T_co = TypeVar("_T_co", covariant=True)

class _RationalP(Protocol[_T_co]):
    def __call__(self, numerator: int, denominator: int, /) -> _T_co:
        ...

@overload
def generate4(n: int) -> List[Tuple[float, Fraction]]:
    ...

@overload
def generate4(n: int, constructor: _RationalP[_T]) -> List[Tuple[float, _T]]:
    ...

def generate4(n: int, constructor: _RationalP[_T] = Fraction) -> List[Tuple[float, _T]]:
    return [(float(i), constructor(i, 2)) for i in range(n)]

res4 = generate4(5)  # res4 should be of type Iterator[Tuple[float, Fraction]]
res4_list = list(res4)
print(res4_list)
assert res4_list == [
    (0.0, Fraction(0, 1)),
    (1.0, Fraction(1, 2)),
    (2.0, Fraction(1, 1)),
    (3.0, Fraction(3, 2)),
    (4.0, Fraction(2, 1)),
]

☝️ That is some work-around!

Thanks for all your help narrowing this down!

[hauntsaninja]

Linking #3737

[posita]

Thanks @hauntsaninja! It does look like this has overlap with #3737, and is probably even a dup, since removing the default argument works, even for the non-protocol version of my simple test:

from typing import Callable, List, TypeVar

_T = TypeVar("_T")

def generate(n: int, constructor: Callable[[int], _T]) -> List[_T]:
    return [constructor(i) for i in range(n)]

res = generate(5, str)
assert res == ["0", "1", "2", "3", "4"]
assert generate(5, float) == [0.0, 1.0, 2.0, 3.0, 4.0]

☝️ No errors with that one.

In any event, this definitely feels like a dupe of #10504, which I just now found (via #3737).

[posita]

Somewhere between 0.993 and 1.4.1 (yes, I know that's a big range), the work-around identified in #10854 (comment) no longer works. 😞 It results in:

...: error: Incompatible default for argument "constructor" (default has type "type[Fraction]", argument has type "_RationalP[_T]")  [assignment]
...: note: "_RationalP[_T].__call__" has type "Callable[[int, int], _T]"

[posita]

New work-around (using None for the default argument and overload to narrow the return type when omitted):

from fractions import Fraction
from typing import Callable, TypeVar, overload

_T = TypeVar("_T")

@overload
def generate(
    n: int,
) -> list[tuple[float, Fraction]]:
    pass

@overload
def generate(
    n: int,
    constructor: Callable[[int, int], _T],
) -> list[tuple[float, _T]]:
    pass

def generate(
    n: int,
    constructor: Callable[[int, int], _T] | None = None,
) -> list[tuple[float, _T]]:
    if constructor is None:
        # This is a persistent error, but it's limited to the implementation. The interface remains usable.
        constructor = Fraction  # type: ignore
        assert constructor is not None

    return [(float(i), constructor(i, 2)) for i in range(n)]

res1 = generate(5, Fraction)
reveal_type(res1)  # list[tuple[float, Fraction]]

res2 = generate(5, lambda n, d: n / d)
reveal_type(res2)  # list[tuple[float, float]]

res3 = generate(5)
reveal_type(res3)  # list[tuple[float, Fraction]]