Convertible-to constraints

[abarnert]

Many type constraints in both stdlib and external functions are not of the form "x is an instance of type T (or a subtype thereof)", but of the form "x is something that can be converted to type T (or a subtype thereof)".

In fact, the de-facto dynamic type system in Python is in some ways closer to C++'s static type system, which is built all around conversions, than to a more traditional static type system. But in C++, convertibility is explicitly definable.*

The problem is that there are a variety of kinds of type-conversion systems in use, many of them user-extensible, including:

• Dunder-method protocols, like "x.__index__ exists". This one's almost easy—an ABC with a __subclasshook__ takes care of it. Except that the static type checker then has to understand ABCs besides the ones that are built in, down to the level of being able to call their __subclasshook__ methods.
• Construction: T(x) would succeed. If there were some way to specify "any type that matches the second argument type of T.__new__ and T.__init__", but that isn't.
• Alternate construction: Some function f(x) (or classmethod T.f(x)) would succeed. (For example, I think the only hard part of defining NumPy's "array-like" is that it's a Union of a bunch of things—ndarray, the NumPy scalar types, native Python numbers, and anything for which np.array(x) is allowed.)
• Delegation to methods. Besides the obvious, this includes delegation to methods which are explicitly intended to be overridden in subclasses. For example, JSONEncoder.encode takes any type that self.default takes.
• Delegation to attributes: T(x._as_parameter_) would succeed. This can also include callables; the return type of a ctypes function is f.restype if it's not callable, but if if it's a Callable[[X], Y] it's Y.
• Registries. Consider how various SQL query libraries handle "can be stored in a REAL column" by, e.g., looking in a map to see if register_adapter(type(x), REAL) has been called.

Not all of these cases need to be statically type-checked, of course. (And ctypes seems like it pretty obviously could/should be a special case, at least in a v2 proposal.) But I think things like MySequence.__getitem__ or np.add are the kinds of things people might want to type. And it would be nice to be able to declare that my function can take anything convertible to, say, ip_address, instead of having to manually Union together ip_address and its constructor's argument type. And so on.

_{* It's actually quite a mess. An implicit conversion (like passing an argument to a function) you have both built-in rules (subtype conversion for reference or pointer types, const qualification, traditional C coercions, etc.) and user-defined (either x.operator T() exists and is accessible, or there's an unambiguous overload for T(x) and it's accessible and not explicit) ones, and can have two conversion steps as long as only one of them is user-defined; an explicit conversion (like assigning an initial value to a variable declaration) has different rules. But at least it's a clearly-documented mess of unbreakable rules, and if you want to extend it with, say, a registry of adapters, you have to write templates that do that at compile time in terms of the rules.}

[gvanrossum]

Hm... I think the first three are ways of saying "the type must be acceptable as the argument to the XXX builtin", right? E.g. I could imagine something that takes either an iterable of pairs or a mapping, and passing it to a dict. But then it just comes down to being able to spell the argument of dict() in the type system, so I don't think something new is needed. (You may have a problem spelling the XXX argument type, but we should face such cases head on.)

JSON is such a dynamic mess that I don't want to worry about it yet. (Basic JSON can be spelled easily as something four-way recursive or such, but the encoder/decoder machinery makes this completely untractable.)

Delegation to attribute sounds like something that structural subtyping could solve; we're not doing that in this PEP but there might be a follow-up PEP. Perhaps this can also handle certain registries. See #11. (It's closed because we decided not to do it in PEP 484, but maybe it should remain open to focus the discussion for a future PEP?)

[JukkaL]

Representing the valid arguments to a built-in X may require a complex union type. I think that it's okay, since we can define a type alias for that union type, and attach a comment to it (saying something like 'these correspond to the valid argument types to foo()'.

Many of the cases presented above may indeed be common needs, at least for certain popular libraries, but I don't think we have enough evidence yet to decide which of these might warrant inclusion in typing (assuming we can figure out how to support the idioms). Even some fairly obviously useful things such as protocols/structural subtyping have been postponed, so postponing additional features until we have more experience seems like a good default approach.

[abarnert]

I think the first three are ways of saying "the type must be acceptable as the argument to the XXX builtin", right?

I think you mean the second through fourth, right? (The first one is probably covered by protocols or more advanced ABC functionality or something else that will already probably be in v2, so that's OK.) If you change "the XXX builtin" to "callable XXX", that's pretty much right. The variations are on where the callable comes from—a builtin or other type constructor, some arbitrary function that we have a handle to, a (contravariant) method of each subclass of the currently-defined class, etc.

Representing the valid arguments to a built-in X may require a complex union type. I think that it's okay, since we can define a type alias for that union type, and attach a comment to it (saying something like 'these correspond to the valid argument types to foo()'.

Writing such types for "acceptable as the argument to XXX" for common XXX values would cover many of the cases (and it could be extended by third-party libraries for their own common XXX, like NumPy's np.array-able, even if array isn't actually a type but an alternate constructor that usually generates instances of np.ndarray types).

But I'm not sure it's a good solution. Think of what this would mean for dicts: you could define Dictable[X, Y] as Union[Mapping[X, Y], Iterable[Tuple[X, Y]]] or something (I realize Dictable is a horrible name; ignore that…), but then users have to know about three different generic types—the concrete Dict, the abstract Mapping, and the conversion-based Dictable. Won't that get confusing?

It would be much nicer if you could just write Constructible[Dict[X, Y]] or Delegate[Dict[X, Y]] (which is also extensible to things like Delegate[self.default] for contravariant subclass method overrides), so you just have to learn one new generic type, Constructible or Delegate, instead of a whole parallel tree of them.

But practically, I don't think either of these belongs in v1. Maybe this should be left open for v2; meanwhile, people can just mark the type of dict-convertible arguments or override-delegated arguments or whatever as Any and make a note of the example for contributing to the design of v2.

[Tronic]

Almost five years later, is there still any solution to this? I.e. can you even declare np.array compatible, or -- ideally -- partial constraints on array dimensions (following the broadcasting principles of Numpy).

[danmou]

I also think a general solution to this would be nice, but for anyone like me coming here in search of a way to say something like any x where float(x) is valid, the typing module defines SupportsFloat and several similar types.

@Tronic as for NumPy, they are working on typing stubs here and shape annotations are on the road map, but probably not any time soon.

[srittau]

Six years later, typing has more tools to handle the shape of objects. I am closing this issue as being too broad, but more specific ideas are still welcome (if they not already have issues here).