Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid type size explosion when expanding types #17842

Merged
merged 1 commit into from
Sep 27, 2024
Merged

Conversation

JukkaL
Copy link
Collaborator

@JukkaL JukkaL commented Sep 27, 2024

If TypedDict A has multiple items that refer to TypedDict B, don't duplicate the types representing B during type expansion (or generally when translating types). If TypedDicts are deeply nested, this could result in lot of redundant type objects.

Example where this could matter (assume B is a big TypedDict):

class B(TypedDict):
    ...

class A(TypedDict):
    a: B
    b: B
    c: B
    ...
    z: B

Also deduplicate large unions. It's common to have aliases that are defined as large unions, and again we want to avoid duplicating these unions.

This may help with #17231, but this fix may not be sufficient.

If TypedDict A has multiple items that refer to TypedDict B, don't
duplicate the types representing B during type expansion (or generally
when translating types). If TypedDicts are deeply nested, this could
result in lot of redundant type objects.

Example where this could matter (assume B is a big TypedDict):

```
class A(TypedDict):
    a: B
    b: B
    c: B
    ...
    z : B

```

Also deduplicate large unions. It's common to have aliases
that are defined as large unions, and again we want to avoid
duplicating these unions.
Copy link
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@JukkaL JukkaL merged commit 26a77f9 into master Sep 27, 2024
18 of 19 checks passed
@JukkaL JukkaL deleted the deduplicate-types branch September 27, 2024 16:04
@hauntsaninja
Copy link
Collaborator

I ran this commit against mypy_primer with at compile level 3 using an average of 5 runs

This made https://github.com/vega/altair 6.4x faster, going from 110s to 17s!

(all other meaningful projects are in the 0.95x to 1.05x range)

@JukkaL
Copy link
Collaborator Author

JukkaL commented Oct 11, 2024

Nice, thanks for looking into the perf impact! And I think we still have a bunch of improvements we can make to very large types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants