[FIRRTL][Dedup] Rework hashing for perf and bug fixes. #7420
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mikeurbach
approved these changes
Aug 1, 2024
dtzSiFive
commented
Aug 1, 2024
Primary change is to only generate and populate mappings for sources of values, and not each value themselves. Values are identified using these base numberings plus an appropriate offset. The main benefit of this is to greatly reduce the number of entries in the `indices` map. When handling operations with many block arguments (module-like's with many ports) or with many results (instances of those module-like's) this greatly reduces the pressure on the `indices` map. For these designs, dedup now runs dramatically faster and uses significantly less memory. Also separates location of the value impl, such that if a Value's impl is storage inline into an Operation or Block such that there is aliasing, the two are given different numbers (and especially the numbering isn't changed). On a synthetic design containing a module with 2^20 ports and 256 instances of that module, this is the difference between completing in 20s and OOM'ing on my machine after running for 30 minutes. Functional changes: Fixes llvm#7415. Fixes llvm#7416. Also fixes deduping if block arg types are different (but unused). This is done by hashing block count, and each block's numbering between as well as the types of its arguments before that block's operations. Additionally fixes use of numberings (indices) before it was populated where attribute processing for inner symbol ports hashed using the block argument's numbering before it was populated.
dtzSiFive
force-pushed
the
fix/dedup-blocks-and-perf
branch
from
64f0f1c to
6d08991
Compare
|
Landing, thanks for reviews folks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Primary change is to only generate and populate mappings for sources of values, and not each value themselves.
Values are identified using these base numberings plus an appropriate offset.
The main benefit of this is to greatly reduce the number of entries in the
indicesmap.When handling operations with many block arguments (module-like's with many ports) or with many results (instances of those module-like's) this greatly reduces the pressure on the
indicesmap. For these designs, dedup now runs dramatically faster and uses significantly less memory.Also separates location of the value impl, such that if a Value's impl is storage inline into an Operation or Block such that there is aliasing, the two are given different numbers (and especially the numbering isn't changed).
On a synthetic design containing a module with 2^20 ports and 256 instances of that module, this is the difference between completing in 20s and OOM'ing on my machine after running for 30 minutes.
Functional changes:
Fixes #7415.
Fixes #7416.
Also fixes deduping if block arg types are different (but unused).
This is done by hashing block count, and each block's numbering between as well as the types of its arguments before that block's operations.
Additionally fixes use of numberings (indices) before it was populated where attribute processing for inner symbol ports hashed using the block argument's numbering before it was populated.