Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share common logic tree data across processes #31

Open
chrisdicaprio opened this issue Apr 26, 2024 · 0 comments
Open

Share common logic tree data across processes #31

chrisdicaprio opened this issue Apr 26, 2024 · 0 comments

Comments

@chrisdicaprio
Copy link
Collaborator

chrisdicaprio commented Apr 26, 2024

Before looping though sites and imts we pre-calculate the weights, branch_hash_table, and component_branches

  • weights numpy array
  • branch_hash_table List[List[str]] (1_000_000[4])
  • component_branches: List[HazardComponentBranch]

We want to distribute calc_aggregation() across processes with MultiProcess. We should share the common objects to avoid copying the arrays and lists (may not need to bother with component_branches as it can be calculated on the fly quite quickly)

Note that for the full NSHM_v1.0.4 the weight array and hash table are quite small and perhaps this is not necessary:

chrisdc@glacier:~/.../DEV/toshi-hazard-post$ NZSHM22_THS_LOCAL_DIR=/data/toshi_hazard_store/pq-CDC2/ poetry run thp aggregate  demo/hazard_v2.toml 
warning openquake module dependency not available, maybe you want to install
                with nzshm-model[openquake]
Toshi Hazard Post: hazard curve aggregation
=================================================
2024-04-26 15:33:15,329 - toshi_hazard_post.version2.aggregation - INFO - getting sites . . .
2024-04-26 15:33:15,329 - toshi_hazard_post.version2.aggregation - INFO - getting logic trees . . . 
2024-04-26 15:33:15,406 - toshi_hazard_post.version2.aggregation - INFO - calculating weights and branch hash table . . . 
2024-04-26 15:33:15,406 - toshi_hazard_post.version2.logic_tree - DEBUG - generating composite branches
2024-04-26 15:33:43,714 - toshi_hazard_post.version2.aggregation - INFO - time to build weight array and hash table 28.31 seconds
2024-04-26 15:33:43,714 - toshi_hazard_post.version2.aggregation - INFO - Size of weight array: 7MB
2024-04-26 15:33:43,714 - toshi_hazard_post.version2.aggregation - INFO - Size of hash table: 8MB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant