Associations upload memory leak?

[ddobie]

During a large pipeline run (https://dev.pipeline.vast-survey.org/piperuns/70) I managed to crash vast-data at the association upload step (

vast-pipeline/vast_pipeline/pipeline/loading.py

Line 223 in 1598169

def make_upload_associations(associations_df: pd.DataFrame) -> None:

)

2024-02-16 16:44:52,155 finalise DEBUG related_df memory after upload: 17.64984MB
2024-02-16 16:44:52,156 finalise DEBUG Current memory usage: 15.524GB
2024-02-16 16:45:06,674 finalise DEBUG sources_df memory after srcs_df merge: 11053.352438MB
2024-02-16 16:45:06,674 finalise DEBUG Current memory usage: 21.095GB
2024-02-16 16:45:06,674 loading INFO Upload associations...
2024-02-16 16:45:09,239 loading DEBUG associations_df memory usage: 11053.352438MB
2024-02-16 16:45:09,240 loading DEBUG Current memory usage: 21.099GB
2024-02-16 16:45:09,240 model_generator DEBUG Building 20815521 association generators

Grafana memory usage for the run is here - things go pear shaped almost immediately.

[ddobie]

Possible solutions here?

https://medium.com/@rui.jorge.rei/today-i-learned-django-memory-leak-and-the-sql-query-cache-1c152f62f64