-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Delete output files on re-run & UI run check #421
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See my comment below
@srggrs the forced measurements parquet was a little confusing in the current state - as it stands, until we are able to add an image, and because we don't store the job running config in the DB, when re-running a pipeline the forced measurements should always be checked for and removed. So I've removed the removal step in the actual forced extraction part of the code and made the removal of the old data in the run pipeline command take care of everything. It will check for the forced measurements to delete from the database on every re-run and delete all the parquets. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All good! Merge master after merging #401 then I'll approve
This fixes #418 and removes the parquet and arrow files before the run is re-run.
It also adds a check when running though the UI that the run is not already running or is queued (maybe an old tab is open or something).
For future reference, two things will need to changed: