Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make Image and Measurement creation atomic together #648

Merged
merged 5 commits into from
Apr 11, 2022

Conversation

marxide
Copy link
Contributor

@marxide marxide commented Mar 29, 2022

When ingesting inputs for a pipeline run, an Image object is created and saved to the database, then the catalogue is read, its contents are saved to the database as Measurement objects, and a copy of the catalogue is saved to the filesystem in a parquet file.

This PR puts the Image and Measurement object creation in an atomic transaction, i.e. if one fails, all operations conducted during the transaction are undone. If the catalogue read fails, the Image will be deleted from the database.

Fixes #647

@marxide marxide added bug Something isn't working python Pull requests that update Python code labels Mar 29, 2022
@marxide marxide self-assigned this Mar 29, 2022
@marxide marxide force-pushed the issue-647-catalog-read-failure branch from a033af4 to ace35c9 Compare April 7, 2022 21:57
@marxide marxide marked this pull request as ready for review April 8, 2022 16:39
@marxide marxide requested a review from ajstewart April 8, 2022 16:39
@ajstewart
Copy link
Contributor

Is the sky region, if it was a new one, also dumped if the image fails? Can't quite see it from that changed code, though maybe it's not even created there.

Is this something that should be included in testing?

@marxide
Copy link
Contributor Author

marxide commented Apr 8, 2022

Is the sky region, if it was a new one, also dumped if the image fails? Can't quite see it from that changed code, though maybe it's not even created there.

The sky regions are retrieved/created along with the image, so it should be within the same atomic transaction, just buried elsewhere.

Is this something that should be included in testing?

Good point. I've added one in 05adcf7. While I was there I also added a tearDown for RunPipelineTest that cleans up the run products so the pipeline-run folders are back in their original state after the tests are complete.

@marxide marxide merged commit 0ded985 into dev Apr 11, 2022
@marxide marxide deleted the issue-647-catalog-read-failure branch April 11, 2022 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working python Pull requests that update Python code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Image objects persist after a catalogue read failure
2 participants