Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stopping fix: backend fixes for #836 + prep for additional status fields #837

Merged
merged 8 commits into from
May 8, 2023

Conversation

ikreymer
Copy link
Member

@ikreymer ikreymer commented May 7, 2023

  • sets 'stopping' field on crawl when crawl is being stopped (both via db and on k8s object)
  • k8s: show 'stopping' as part of crawljob object, update subchart
  • set 'currCrawlStopping' on workflow
  • tests: add tests to test stopping and canceling crawls

operator: additional optimiziations:

  • run pvc removal as background task
  • catch any exceptions in finalizer stage (eg. if db is down), return false until finalizer completes

crawlconfigs:

ikreymer added 2 commits May 6, 2023 22:56
- sets 'stopping' field on crawl when crawl is being stopped (both via db and on k8s object)
- k8s: show 'stopping' as part of crawljob object, update subchart
- set 'currCrawlStopping' on workflow
- tests: add tests to test stopping and canceling crawls

operator: additional optimiziations:
- run pvc removal as background task
- catch any exceptions in finalizer stage (eg. if db is down), return false until finalizer completes
support old and new browsertrix-crawler stopping keys
@tw4l
Copy link
Member

tw4l commented May 8, 2023

Looks good, thanks for these fixes and the tests!

The CI issue seems to be the same one affecting the test_crawl_timeout nightly test, where the crawl is stopped (in that case due to hitting the timeout) and we'd expect a partial_complete state, but in k3s CI (and seemingly that environment alone) we instead get complete - e.g. https://github.com/webrecorder/browsertrix-cloud/actions/runs/4906101106/jobs/8760308684#step:14:73

@ikreymer ikreymer merged commit fd7e81b into main May 8, 2023
@ikreymer ikreymer deleted the backend-fix-stopping branch May 8, 2023 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants