-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Do not store the hypothetically produced mime-types always #3583
Conversation
Im missing the default mimetypes now: |
@noamblitz, strange, I do have the default mime-types: |
Boefjes that do not return mimetypes in their boefje now also do not seem to stop running and fail after a while. |
Yeah these are the ones that you get when boefjes fail right. |
…om/minvws/nl-kat-coordination into fix/do-not-add-produces-mime-types
Seems to work corectly now again! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No remarks
This is not What I'd expect to be honest. If a boefje creates 4 raw files, all with the same mime-types (or none). I'd expect there to be 4 raw files attached to the boefje-job. |
QA: I didn't see anything weird, and found no errors in the normalizer container logs |
Changes
This is a bug introduced during the create/copy boefje feature, where I needed to start searching for non-local plugins in the job handler. Here I joined - although still unclear to my why I did this - the
plugin.produces
mime-types always, but this is a large list for only the webpage-analysis boefje, meaning we get issues where the saved files had the same big list of mime-types, Bytes deduplicating on the mime-type set as expected. I realised this at one point but perhaps didn't see the harm. The irony.Issue link
Closes #3570
Demo
QA notes
Should resolve the issue described in #3570
Code Checklist
Checklist for code reviewers:
Copy-paste the checklist from the docs/source/templates folder into your comment.
Checklist for QA:
Copy-paste the checklist from the docs/source/templates folder into your comment.