-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extract parser: update heuristic for mixed logs #717
Conversation
Follow up from the chat we had yesterday. BTW, I don't mind going back to checking the first character (after trimming) like we were doing, and we can also apply both parsers and drop error on JSON (I don't know the original reason for that). What felt non-negotiable was only checking the first line. |
To recap our discussion in slack: This is a harder problem then it looks, if the result is mostly one format or the other, but occasionally mixed, we risk firing extra requests, or request looping despite any checking we do on the client. We can always choose to use both parsers, which reduces the number of requests, but increases the cost of every query. I know @cyriltovena gave us the ok in the previous version of the app to use both parsers, but maybe it's worth having the conversation again? My gut says that the cost of running unnecessary queries is higher then using two parsers all the time, but that has performance implications for users that have good data that is 100% one format. Straw person: We use both parsers, and if users need to squeeze out more performance we add a config to the loki datasource to assert if the datasource is json, logfmt, or mixed? Crazy idea: Can we have a new "auto" parser? |
Auto parser would be amazing. |
Would be a win for logQL users as well |
3e90712
to
c6ba227
Compare
What do you think? Should we go with this improvement and resume the conversation of smarter alternatives later? |
src/services/fields.ts
Outdated
const linesField = data.fields.find((f) => f.name === 'Line' || f.name === 'body'); | ||
result.type = linesField?.values[0]?.[0] === '{' ? 'json' : 'logfmt'; | ||
linesField?.values.forEach((value: string) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lets use a normal for loop or some or something that will stop executing when we find a non-json log line
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no way to stop or break a forEach() loop other than by throwing an exception. If you need such behavior, the forEach() method is the wrong tool.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. No need to be suboptimal.
This is a good change, but I'll want some more time to test on Monday, worried about re-introducing the request looping bug |
afd04d2
to
1c5f610
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, and is working well locally. I like how this only executes another query for mixed json results! Great work Matias
The first log of the sample can be anything, and the fact that we're only checking one felt awkward.
Updated the implementation to: