Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support mariadb slowlog #9731

Merged
merged 22 commits into from
Jan 18, 2019
Merged

Support mariadb slowlog #9731

merged 22 commits into from
Jan 18, 2019

Conversation

jsoriano
Copy link
Member

@jsoriano jsoriano commented Dec 20, 2018 •
edited
Loading

Grok expression in mysql slowlog fileset is modified to support also mariadb.

Continuation of #7422

@jsoriano jsoriano added in progress Pull request is currently in progress. module Metricbeat Metricbeat Team:Integrations Label for the Integrations team labels Dec 20, 2018
@jsoriano jsoriano self-assigned this Dec 20, 2018
@jsoriano jsoriano requested a review from a team as a code owner December 20, 2018 19:32
ruflin
ruflin reviewed Dec 21, 2018
View reviewed changes
Copy link
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are 2 options moving forward. We get this PR in mostly as is and do ECS conversion afterwards or do it directly in the PR. Normally I would say all new modules should be ECS but this is a take over of code.

filebeat/docs/modules/mariadb.asciidoc Outdated Show resolved Hide resolved
filebeat/module/mariadb/_meta/kibana/5/dashboard/Filebeat-MariaDB-Dashboard.json Outdated Show resolved Hide resolved
filebeat/module/mariadb/error/ingest/pipeline.json Outdated Show resolved Hide resolved
filebeat/module/mariadb/error/ingest/pipeline.json Outdated Show resolved Hide resolved
filebeat/module/mariadb/error/ingest/pipeline.json Outdated Show resolved Hide resolved
filebeat/module/mariadb/error/ingest/pipeline.json Outdated Show resolved Hide resolved
filebeat/module/mariadb/slowlog/ingest/pipeline.json Outdated Show resolved Hide resolved
@jsoriano
Copy link
Member Author

@ruflin thanks for the review. There are still some things I want to test, specially related to the possible merge with the mysql module, and some cleanup is also pending.

sayden
sayden previously requested changes Jan 4, 2019
View reviewed changes
Copy link
Contributor

@sayden sayden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. See comments

filebeat/module/mariadb/error/_meta/fields.yml Outdated Show resolved Hide resolved
filebeat/module/mariadb/slowlog/_meta/fields.yml Outdated Show resolved Hide resolved
filebeat/module/mariadb/slowlog/_meta/fields.yml Outdated Show resolved Hide resolved
filebeat/module/mariadb/slowlog/_meta/fields.yml Outdated Show resolved Hide resolved
filebeat/module/mariadb/slowlog/_meta/fields.yml Outdated Show resolved Hide resolved
filebeat/module/mariadb/slowlog/_meta/fields.yml Outdated Show resolved Hide resolved
filebeat/module/mariadb/slowlog/_meta/fields.yml Outdated Show resolved Hide resolved
filebeat/module/mariadb/slowlog/_meta/fields.yml Outdated Show resolved Hide resolved
filebeat/module/mariadb/slowlog/_meta/fields.yml Outdated Show resolved Hide resolved
@jsoriano jsoriano added Filebeat Filebeat and removed Metricbeat Metricbeat labels Jan 14, 2019
@jsoriano jsoriano force-pushed the rdglinux-filebeat-mariadb branch from 21c7ad4 to dca81c5 Compare January 15, 2019 18:11
@jsoriano jsoriano requested review from a team as code owners January 15, 2019 18:11
@jsoriano jsoriano dismissed sayden’s stale review January 15, 2019 18:15

Code has changed a lot, comments were on discarded mariadb module.

@jsoriano jsoriano added needs_backport PR is waiting to be backported to other branches. v6.7.0 labels Jan 15, 2019
@jsoriano
Copy link
Member Author

Finally merged all grok expressions on the mysql module, so it supports both MySQL and MariaDB. This would be ready for review.

@jsoriano jsoriano changed the title Add mariadb module Support mariadb slowlog Jan 15, 2019
@jsoriano jsoriano force-pushed the rdglinux-filebeat-mariadb branch from 4b62bd5 to a703ae1 Compare January 15, 2019 18:25
jsoriano
jsoriano commented Jan 15, 2019
View reviewed changes
filebeat/module/mysql/slowlog/ingest/pipeline.json Outdated
],
"pattern_definitions" : {
"GREEDYMULTILINE" : "(.|\n)*"
"GREEDYMULTILINE": "(.|\n)*",
"METRICSPACE": "([ #\n]*)"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pattern is introduced to separate metrics, this way it doesn't matter if they are separated only by spaces or in different lines. This will help to support new metrics when introduced, and to support different implementations. For example in percona Rows_affected is in the same line after Rows_examined, but in MariaDB it is in the next line.

It still requires that metrics appear in the same order, but I haven't seen differences in order between implementations.

@jsoriano jsoriano mentioned this pull request Jan 16, 2019
Closed
ruflin
ruflin approved these changes Jan 17, 2019
View reviewed changes
Copy link
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. But could you rebase on master so at least Travis goes green?

@jsoriano
Copy link
Member Author

Rebased, added a more complete mariadb slowlog, and added a pattern to ignore by now the explain section that can be optionally added.

@jsoriano jsoriano merged commit 4140305 into elastic:master Jan 18, 2019
@jsoriano jsoriano deleted the rdglinux-filebeat-mariadb branch January 18, 2019 09:37
This was referenced Jan 18, 2019
Closed
Merged
Merged
@jsoriano jsoriano removed the needs_backport PR is waiting to be backported to other branches. label Jan 18, 2019
jsoriano added a commit to jsoriano/beats that referenced this pull request Jan 18, 2019
@jsoriano @rdglinux
Support mariadb slowlog (elastic#9731)
4294e0d 
Add support for MariaDB in the Filebeat module for MySQL.

slowlog grok pattern is modified to support more versions of
this log file and to make it easier to support more in the future.
More example files are added to test this.

(cherry picked from commit 4140305)

Co-authored-by: Rodrigo Floriano de Souza <[email protected]>
jsoriano added a commit that referenced this pull request Jan 21, 2019
@jsoriano @rdglinux
Support mariadb slowlog (#9731) (#10167)
107130a 
Add support for MariaDB in the Filebeat module for MySQL.

slowlog grok pattern is modified to support more versions of
this log file and to make it easier to support more in the future.
More example files are added to test this.

(cherry picked from commit 4140305)

Co-authored-by: Rodrigo Floriano de Souza <[email protected]>
@idonda
Copy link

idonda commented Jan 22, 2019

@jsoriano thank you for adding mariadb/percona slowlog support. will it be released in the next 6.5.5 version?

@jsoriano
Copy link
Member Author

@idonda this will probably get released first on 6.7.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ruflin ruflin ruflin approved these changes

@sayden sayden sayden left review comments

Assignees

@jsoriano jsoriano

Labels
Filebeat Filebeat module review Team:Integrations Label for the Integrations team v6.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jsoriano @idonda @ruflin @sayden @rdglinux