Enable profiling of Vitess on demand #610
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Signed-off-by: Florent Poinsard <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new capability to both the backend and the admin UI, we can now request profiles of vitess binaries on demand. The enhanced page on the UI enable the admin to ask for profiles and configure them: which binary (vtgate or vttablet) and which mode (cpu, mem, etc). The ansible backend has been modified too in order to support this. If the admin has requested a profile, we set the proper flag to either vtgate or vttablet and wait for the sysbench benchmark to be about to start to send an
USR1signal to start the benchmark, the same is done when sysbench just finished executing.The profiles are stored directly on the benchmarking node, there is no mechanism yet to get it back automatically or to have it published somewhere.
When the benchmark is done and preferrably when no other benchmarks are running, the admin can SCP onto the benchmarking node and fetch the profiles which are stored under:
/pprof/{execution UUID }/{vtgate|vtgate}-{process-id}/cpu.pprof.I added a new page to the admin to clear out the execution queue too, useful when the queue is overcrowded and we need a specific benchmark quickly.