Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add results info to the status annotation and status cmd output #829

Merged
merged 1 commit into from
Aug 14, 2019
Merged

Add results info to the status annotation and status cmd output #829

merged 1 commit into from
Aug 14, 2019

Conversation

johnSchnake
Copy link
Contributor

@johnSchnake johnSchnake commented Aug 8, 2019

What this PR does / why we need it:

Which issue(s) this PR fixes

Special notes for your reviewer:
Example output as of now:

"sonobuoy.hept.io/status": "{\"plugins\":[{\"plugin\":\"e2e\",\"node\":\"global\",\"status\":\"complete\",\"result-status\":\"passed\",\"result-counts\":{\"passed\":1,\"skipped\":3584}},{\"plugin\":\"systemd-logs\",\"node\":\"kind-control-plane\",\"status\":\"complete\",\"result-status\":\"passed\",\"result-counts\":{\"passed\":1}}],\"status\":\"complete\",\"tar-info\":{\"name\":\"201908081539_sonobuoy_e3a5f742-7cb2-490e-b0ee-91c6e2c8decb.tar.gz\",\"created\":\"2019-08-08T15:39:45.1724467Z\",\"sha256\":\"bbf3d5e43fe26e101d6c10b0a829ce94e6b2954366a46775172c0696793ce20b\",\"size\":601067}}"

TODO:

  • Update the status command to show some of this info
  • Clean up the code and ensure someone using this as a library can easily gather this information.
  • Main piece of code that needs a test is the code that checks all the leaf nodes' status.
  • TBD: Should I also always add a total field to the result counts? It would make it so people dont have to add them, but it also is a special case that wouldn't really be a count of a particular status so I'm not sure it is a great idea.

Release note:

Added plugin results (based on post-processing results) to the PluginStatus object: both the overall status and the count of all the individual test results. In addition, the Status object itself (which wraps all the PluginStatus objects) has added information on the tarball name, size, and SHA256.

@johnSchnake johnSchnake requested a review from zubron August 8, 2019 16:40
@johnSchnake
Copy link
Contributor Author

Still in a WIP phase but the gist of the logic is here. Let me know what you think.

@codecov-io
Copy link

codecov-io commented Aug 8, 2019

Codecov Report

Merging #829 into master will decrease coverage by 0.24%.
The diff coverage is 44.31%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #829      +/-   ##
==========================================
- Coverage   44.68%   44.44%   -0.25%     
==========================================
  Files          75       76       +1     
  Lines        4565     4687     +122     
==========================================
+ Hits         2040     2083      +43     
- Misses       2392     2473      +81     
+ Partials      133      131       -2
Impacted Files Coverage Δ
pkg/plugin/aggregation/status.go 45.45% <ø> (ø) ⬆️
pkg/discovery/discovery.go 7.93% <18%> (+3.68%) ⬆️
cmd/sonobuoy/app/status.go 59.43% <78.94%> (+2.29%) ⬆️
cmd/sonobuoy/app/gen_plugin.go 28.07% <0%> (ø)
cmd/sonobuoy/app/root.go 93.33% <0%> (+0.47%) ⬆️
pkg/plugin/aggregation/aggregator.go 83.2% <0%> (+4%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f68cb44...0dc3843. Read the comment docs.

}

for i := range podStatus.Plugins {
if podStatus.Plugins[i].Plugin == pluginType {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the plugin result type unique per run? I think I remember seeing that it was but it's slightly confusing to me here because podStatus.Plugins[i].Plugin seems like it would be the plugin's name. I know that our built in plugins use the same values for the name and the result type though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes the type is unique; agree that name/type duality is confusing and I want to fix that. See #830 ; someone reported an issue which was caused because of that too.

@zubron
Copy link
Contributor

zubron commented Aug 8, 2019

I think this is looking good so far. I agree with your comment about adding a total field to the result counts. I think that if a sum is needed, it can be calculated. It makes more sense to me to have only the actual statuses as keys in that object.

@johnSchnake
Copy link
Contributor Author

Clean up the code and ensure someone using this as a library can easily gather this information.

Going to review the code myself for cleanup, but this information is available as part of the status on the object so I think this is already pretty easy with this library (there are methods to get the status). Just not to get this info from a tarball without the pod. That can be a TBD I think.


fmt.Fprintf(tw, "PLUGIN\tNODE\tSTATUS\n")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lack of a trailing \t caused this last column to not be aligned at all

func printAll(w io.Writer, status *aggregation.Status) error {
tw := tabwriter.NewWriter(w, 1, 8, 1, '\t', tabwriter.AlignRight)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alignment was ignored since the padding char was \t, it also seemed to make it impossible to slightly tweak the output in a clear fashion. Changed this out for space padding.


// Effectively making a pivot chart to count the unique combinations of status/result.
statusResultKey := func(p aggregation.PluginStatus) string {
return p.Status + ":" + p.ResultStatus
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We were just iterating over values and rolling on on some value before (status). Now we roll up on the tuple (status, resultStatus).

@@ -161,23 +206,6 @@ func printSummary(w io.Writer, status *aggregation.Status) error {
return nil
}

type pluginSummaries []pluginSummary
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: moved this type data to the top of the page; somewhat of a nit/preference but I have always understood types to be defined at the top of the file under constants/variables.

@@ -94,7 +95,7 @@ func TestPrintStatus(t *testing.T) {
}

if b.String() != test.expected {
t.Errorf("expected output to be %q, got %q", test.expected, b.String())
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tweaked this so that each of "expected" and "got" would start on a new line; just thought it looked more clear at a glance.

@johnSchnake johnSchnake changed the title WIP Add results info to the status annotation and status cmd output Aug 12, 2019
Recently we added some post-processing which would allow users
to analyze arbitrary plugins for how many tests passed/failed/etc.

This change adds some of that information into the status annotation
which we put onto the aggregator pod. This will allow people to check
results before they choose to download the tarball.

In addition, we added a bit mnore metadata into that object describing
the tarball size/name/sha which may be useful for users wanting to
confirm the data that received or save the metadata alongside the
tarball.

Fixes #819

Signed-off-by: John Schnake <[email protected]>
@johnSchnake johnSchnake merged commit 1cce037 into vmware-tanzu:master Aug 14, 2019
@johnSchnake johnSchnake deleted the statusResultsInfo branch August 14, 2019 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Result information should go into PluginStatus
3 participants