docs: update quality gate docs

test/quality/README.md

@@ -14,6 +14,8 @@ static set of reference container images. The kinds of comparisons made are:
 
 ## Getting started
 
+For information about required setup see: [Required setup](#required-setup).
+
 To capture raw tool output and store into the local `.yardstick` directory for
 further analysis:
 ```
@@ -138,3 +140,183 @@ to keep in mind:
   Pushing this ceiling will likely raise the number of unlabled matches
   significantly for all images. Only bump this ceiling if all possible matches
   are labeled.
+
+## Workflow
+
+One way of working is to simply run `yardstick` and `gate.py` in the `test/quality` directory.
+You will need to make sure the `vulnerabilty-match-labels` submodule has been initialized. This happens automatically
+for some `make` commands, but you can ensure this by `git submodule update --init`. After the submodule has been
+initialized, the match data from `vulnerabilty-match-labels` will be available locally.
+
+**TIP**: when dealing with submodules, it may be convenient to set the git config option `submodule.recurse` to `true`
+so `git checkout` will automatically update submodules to the correct commit:
+```shell
+git config submodule.recurse true
+```
+
+To do this we need some results to begin with. As noted above, start with (this does ensure the submodule is initialized):
+```shell
+make capture
+```
+
+This will download prebuilt SBOMs for the configured images and generate match results for configured tools (here:
+the previous Grype version as well as the local version).
+
+After `make capture` has finished, we should have results and can now start inspecting and
+modifying the comparison labels.
+
+To get started, let's assume we see some quality gate failure in like this (something found in CI
+or after running `./gate.py`):
+```
+Running comparison against labels... 
+   Results used:
+    ├── f4fb4e6e-c911-41b6-9a10-f90b3954a41a : [email protected] against docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
+    └── fcebdd0b-d80a-4fe2-b81a-802c7b98d83b : [email protected] against docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
+
+Match differences between tooling (with labels):
+   TOOL PARTITION      PACKAGE       VULNERABILITY   LABEL         COMMENTARY
+   [email protected] ONLY  [email protected]  CVE-2021-44531  TruePositive  (this is a new FN 😱)
+   [email protected] ONLY  [email protected]  CVE-2021-44532  TruePositive  (this is a new FN 😱)
+   [email protected] ONLY  [email protected]  CVE-2021-44533  TruePositive  (this is a new FN 😱)
+
+Failed quality gate
+   - current F1 score is lower than the latest release F1 score: current=0.80 latest=0.80 image=docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
+   - current indeterminate matches % is greater than 10%: current=13.60% image=docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
+   - current false negatives is greater than the latest release false negatives: current=6 latest=3 image=docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
+```
+
+This tells us some important information: which package, version, and vulnerability had a difference;
+how it was previously labeled, and most importantly: the image we need to focus on (`docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9`).
+
+Using the SHA above, we can run `yardstick` to see which results are available:
+```shell
+$ yardstick result list --result-set pr_vs_latest_via_sbom | grep 808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
+
+5bf0611b-183f-4525-a1ab-f268f62f48b6  docker.io/anchore/test_images:appstreams-centos-stream-8-1a287dd@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9          [email protected]              2022-12-09 20:49:56+00:00
+43a9650a-d5de-4687-b3ba-459105e32cb8  docker.io/anchore/test_images:appstreams-centos-stream-8-1a287dd@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9          [email protected]  2022-12-09 20:49:53+00:00
+67913f57-690f-4f35-a2d9-ffccd2a0b2a1  docker.io/anchore/test_images:appstreams-centos-stream-8-1a287dd@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9          [email protected]               2022-11-01 20:30:52+00:00
+```
+
+We'll need to use the UUIDs to explore the labels, so copy the first UUID, which we can see was run against the last Grype release (`[email protected]`). Use the UUID to explore and edit the results with
+`yardstick label explore`:
+```shell
+yardstick label explore 5bf0611b-183f-4525-a1ab-f268f62f48b6
+```
+
+At this point we can use the TUI to explore and modify the match data, by deleting things or labeling as
+true positives, false positives, etc.. **After making changes make sure to save the results** (`Ctrl-S`)!
+
+At this point you can run the quality gate using updated label data. The quality gate can run against 
+just one image, for example the image we first found in the failure, so run the quality gate and see
+how changes to the label data have affected the result:
+```shell
+./gate.py --image docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
+```
+
+After iterating on all the changes we need using `yardstick label explore`, we're now ready to commit changes. Since
+we're using `git submodules`, we need to do two steps:
+1. get the changes merged to the `vulnerability-match-labels` repository `main` branch
+2. update the submodule in this repository
+
+To create a pull request for the `vulnerability-match-labels`, make sure you are in the `vulnerability-match-labels`
+subdirectory and create a branch -- something like:
+```shell
+git checkout --no-track -b my-branch-name
+```
+
+Commit the changes to this branch, push, create a pull request like normal. NOTE: you may need to add a fork
+(`git remote add ...`) and push to the fork if you don't have push permissions against the main
+`vulnerability-match-labels` repo. After the PR is approved and merged to `vulnerability-match-labels` repo's `main`
+branch, update the submodule locally using:
+```shell
+git submodule update --remote
+```
+
+Next, _commit the submodule change_ as part of any other changes 
+to the Grype pull request and push as part of the in-progress PR
+against Grype. The PR will now use the updated match labels when running
+the quality check.
+
+## Required setup
+
+In order to manage Python versions, [pyenv](https://github.com/pyenv/pyenv) can be used. (e.g. `brew install pyenv`)
+
+Both this project and `yardstick` require Python 3.10.
+
+Using `pyenv`, see which python versions are available, for example:
+```shell
+$ pyenv install --list|grep 3.10
+  3.10.0
+  ...
+  3.10.7
+  ...
+```
+
+In this case, we see `3.10.7` is the latest version, so we'll use that for the rest of the setup:
+
+Install this version using `pyenv`:
+```shell
+pyenv install 3.10.7
+```
+
+NOTE: to view the specific Python versions installed use `pyenv versions`:
+```shell
+$ pyenv versions
+  system
+* 3.8.13 (set by /Users/usr/.pyenv/version)
+  3.10.7
+```
+
+To select the `3.10` version use the exact version number:
+```shell
+pyenv shell 3.10.7
+```
+
+(or maybe just: `pyenv shell $(pyenv versions | grep 3.10 | tail -1)`)
+
+Verify this has worked properly by running:
+```shell
+python --version
+```
+
+**Important:** it is also required to have `oras` installed (e.g. `brew install oras`)
+
+**After** setting the working Python version to 3.10, in the `test/quality` directory,
+you need to set up a virtual environment using:
+```shell
+make venv
+```
+
+**After** creating the virtual environment, you can now activate it to set up a
+working shell using:
+```shell
+. venv/bin/activate
+```
+
+You should now have a shell running in the correct virtual environment, it might look something
+like this:
+```shell
+(venv) user@HOST quality %
+```
+
+Now you should be able to run both `yardstick` and `./gate.py`.
+
+## Troubleshooting
+
+As noted above, yardstick requires Python 3.10. If you try to run with an older version, such as
+the default macOS 3.8 version, you will likely see an error similar to:
+
+```
+Traceback (most recent call last):
+  File "./vulnerability-match-labels/sboms.py", line 12, in <module>
+    import yardstick
+  File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/__init__.py", line 4, in <module>
+    from . import arrange, artifact, capture, cli, comparison, label, store, tool, utils
+  File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/arrange.py", line 4, in <module>
+    from yardstick import artifact
+  File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/artifact.py", line 482, in <module>
+    class ResultSet:
+  File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/artifact.py", line 484, in ResultSet
+    state: list[ResultState] = field(default_factory=list)
+TypeError: 'type' object is not subscriptable
+```