RFC: manifest file

docs/edit/manifest.md

@@ -0,0 +1,130 @@
+### Context
+
+The usual way to declare what will be tested or not are decorators (`@released`, `@bug`, `@missing_feature`...). This solution offers several advantages :
+
+- declarations are as close as possible to the test, making the link obvious
+- and complex condition (several components involved, complex version range...) is easy to do, as it's declared as python code
+
+### Issue
+
+Unfortunatly, it comes with a major drawback: Activating a single test can become a hell due to confilct between PRs.
+
+### Proposition
+
+To solve this, we'll try to leverage two points :
+
+1. Often, those declarations involve only one component
+2. Often, a component is modified by only one developper
+
+The idea is to offer an alternative way to declare those metadata, using one file per component, also known as manifest file. Those files will be inside system tests repo, declaring for a given component some metadata.
+
+Pro :
+
+- Will solve the majority of conflict during test activation PRs
+- each team will have ownership of its manifest file, easing the PR preview process
+
+Cost :
+
+- declarations will be a little bit far away from the test, asking a small addition in the learning curve
+- re-naming will be a little bit harder, and will spam all teams
+
+## Implementation
+
+- Manifests files will be inside `utils/manifest/`, one per term, named with the argument name used in decorators (`utils/manifest/golang.yml`, `utils/manifest/agent.yml`...)
+- Each team will have ownership of its manifest file
+- pytest will load those files, and decorate tests node during collection
+
+## Supported features
+
+Will suport:
+
+- declaring a skip reason (bug, flaky, irrelevant, missing_feature) for entire file
+- declaring released version (or a skip reason) for a class
+- declaring released version (or a skip reason) for a class, with details for weblog variants
+- The wildcard `*` is supported for weblog declarations
+
+Won't support:
+
+- declaring metadata (bug, flaky, irrelevant) for test methods
+  - because their namings are not stable, it would lead to frequent modifications of manifest files, spaming every team
+  - because conflict mostly happen at class level
+
+
+## Example
+
+```yaml
+tests/:
+  specific.py: irrelevant (see this link) # let skip  an entire file
+
+  appsec/: # more regular declarations:
+    test_distributed.py:
+      Test_FeatureA: v1.14 # declare a version for a class
+
+      Test_FeatureB: flaky # skip a class with bug, flaky, irrelevant ...
+
+      Test_FeatureC: # declare a version for a class, depending on weblog
+        django: v1.2
+        flask: v1.3
+        uwsgi: bug (jira ticket) # For a weblog, skip it with bug, or flaky
+        "*": "?" # All other weblogs: not yet available
+```
+
+## Format [WIP]
+
+```json
+{
+  "type": "object",
+
+  "properties": {
+    "tests/": { "$ref": "#/$defs/folder_content" }
+  },
+
+  "$defs": {
+    "folder_content": {
+      "type": "object",
+      "patternProperties": {
+        ".*/": { "$ref": "#/$defs/folder_content" },
+        "test_.*\\.py": { "$ref": "#/$defs/file_content" }
+      }
+    },
+
+    "file_content": {
+      "oneOf": [
+        {
+          "type": "object",
+          "patternProperties": {
+            "Test_.*": {
+              "oneOf": [
+                { "$ref": "#/$defs/version" },
+                {
+                  "type": "object",
+                  "patternProperties": {
+                    ".*": { "$ref": "#/$defs/class_content" }
+                  }
+                }
+              ]
+            }
+          }
+        },
+        {
+          "$ref": "#/$defs/skipped_declaration"
+        }
+      ]
+    },
+
+    "class_content": {
+      "oneOf": [
+        { "$ref": "#/$defs/version" },
+        { "$ref": "#/$defs/skipped_declaration" }
+      ]
+    },
+
+    "version": { " type": "string", "pattern": "(v.*|\\?)" },
+
+    "skipped_declaration": {
+      "type": "string",
+      "pattern": "^(bug|flaky|irrelevant) \\(.*\\)$"
+    }
+  }
+}
+```