README and docs restructure

CONTRIBUTING.md

@@ -5,7 +5,6 @@ taking the time to contribute!
 
 Before starting, please take some time to familiarize yourself with the [Code of Conduct](CODE_OF_CONDUCT.md).
 
-
 ## Getting Started
 
 We welcome many different types of contributions and not all of them need a

docgen/docs.go

@@ -15,8 +15,10 @@
 package main
 
 import (
+	"bytes"
 	"flag"
-	"log"
+	"fmt"
+	"os"
 
 	"github.com/in-toto/witness/cmd"
 	"github.com/spf13/cobra/doc"
@@ -30,8 +32,27 @@ func init() {
 }
 
 func main() {
-	// Generate CLI docs
-	if err := doc.GenMarkdownTree(cmd.New(), directory); err != nil {
-		log.Fatalf("Error generating docs: %s", err)
+	mdContent := "# Witness CLI Reference\n\nThis is the reference for the Witness command line tool, generated by [Cobra](https://cobra.dev/).\n\n"
+	// Generate markdown content for all commands
+	for _, command := range cmd.New().Commands() {
+		// We are not generating docs for the completion command right now, as it doesn't render in Markdown correctly
+		if command.Use == "completion [bash|zsh|fish|powershell]" {
+			continue
+		}
+
+		buf := new(bytes.Buffer)
+		err := doc.GenMarkdown(command, buf)
+		if err != nil {
+			fmt.Println("Error generating markdown for command:", command.Use)
+			continue
+		}
+		mdContent += buf.String()
+	}
+
+	// Write the combined markdown content to a file
+	err := os.WriteFile(fmt.Sprintf("%s/commands.md", directory), []byte(mdContent), 0644)
+	if err != nil {
+		fmt.Println("Error writing to file:", err)
+		os.Exit(1)
 	}
 }

docgen/verify.sh

@@ -19,7 +19,7 @@ set -e
 # Verify that generated Markdown docs are up-to-date.
 tmpdir=$(mktemp -d)
 tmpdir2=$(mktemp -d)
-cp docs/witness*.md "$tmpdir2/"
+cp docs/commands.md "$tmpdir2/"
 go run ./docgen --dir "$tmpdir"
 echo "###########################################"
 echo "If diffs are found, run: make docgen"

docs/about/how-witness-works.md

@@ -0,0 +1,11 @@
+# How Witness Works
+
+### Signing
+Witness is able to observe your software development life-cycle (SDLC) by wrapping around commands executed within them. By passing any command to Witness as an argument, the tool is able to understand what was executed but also on what infrastructure, by what user or service account and more. The information that Witness gathers while the command is running is down to which [Attestors](docs/attestor.md) are used. Attestors are implementations of an interface that find and assert facts about the system Witness is running on (e.g., [AWS Attestor](docs/attestors/aws-iid.md)). Finally, Witness can compile this information into an [in-toto attestation](https://github.com/in-toto/attestation), place it in a [DSSE Envelope](https://github.com/secure-systems-lab/dsse) and sign that envelope with the key that was supplied by the user. 
+
+### Storing
+For storage, the Witness project can upload signed attestations to an [Archivista](https://github.com/in-toto/archivista) server, a graph and storage service for in-toto attestations. This enables the discovery and retrieval of attestations for verification of software artifacts.
+
+### Verifying
+Witness allows users to verify the attestations that they generate by providing the `witness verify` command. To achieve this, Witness uses a [policy file](./docs/policy.md) defined by the user to check for presence of the expected attestations and that they were signed by the appropriate functionaries (Public keys or roots of trust that are trusted to sign certain types of attestation). To verify the attestation body itself, Witness supports defining [OPA Rego](https://www.openpolicyagent.org/docs/latest/policy-language/) policies inside the policy file. This allows users to ensure the facts asserted by the Attestors are reported expected.
+

docs/commands.md

@@ -0,0 +1,175 @@
+# Witness CLI Reference
+
+This is the reference for the Witness command line tool, generated by [Cobra](https://cobra.dev/).
+
+## witness run
+
+Runs the provided command and records attestations about the execution
+
+```
+witness run [cmd] [flags]
+```
+
+### Options
+
+```
+      --archivista-server string                      URL of the Archivista server to store or retrieve attestations (default "https://archivista.testifysec.io")
+  -a, --attestations strings                          Attestations to record ('product' and 'material' are always recorded) (default [environment,git])
+      --attestor-product-exclude-glob string          Pattern to use when recording products. Files that match this pattern will be excluded as subjects on the attestation.
+      --attestor-product-include-glob string          Pattern to use when recording products. Files that match this pattern will be included as subjects on the attestation. (default "*")
+      --enable-archivista                             Use Archivista to store or retrieve attestations
+      --hashes strings                                Hashes selected for digest calculation. Defaults to SHA256 (default [sha256])
+  -h, --help                                          help for run
+  -o, --outfile string                                File to which to write signed data.  Defaults to stdout
+      --signer-file-cert-path string                  Path to the file containing the certificate for the private key
+      --signer-file-intermediate-paths strings        Paths to files containing intermediates required to establish trust of the signer's certificate to a root
+  -k, --signer-file-key-path string                   Path to the file containing the private key
+      --signer-fulcio-oidc-client-id string           OIDC client ID to use for authentication
+      --signer-fulcio-oidc-issuer string              OIDC issuer to use for authentication
+      --signer-fulcio-oidc-redirect-url string        OIDC redirect URL (Optional). The default oidc-redirect-url is 'http://localhost:0/auth/callback'.
+      --signer-fulcio-token string                    Raw token string to use for authentication to fulcio (cannot be used in conjunction with --fulcio-token-path)
+      --signer-fulcio-token-path string               Path to the file containing a raw token to use for authentication to fulcio (cannot be used in conjunction with --fulcio-token)
+      --signer-fulcio-url string                      Fulcio address to sign with
+      --signer-spiffe-socket-path string              Path to the SPIFFE Workload API Socket
+      --signer-vault-altnames strings                 Alt names to use for the generated certificate. All alt names must be allowed by the vault role policy
+      --signer-vault-commonname string                Common name to use for the generated certificate. Must be allowed by the vault role policy
+      --signer-vault-namespace string                 Vault namespace to use
+      --signer-vault-pki-secrets-engine-path string   Path to the Vault PKI Secrets Engine to use (default "pki")
+      --signer-vault-role string                      Name of the Vault role to generate the certificate for
+      --signer-vault-token string                     Token to use to connect to Vault
+      --signer-vault-ttl duration                     Time to live for the generated certificate. Defaults to the vault role policy's configured TTL if not provided
+      --signer-vault-url string                       Base url of the Vault instance to connect to
+  -s, --step string                                   Name of the step being run
+      --timestamp-servers strings                     Timestamp Authority Servers to use when signing envelope
+      --trace                                         Enable tracing for the command
+  -d, --workingdir string                             Directory from which commands will run
+```
+
+### Options inherited from parent commands
+
+```
+  -c, --config string      Path to the witness config file (default ".witness.yaml")
+  -l, --log-level string   Level of logging to output (debug, info, warn, error) (default "info")
+```
+
+### SEE ALSO
+
+* [witness](witness.md)	 - Collect and verify attestations about your build environments
+
+## witness sign
+
+Signs a file
+
+### Synopsis
+
+Signs a file with the provided key source and outputs the signed file to the specified destination
+
+```
+witness sign [file] [flags]
+```
+
+### Options
+
+```
+  -t, --datatype string                               The URI reference to the type of data being signed. Defaults to the Witness policy type (default "https://witness.testifysec.com/policy/v0.1")
+  -h, --help                                          help for sign
+  -f, --infile string                                 Witness policy file to sign
+  -o, --outfile string                                File to write signed data. Defaults to stdout
+      --signer-file-cert-path string                  Path to the file containing the certificate for the private key
+      --signer-file-intermediate-paths strings        Paths to files containing intermediates required to establish trust of the signer's certificate to a root
+  -k, --signer-file-key-path string                   Path to the file containing the private key
+      --signer-fulcio-oidc-client-id string           OIDC client ID to use for authentication
+      --signer-fulcio-oidc-issuer string              OIDC issuer to use for authentication
+      --signer-fulcio-oidc-redirect-url string        OIDC redirect URL (Optional). The default oidc-redirect-url is 'http://localhost:0/auth/callback'.
+      --signer-fulcio-token string                    Raw token string to use for authentication to fulcio (cannot be used in conjunction with --fulcio-token-path)
+      --signer-fulcio-token-path string               Path to the file containing a raw token to use for authentication to fulcio (cannot be used in conjunction with --fulcio-token)
+      --signer-fulcio-url string                      Fulcio address to sign with
+      --signer-spiffe-socket-path string              Path to the SPIFFE Workload API Socket
+      --signer-vault-altnames strings                 Alt names to use for the generated certificate. All alt names must be allowed by the vault role policy
+      --signer-vault-commonname string                Common name to use for the generated certificate. Must be allowed by the vault role policy
+      --signer-vault-namespace string                 Vault namespace to use
+      --signer-vault-pki-secrets-engine-path string   Path to the Vault PKI Secrets Engine to use (default "pki")
+      --signer-vault-role string                      Name of the Vault role to generate the certificate for
+      --signer-vault-token string                     Token to use to connect to Vault
+      --signer-vault-ttl duration                     Time to live for the generated certificate. Defaults to the vault role policy's configured TTL if not provided
+      --signer-vault-url string                       Base url of the Vault instance to connect to
+      --timestamp-servers strings                     Timestamp Authority Servers to use when signing envelope
+```
+
+### Options inherited from parent commands
+
+```
+  -c, --config string      Path to the witness config file (default ".witness.yaml")
+  -l, --log-level string   Level of logging to output (debug, info, warn, error) (default "info")
+```
+
+### SEE ALSO
+
+* [witness](witness.md)	 - Collect and verify attestations about your build environments
+
+## witness verify
+
+Verifies a witness policy
+
+### Synopsis
+
+Verifies a policy provided key source and exits with code 0 if verification succeeds
+
+```
+witness verify [flags]
+```
+
+### Options
+
+```
+      --archivista-server string   URL of the Archivista server to store or retrieve attestations (default "https://archivista.testifysec.io")
+  -f, --artifactfile string        Path to the artifact to verify
+  -a, --attestations strings       Attestation files to test against the policy
+      --enable-archivista          Use Archivista to store or retrieve attestations
+  -h, --help                       help for verify
+  -p, --policy string              Path to the policy to verify
+      --policy-ca strings          Paths to CA certificates to use for verifying the policy
+  -k, --publickey string           Path to the policy signer's public key
+  -s, --subjects strings           Additional subjects to lookup attestations
+```
+
+### Options inherited from parent commands
+
+```
+  -c, --config string      Path to the witness config file (default ".witness.yaml")
+  -l, --log-level string   Level of logging to output (debug, info, warn, error) (default "info")
+```
+
+### SEE ALSO
+
+* [witness](witness.md)	 - Collect and verify attestations about your build environments
+
+## witness version
+
+Prints out the witness version
+
+### Synopsis
+
+Prints out the witness version
+
+```
+witness version [flags]
+```
+
+### Options
+
+```
+  -h, --help   help for version
+```
+
+### Options inherited from parent commands
+
+```
+  -c, --config string      Path to the witness config file (default ".witness.yaml")
+  -l, --log-level string   Level of logging to output (debug, info, warn, error) (default "info")
+```
+
+### SEE ALSO
+
+* [witness](witness.md)	 - Collect and verify attestations about your build environments
+

docs/concepts/attestor.md

@@ -0,0 +1,27 @@
+# Attestors
+
+A Witness attestor is a programming interface that defines an object that can assert facts about a system and store those facts in a versioned schema. An attestor has a `Name`, `Type` and `RunType`. The `Type` is a versioned string corresponding to the JSON schema of the attestation. For example, the AWS attestor is defined as follows:
+```
+  Name    = "aws"
+  Type    = "https://witness.dev/attestations/aws/v0.1"
+  RunType = attestation.PreRunType
+```
+Attestation types are leveraged to ensure the correct version schema is used when we evaluate policy against these attestations.
+
+## Attestor Security Model
+
+Attestations are only as secure as the data that feeds them. Where possible cryptographic material should be validated, evidence of validation should be included in the attestation for out-of-band validation.
+
+Examples of cryptographic validation is found in the [GCP](https://github.com/testifysec/witness/tree/main/pkg/attestation/gcp-iit), [AWS](https://github.com/testifysec/witness/blob/main/pkg/attestation/aws-iid/aws-iid.go), and [GitLab](https://github.com/testifysec/witness/tree/main/pkg/attestation/gitlab) attestors.
+
+## Attestor Life Cycle
+
+- **Pre-material:** Pre-material attestors run before any other attestors. These attestors generally collect information about the environment.
+
+- **Material:** Material attestors run after any prematerial attestors and prior to any execute attestors. Generally these collect information about state that may change after any execute attestors, such as file hashes.
+
+- **Execute:**: Execute attestors run after any material attestors and generally record information about some command or process that is to be executed.
+
+- **Product:** Product attestors run after any execute attestors and generally record information about what changed during the execute lifecycle step, such as changed or created files.
+
+- **Post-product:** Post-product attestors run after product attestors and generally record some additional information about specific products, such as OCI image information from a saved image tarball.