Working around NATs 64M entry size limit

[lumjjb]

Due to the limits of NATs having a data limit of 64MB, we can't pass in documents through the regular collector flow. In the longer term, we want to have a solution that will be able to pass a reference to a document with a temporary data store that can be pointed to in the NATs entry. However, in the meantime, the alternative is that for file ingestion, we will do the ingestion as part of the binary and talk directly to the GraphQL mutation API. I.e., we will use guacone files for file ingestion which will do ingestion (parsing) locally.

[lumjjb]

Design doc: https://docs.google.com/document/d/1NL6KOvoD8uj_caACH3S8LNIAOKbGOCaWYo5lMwq7yt0/edit

[lumjjb]

For this issue, in the time being to not add in any additional complexity, we have a short term gap close by having all the collectors mimic guacone and talk directly to the graphQL endpoint #1104

[lumjjb]

Another proposal that came up last community call from @dejanb is to have an API to handle the documents directly by the ingestor, this would not introduce any new running services.

[pxp928]

Based on further discussion, the decision was made to use a blob store (via cloud provider) to store SBOMs and emit an entry to the URI in the pub/sub queue. For quick demos and testing, the pub/sub will not be used to allow for quick usability without the overhead of the blob store.

Tasks based on these:

• Remove nats, guaccollect, and guacIngest from docker-compse (for quick way to run without having to setup S3 or NATS)
• Add all collectors to both guacone and guacCollect (need to check which ones are missing from each)
  • Add deps.dev collector into guacone
• Provide instructions for setting up S3 (or cloud-provided storage) and connect via go-cloud for development or production environment using NATS
  • Go-cloud does support file system (so can be used for testing)
  • All auth is environment-based
  • Go-cloud wraps around messaging service
  • https://gocloud.dev/howto/pubsub/publish/
  • https://gocloud.dev/howto/pubsub/subscribe/
• Replace NATS with go-cloud to allow for other messaging services to be used (for now can keep NATS)
• NATS message structure
  • CD Events: Add an SBOM URI field to artifact events cdevents/spec#171
  • Reference an item in a bucket
• Open Questions: Do we delete the SBOMs once they are ingested or keep them?

[afrittoli]

FYI - cdevents/spec#171 is now merged, I hope we'll be able to release v0.4 in the next month with the SBOM URI in.

[lumjjb]

This is fixed via #1650 #1630