Enable custom builds of OpenTelemetry Collector to more easily include smartagent extension/receiver

[evantorrie]

The current directory and package structure of this repo makes it very difficult to build a custom OpenTelemetry Collector distribution that includes the SignalFX-specific collector modules such as smartagent receiver/extension.

The recommended method to build a custom distribution of the Otel Collector is to use the project-provided ocb with a configuration .yaml file that lists the Go modules/packages to be built, e.g.

receivers:
  - gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/signalfxreceiver v0.56.0"

We would like to use ocb to build our own OpenTelemetry Collector distribution as we require a company-internal plugin. However, we'd still like to use Splunk Observability Platform for our k8s observability solution. Doing so with full infrastructure monitoring requires smartagent receivers and plugins to be built into the Otel Collector binary.

Attempting to build a distribution that includes the smartagent receiver is not easy because

1. It is located underneath the internal/ subdirectory and cannot be imported by a different module as generated by ocb.
2. If the above issue were fixed by moving the relevant subdirectories to say, a plugin/ directory, it is still onerous to build via ocb since it lives underneath the go.mod module defined at the top level of the repo. This includes every dependency required for building the Splunk otel collector, many of which should be unnecessary for building just the extension or receiver plugin. In contrast, the signalfx receiver in the opentelemetry-collector-contrib repo has its own directory local go.mod which allows the ocb build configuration yaml to specify a direct dependency on just that module.
3. If the above issue were addressed by giving each plugin subdirectory its own go.mod module, it would also require adding a git tag for each module defined inside this repo, since that allows the Go module subsystem to refer directly to a specific tagged version of a github subdirectory, e.g. https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/tag/receiver%2Fsignalfxreceiver%2Fv0.56.0

Please consider reorganizing the directory and module structure to make this task easier.

[aunshc]

Hello @evantorrie , thank you creating this issue with a detailed walkthrough of what you are trying to do and the challenges you are running into to create your own OpenTelemetry Collector distribution. We are currently discussing the implications of the changes involved to make the above task easier and will circle back with a response as soon as we can.

[evantorrie]

Related PR: #2141

[atoulme]

This is done. More work may be required to support builds with the dependencies to signalfx-agent internal packages, and we will work to shore up support for those. Closing.