go: use module sources digest for each of its packages

[tdyas]

Problem

As reported in #17592, third-party Go modules are allowed to assume that all files packaged with the module will be visible during compilation. In that particular issue, the Confluent Kafka client library uses Cgo and expected a subdirectory with files used during C compilation will be visible to the package using Cgo.

Pants, however, subsets the modules sources digest to just the files in the exact subdirectory for each of the module's packages. This means that the subdirectory files were not visible when compiling the package in confluent-kafka-go in the immediate parent directory.

Solution

Since Pants cannot predict how a third party package will use the files in the modules sources, Pants cannot reasonably subset the modules source digest. Solve the issue by not subsetting the module sources digest to produce a package-specific digest.

This is a partial fix for #17592.

[benjyw]

Thanks for fixing!

I'm curious why we ever did this subsetting to begin with?

[tdyas]

Because at the time I had the notion that package at a time compilation could be best served by only exposing the files in a package so Pants was not materializing the whole module's files for each package.

In hindsight this performance optimization compromised correctness for some modules.

[Eric-Arellano]

Thanks! Given the perf cost of subsetting, this might end up being a wash for performance. Either way, you're right about correctness mattering more.