Skip to content

Commit

Permalink
go: use module sources digest for each of its packages (#17728)
Browse files Browse the repository at this point in the history
## Problem

As reported in #17592, third-party Go modules are allowed to assume that all files packaged with the module will be visible during compilation. In that particular issue, the [Confluent Kafka client library](https://github.com/confluentinc/confluent-kafka-go) uses Cgo and expected a subdirectory with files used during C compilation will be visible to the package using Cgo.

Pants, however, subsets the modules sources digest to just the files in the exact subdirectory for each of the module's packages. This means that the [subdirectory files](https://github.com/confluentinc/confluent-kafka-go/tree/master/kafka/librdkafka_vendor) were not visible when compiling [the package in confluent-kafka-go in the immediate parent directory](https://github.com/confluentinc/confluent-kafka-go/tree/master/kafka/).

## Solution

Since Pants cannot predict how a third party package will use the files in the modules sources, Pants cannot reasonably subset the modules source digest. Solve the issue by not subsetting the module sources digest to produce a package-specific digest.


This is a partial fix for #17592.
  • Loading branch information
tdyas authored Dec 9, 2022
1 parent 74ac7aa commit 39ee73a
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 15 deletions.
17 changes: 3 additions & 14 deletions src/python/pants/backend/go/util_rules/third_party_pkg.py
Original file line number Diff line number Diff line change
Expand Up @@ -430,20 +430,8 @@ async def analyze_go_third_party_package(
"the third-party module."
)

package_digest = await Get(
Digest,
DigestSubset(
request.module_sources_digest,
PathGlobs(
[os.path.join(request.package_path, "*")],
glob_match_error_behavior=GlobMatchErrorBehavior.error,
description_of_origin=f"the analysis of Go package {import_path}",
),
),
)

analysis = ThirdPartyPkgAnalysis(
digest=package_digest,
digest=request.module_sources_digest,
import_path=import_path,
name=request.pkg_json["Name"],
dir_path=request.package_path,
Expand Down Expand Up @@ -485,7 +473,8 @@ async def analyze_go_third_party_package(
Get(Digest, CreateDigest([FileContent("patterns.json", patterns_json)])),
)
input_digest = await Get(
Digest, MergeDigests((package_digest, patterns_json_digest, embedder.digest))
Digest,
MergeDigests((request.module_sources_digest, patterns_json_digest, embedder.digest)),
)
embed_result = await Get(
FallibleProcessResult,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -170,9 +170,10 @@ def assert_pkg_info(
assert pkg_info.go_files == go_files
assert not pkg_info.s_files
snapshot = rule_runner.request(Snapshot, [pkg_info.digest])
assert set(snapshot.files) == {
expected_files = {
os.path.join(dir_path, file_name) for file_name in (*go_files, *extra_files)
}
assert expected_files.issubset(snapshot.files)
assert pkg_info.minimum_go_version == minimum_go_version

assert_pkg_info(
Expand Down

0 comments on commit 39ee73a

Please sign in to comment.