go_repository: support modules

[EvyBongers]

When I try to build the example program hdfscat, I get an error that prevents a successful build.

When attempting to build the project, I get this error:

ERROR: Analysis of target '//cmd/hdfscat:hdfscat' failed; build aborted: no such package '@com_github_colinmarc_hdfs//v2/internal/protocol/hadoop_hdfs': BUILD file not found on package path

Reviewing the code, I found that the hdfs library uses semantic import versioning. In my WORKSPACE file, I do specify to use version two of the library:

go_repository(
    name = "com_github_colinmarc_hdfs",
    importpath = "github.com/colinmarc/hdfs",
    sha256 = "36a039f827742b49ef96eab1906a5b5f35aa3715a4448608224193eada35ffd8",
    strip_prefix = "hdfs-2.0.0",
    urls = ["https://github.com/colinmarc/hdfs/archive/v2.0.0.tar.gz"],
)

However, I can't find any way to tell bazel that it should expose the library as package @com_github_colinmarc_hdfs//v2/internal/protocol/hadoop_hdfs.

Is there currently any way to solve this problem with bazel/gazelle?

[jayconrod]

This may work if you make the following change:

go_repository(
    name = "com_github_colinmarc_hdfs_v2",
    importpath = "github.com/colinmarc/hdfs/v2",
    sha256 = "36a039f827742b49ef96eab1906a5b5f35aa3715a4448608224193eada35ffd8",
    strip_prefix = "hdfs-2.0.0",
    urls = ["https://github.com/colinmarc/hdfs/archive/v2.0.0.tar.gz"],
)

However, this is something that needs to be improved in go_repository. I think we'll probably want something like this:

go_repository(
    name = "com_github_colinmarc_hdfs_v2",    
    module_path = "github.com/colinmarc/hdfs",
    module_version = "v2.0.0",
    module_sum = "01234abcd",
)

Some things which are important to note:

• name should include the major version for versions 2 or greater. The same module may be imported at different major versions, and we need a way to disambiguate.
• module_path and module_version will replace importpath. go_repository may use go get (or something equivalent) to download the module. It should respect GOPROXY.
• module_sum will be whatever would appear in the go.sum file for this module. This is a hash of the module contents, not the .zip or .tar.gz file, so we can't use sha256 here.

[jayconrod]

cc @Globegitter

Related #307
Related #394

[Globegitter]

@jayconrod Just starting to put this on my radar again, so this is essentially a blocker for #394 right? So thinking highl-level about an implementation what we would need is:

• add the new attributes as you mentioned
• when the attributes are set use the go cli to download the modules into the repository
• then just hook that up with the existing go_repository gazelle logic

These steps are how I imagine this in my head without actually having looked at the go_repository logic and I have also not looked at how the go cli downloads modules.

The next question then is should we per default download the go cli hermetically? Should there be an option to use a local go cli if the user wants? And then we of course need to make sure that things are only cached when GOPATH is set.

Does this all make sense from your point of view?

[jayconrod]

That makes sense. There are a few other considerations, too -- we should use a shared cache between go_repository instances, we should use the toolchain from go_sdk if we can, and we should make sure to respect GOPROXY.

Actually, I think I'm going to try to implement this today. This came up again in bazel-contrib/rules_go#1953, so we should really have this soon.

[commure-stabai]

I know this has been closed for a while, but I just ran into the problem described in bazel-contrib/rules_go#1953. The workaround seems to fix the issue for me, but thought it was worth calling out here.

I'm guessing the reason it wasn't covered by this fix is that the dependency I encountered this with has multiple modules in the same repo, as well as one at the root (which all the others are nested within). I'm pretty sure that practice is discouraged, but still, I want to depend on their code.