Proposal: cmd/go: parallel execution of //go:generate

[tejasmanohar]

This is a proposal to add a -parallel <n> flag to go generate that runs //go:generate commands concurrently.

If -paralell is provided without a number, concurrency would be unbounded. If parallel is provided with an integer , that would be the max concurrent procs. If -parallel is not supplied, //go:generate commands are run serially as they are now.

If there's mutual interest, I can submit code for review, but I wanted to get feedback first.

[mvdan]

Out of curiosity, why do you need this? go generate is something that you don't run often, so performance isn't particularly important so to speak.

Also worth noting that you could handle parallelization in a script, and then do //go:generate ./parallel.sh.

[tejasmanohar]

Personally, I think it's nice to have things fast, kinda like go build. Sometimes, to speed things up, I go find specific //go:generate statements and just run them, but it'd be nice to be able to just run it for the whole project without waiting a long time. We also run go generate in CI to keep things up-to-date, and this slows down the builds.

Your solution would work, but it's nice to keep //go:generate comments next to the affected code (e.g. //go:generate mockery ... above type declaration for struct being mocked), and parallel.sh doesn't allow for that. Plus, this would work for anyone with Go installed rather than being platform-specific.

[robpike]

I understand the desire but I am not keen to complicate the simple idea that is go generate. I use go generate often, but have never felt it held me back and never wanted parallelism. If you can demonstrate a widespread need for this, I might change my mind, but at the moment the need for parallel execution seems unlikely to be a common issue.

[tejasmanohar]

Fair, maybe it makes sense to keep the issue open for a while to see if others are interested?

[valyala]

We also run go generate in CI to keep things up-to-date, and this slows down the builds.

go generate shouldn't be run in CI if you want quick reproducible builds, i.e. when all the builds on developer machines must be identical to staging builds and to prod builds. go generate during CI may result in non-reproducible builds after the corresponding tool used in go generate is updated in the CI environment.

All the output files that go generate produces must be generated and put to the repository by the developer who modifies the corresponding source files.

[tejasmanohar]

@valyala While this may often be the case, go generate running in CI doesn't imply you go get the tools in CI. The tools could be in the same repository or vendored (see retool) in a way that the versions are consistent between developers, CI, etc.

Example use case-- I've created a tool that generates typed RPC clients given server code called glue. At Segment, we run glue via go generate in CI as a part of some RPC services' release process to ensure the API clients are kept up-to-date. Through git, CI only commits if there's a diff (some releases may not result in API changes).

[mvdan]

Plus, this would work for anyone with Go installed rather than being platform-specific.

Well, if a script is not portable enough, you can always write a small Go file and do go:generate go run ....

I see your point that this slightly complicates your use-case, but it seems like there's no consensus on this being a problem that is widespread enough to warrant adding complexity to go generate.

Are you sure your go generate work would get considerably faster if it ran in parallel, though? It's worth noting that CI services usually limit the amount of resources (e.g. CPU usage and cores) available, so I would be surprised if -parallel cut the run-time considerably.

[rsc]

Sorry, but no. The design decision was made that they run in serial. It's fundamental to how people use it.

[tejasmanohar]

Got it. I understand the concern around complexity and will write an adhoc wrapper.

It's fundamental to how people use it.

Why @rsc? I don't see the connection.

[rsc]

It's allowed to write a sequence of //go:generate lines in which each builds on a step taken by the previous one.

[tejasmanohar]

Ah. We could parallelize on file then because I assume there's no file ordering guarantee. That said, yeah, that sounds too complex to think about. I'll have a wrapper for my own use 👍

[myitcv]

@tejasmanohar

From go generate -help:

Generate processes packages in the order given on the command line,
one at a time. If the command line lists .go files, they are treated
as a single package. Within a package, generate processes the
source files in a package in file name order, one at a time. Within
a source file, generate runs generators in the order they appear
in the file, one at a time.

The order of execution is specified for go generate.

[tejasmanohar]

@myitcv Got it... though I hope people don't commonly depend on that, would be unintuitive in most cases IMO :)

[myitcv]

@tejasmanohar I think it would be more unintuitive for people not to depend on the specified behaviour of go generate!

[tejasmanohar]

@myitcv I'm not suggesting we remove the ordering guarantee. I agree that it's too late for simply adding parallelization since this is documented... but I do not think there are many good use cases for relying on the file ordering guarantee of go generate (that would not be confusing to most readers).

That said, I'm happy to be proven wrong so let me know if you have good examples. The only reasonable use cases I can think of is in some sort of migration setup where you have date prefixed filenames. Otherwise, I think it's room for confusion 😬 .

[cznic]

but I do not think there are many good use cases for relying on the file ordering guarantee of go generate

go generate == extract script lines from *.go source files in well defined order and execute the resulting script.

Rarely one expects a script with no control flow constructs to not be executed top-down.

[tejasmanohar]

relying on the file ordering guarantee

@cznic Agreed. I meant ordering between files (e.g. if a.go or b.go's commands are run first when you go generate ./...). (Though I do think it's intuitive that if you supply a -parallel flag, ordering is not considered.)