Environment is not fully reproducible

[charleskawczynski]

We have an issue where our current buildkite environment is:

- echo "--- Instantiate buildkite env"
- "julia --project=.buildkite --check-bounds=yes -e 'using Pkg; Pkg.develop(path=\".\"); Pkg.instantiate(;verbose=true); Pkg.precompile(;strict=true)'"
- "julia --project=.buildkite -e 'using CUDA; CUDA.precompile_runtime()'"

The issue here is that Pkg.develop can (may?/will?) trigger the resolver (at least we just observed this after the depot was cleared), resulting in potentially different versions being run despite our manifest file being checked in.

Over the years, I've seen Pkg.develop added and removed from pipelines, likely due to this issue. One reason why Pkg.develop is typically added is because, without it, users may update the manifest locally, and forget to Pkg.develop themselves, effectively pinning CI to that version.

I think a better pattern to use moving forwards is to:

• Remove the use of Pkg.develop on buildkite, to guarantee reproducibility
• Document how users should setup their local environment
• Add a test (whether that's via julia code or in the buildkite pipeline), that tests that the repo package (ClimaCore in this case), is developed in that environment. In Julia terms, that is testing that TOML.parse(join(readlines(".buildkite/Manifest.toml")[7:end], "\n"))["deps"]["ClimaCore"][1]["path"] == ".." (this check unfortunately may be julia version-dependent).

We should probably spread this pattern, as this can easily bite users on all of our repos.

Also, just to note, we should always print Pkg.status(), so that we can trace what's going on.