[RFC] Treat stdlib packages more like normal packages

[vchuravy]

This PR has the goal to allow Pkg3 to eventually manage and update stdlib packages.
In order to achieve that goal we can no longer bake the stdlib packages into the sysimage
so I had to revert #24843. The major downside to this is that it increases the load time of
stdlib packages since they are now cached as normal packages. The major upside is that stdlib
packages are now treated as normal packages and the sysimage is faster to build.

I would like to solict comments on the Makefile and contrib/stdlib_cache.jl.
(I had to add an option to tell Pkg2 to not resolve packages in order to achieve reproducible builds)

This is part of my exploration for how to get Pkg3 into Base.

[vchuravy]

Oh it also makes revise work again for stdlib packages and uncovered hidden assumptions in the tests for sparse and IterativeEigensolvers

[KristofferC]

As a data point, Pkg3 crucially needs to be baked into sysimg with precompile statements for acceptable performance. Especially since it needs to be loaded in all new precompilation processes.

[vchuravy]

Interesting right now loadtime for Pkg3 after precompilation on 0.7 is about 2.0s for me. But the first loading triggers compilation.

Especially since it needs to be loaded in all new precompilation processes.

You mean since it needs to be loaded into all processes so that using ... can resolve the right package? Or is there anything precompilation specific?

---

We have basically 3 options as far as I can tell:

1. Tie stdlib packages (or only Pkg) to the release cycle of base Julia and include them in the sysimage
2. On every update to a stdlib package or Pkg rebuild the sysimage.
3. Improve package precompilation and loading times.

In my opinion:

1. is not desirable at all since it goes against the purpose of the standard library (decoupling the release cycle from base Julia) and I think that is true for Pkg as well.
2. is workable, but quite annoying. The main open question would be what to do with Julia installation where the sysimage is no longer writable.
3. Is the best solution, but it punts the problem to the future and it is not clear how to do that. And we would have to live with slower startups times for now. (Sometimes the best way to get people to solve a problem is to include a sleep(10) in the offending code ;) )

Load times on the PR:

Name	Time
Base64	NA
CRC32c	0.000345244
Dates	0.174196897
DelimitedFiles	2.3505e-5
Distributed	6.410770025
FileWatching	0.111332444
IterativeEigensolvers	0.028202508
Logging	0.000222431
Mmap	1.0518e-5
Printf	0.000381584
Profile	0.024661583
SharedArrays	0.013978096
SuiteSparse	0.606024873
Test	0.025469835
Unicode	2.034e-6

Base64 is actually interesting since it is getting imported in the sysimg and used in base/multimedia.jl

[nalimilan]

Base64 is actually interesting since it is getting imported in the sysimg and used in base/multimedia.jl

FWIW, Unicode is actually in a similar situation, since it just reexports functions defined under Base.Unicode.

Why does Distributed take so much time to load? That sounds like the main problem to me, especially if starting workers requires loading the package.

[vchuravy]

Looking at the profile for Base.require(:Distributed) it spends all it time in jl_restore_incremental and over 50% of its time in ml_matches losts of this seems to be subtyping and typemap related.

[KristofferC]

Interesting right now loadtime for Pkg3 after precompilation on 0.7 is about 2.0s for me. But the first loading triggers compilation.

If only using time was what was interesting, things would be a lot simpler. Unfortunately, users might actually want to use Pkg3 which which means inference + compilation time is also relevant. Try pkg> status and compare with the WIP branch I put up that include precompilation statements.

You mean since it needs to be loaded into all processes so that using ... can resolve the right package? Or is there anything precompilation specific?

That, and it is likely that different Pkg environments will use different precompilation paths to avoid having to recompile everything just because of swapping environment.

[StefanKarpinski]

That, and it is likely that different Pkg environments will use different precompilation paths to avoid having to recompile everything just because of swapping environment.

I realized this week that in Pkg3, .ji files generally don't need to look at the sources of files they were loaded from since those are supposed to be immutable. They do, however, need to look at the manifest file they were loaded from.

[amitmurthy]

In case of Distributed please also check for any slowdown doing a local addprocs().

[KristofferC]

I just want to say that this looks really like the (only?) correct way forward to allow stdlib packages to live outside the Base release cycle. My point with the earlier comments is that perhaps Pkg3 is a bit special because it is interactive, so snappiness is important, and the hooks it need into Base are quite intensive.

[tknopp]

But if standard packages should live under a different release cycle, it would be necessary that the move out of JuliaLang/julia. Otherwise they cannot use the versioning implemented in Pkg2/Pkg3.

IMHO this would be the desired move. But in addition it would be important to have super easy system image compilation. Maybe we could have a second system image and the user can configure which packages should be in there. The default is all stdlib packages.