bootstrap spawns dozens of python processes simultaneously, enough to freeze a computer for a few seconds

[cyrilbouvier]

Steps To Reproduce

Run make

Expected Behavior

My machine should still be usable while SageMath is building

Actual Behavior

At the beginning of the make command, my machine becomes unresponsive for a few seconds. If I run top while running, I can clearly see dozens and dozens of python processes all appearing at the same time.

Additional Information

I think this come from the last line of src/doc/bootstrap :

sage-package list --has-file SPKG.rst | OUTPUT_DIR=$OUTPUT_DIR OUTPUT_RST=1 xargs -P 99 -n 1 sage-spkg-info

which will call sage-spkg-info 99 times simultaneously.
I think it should be -P 1 or, if possible, the user could overwrite this value (with 1 as default).

Environment

• OS: Debian 11
• Sage Version: 10.5.beta6

Checklist

• I have searched the existing issues for a bug report that matches the one I want to file, without success.
• I have read the documentation and troubleshoot guide

[fchapoton]

this was done in #37104

[orlitzky]

this was done in #37104

Actually, it used to be even slower. xargs -P 0 means run them all at the same time. Now at least you are limited to 99 processes!

[orlitzky]

I was a little surprised that parallel processing has such an effect on this, since it's all I/O.

With -P4:

real	2m38.781s
user	9m0.266s
sys	1m19.429s

With -P1:

real	7m25.291s
user	6m19.946s
sys	1m5.246s

With -P99:

real	3m5.799s
user	10m51.435s
sys	1m19.453s

So -P<nprocs> is probably the "right" number. It still wastes a lot of CPU time, but it saves real time. What is really outrageous though is that I am spending three minutes copying 1.8 megabytes of text files into a directory.

[user202729]

There's also #38950 .

But the monolithic build system is supposedly "going away soon" (to be replaced with meson) anyway. (realistically it's not going away any time soon.)

As a workaround you can do like taskset -c 0-[number of processors minus 2] make to leave the last processor free to do other work.

---

@orlitzky Conveniently there is sage-build-num-threads script which "replaces" the nprocs, but I don't know if it's available yet at bootstrap time.

And it is indeed inefficient to spend 3 minutes copying 1.8 MB of text files. Yet because bash is quite inefficient and you're launching hundreds of bash processes, it may not be that surprising after all.