Heavy Inefficiency on Raspberry Pi like Computers due to lack of Multi-core Support

[martinvahi]

Usually the bottle-neck is HDD/Flash speed, but in the case of the warc-proxy the command "top" shows on 4-core Raspberry Pi 3 that the "waiting" part is about a few %, while the python process that runs the warc-proxy consumes only slightly more than 1 CPU-core. The rest of the 2-point-something cores sit idle.

An initial workaround might be to read the number of cores from the

/proc/cpuinfo

and then start that many very-low-level threads, one thread per importable WARC-file. A more proper solution is to try to prepare for utilizing hundreds of CPU-cores (archival copy).