Would like to be able to execute build function several times, building ccache wrapers prevents this

[larroy]

Description

See comments below. Ccache is in fact working on cpu openblas builds, hence the change of title.

@marcoabreu

Environment info (Required)

What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio):

MXNet commit hash:
0a2419f

Minimum reproducible example

./dev_menu.py menu 5

This should use CCACHE

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Bug, CI

[marcoabreu]

Please provide more details around the current behaviour, the expected behaviour, how to observe/measure it and in how far the linked PR changed that.

[larroy]

@mxnet-label-bot [ci]

[marcoabreu]

My data does not show any regressions as result of merging #13456 on 14th of December.

[larroy]

It also prevents executing the build function two times. Why do we need to use these wrappers? Why not just using CC and CXX environment variables?

jenkins_slave@013086bf91e1:/work/mxnet$ /work/runtime_functions.sh build_ubuntu_cpu

• NOSE_COVERAGE_ARGUMENTS='--with-coverage --cover-inclusive --cover-xml --cover-branches --cover-package=mxnet'
• NOSE_TIMER_ARGUMENTS='--with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error'
• CI_CUDA_COMPUTE_CAPABILITIES='-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70'
• CI_CMAKE_CUDA_ARCH_BIN=52,70
• set +x
• export CC=gcc
• CC=gcc
• export CXX=g++
• CXX=g++
• build_ccache_wrappers
• set -ex
• '[' -z x ']'
• '[' -z x ']'
• mkdir /tmp/ccache-redirects
  mkdir: cannot create directory '/tmp/ccache-redirects': File exists

[marcoabreu]

It is the recommended way to use ccache - as explained previously: https://ccache.samba.org/manual/latest.html#_run_modes

[marcoabreu]

Please feel free to submit a PR to make a conditional mkdir.

[marcoabreu]

It is required because underlying build-systems don't respect the wrapper methods.

[larroy]

Ok maybe it's calling ccache but it's not obvious because of the wrappers, it's more obvious when the command includes ccache.

Reproduction with:

docker         run         --cap-add         SYS_PTRACE         --rm         --shm-size=500m         -v         /home/piotr/mxnet:/work/mxnet         -v         /home/piotr/mxnet/build:/work/build         -v         /home/piotr/.ccache:/work/ccache         -u         1001:1001         -e         CCACHE_MAXSIZE=500G         -e         CCACHE_TEMPDIR=/tmp/ccache         -e         CCACHE_DIR=/work/ccache         -e         CCACHE_LOGFILE=/tmp/ccache.log         -ti         mxnetci/build.ubuntu_cpu

/work/runtime_functions.sh build_ubuntu_cpu

/work/runtime_functions.sh build_ubuntu_cpu

I expect to be able to execute /work/runtime_functions.sh build_ubuntu_cpu several times.

But seems ccache is in fact working.

[larroy]

@marcoabreu there's two modes to run it according to the documentation:

"There are two ways to use ccache. You can either prefix your compilation commands with ccache or you can let ccache masquerade as the compiler by creating a symbolic link (named as the compiler) to ccache. "

[larroy]

It is required because underlying build-systems don't respect the wrapper methods.

In which case? which build system?

[marcoabreu]

We already had this discussion including all background details in #13456

[larroy]

@marcoabreu please list the submodules and dependencies which have these hardcoded paths, you didn't respond to the latest question by @roywei in the PR you linked ( #13456 ) I don't see we have a direct dependency on zeromq but only through a shell script. I don't think this is a valid reason to require root in all the build functions and override the compiler globally.

I don't see any hardcoded path in PS-Lite, please add the data that you mention in the linked PR and avoid putting the burden of proof in contributors to the project or at least provide a path for proper fix.

Could you also make this data about build time regressions publicly available or provide instructions to views it so other contributors can take it into consideration?