support OpenMP with system macOS toolchain

[kevinushey]

With R 4.0.0, R (as well as packages on CRAN) will be built with the system toolchain (Apple Clang), which unfortunately mean OpenMP support is not provided natively by the compiler. Fortunately, it is still possible to use OpenMP with the system toolchain if you:

1. Instruct clang to enable OpenMP with -Xpreprocessor -fopenmp,
2. Explicitly provide include paths to libomp during compilation,
3. Explicitly provide library paths to libomp during link.

Unfortunately, data.table fails to detect this case as it invokes the compiler with just -fopenmp here:

data.table/configure

Line 64 in b1b1832

printf "#include <omp.h>\nint main () { return omp_get_num_threads(); }" | ${CC} ${CFLAGS} -fopenmp -xc - >/dev/null 2>&1 || R_NO_OPENMP=1;

It would be helpful for users on macOS if support for OpenMP on macOS could be detected + used.

Sorry, I recognize this is kind of a pain but figured it was worth filing since OpenMP is somewhat vital to data.table's performance.

See also:

http://r-sig-mac.29524.n8.nabble.com/R-SIG-Mac-Apple-Clang-does-support-OpenMP-if-libomp-is-available-td458.html#a461

(+ some other recent discussion on R-SIG-Mac)

[MichaelChirico]

I think Jim Hester raised this same issue, let me see...

I see, it was on the orgs page... good to have this in the issue tracker

https://github.com/orgs/Rdatatable/teams/project-members/discussions/34

[krivard]

I am pasting my working version of Makevars here with edits from the version in the install docs marked, so that google may find it and help a future traveller:

# Added: -Xpreprocessor
CC=/usr/bin/clang -Xpreprocessor -fopenmp

# Added: -Xpreprocessor -lomp -I/usr/local/include
CXX=/usr/bin/clang++ -Xpreprocessor -fopenmp -lomp -I/usr/local/include
# -O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe

# Added: -lomp
LDFLAGS=-L/usr/local/opt/gettext/lib -L/usr/lib -Wl,-rpath,/usr/lib -lomp
CPPFLAGS=-I/usr/local/opt/gettext/include -I/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include

macOS Mojave, 10.14.6

with thanks to these two threads on StackOverflow

[dbrookeUAB]

Since @krivard last post was super helpful for me to get things working, here is my working version of Makevars

# 2020-11-15 Added:  -Xpreprocessor -fopenmp 
CC=/usr/local/Cellar/llvm/11.0.0/bin/clang -Xpreprocessor -fopenmp
# 2020-11-15 Added:   -Xpreprocessor -fopenmp -lomp -I/usr/local/include
CXX=/usr/local/Cellar/llvm/11.0.0/bin/clang++  -Xpreprocessor -fopenmp -lomp -I/usr/local/include
CXX11=/usr/local/Cellar/llvm/11.0.0/bin/clang++
CXX14=/usr/local/Cellar/llvm/11.0.0/bin/clang++
CXX17=/usr/local/Cellar/llvm/11.0.0/bin/clang++
CXX1X=/usr/local/Cellar/llvm/11.0.0/bin/clang++
# 2020-11-15 Added:  CFLAGS
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
# 2020-11-15 Added:  CXXFLAGS
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/usr/local/Cellar/llvm/11.0.0/lib
SHLIB_OPENMP_CFLAGS= -fopenmp
SHLIB_OPENMP_CXXFLAGS= -fopenmp

Also Mac Mojave 10.14.6

Please reply if you find anything about this that could be improved or if you find something redundant. I just change lines on a whim because I'm too lazy to spend the time really learning the ins and outs of the Makevars file.

UPDATE
I was able to get macOS clang to compile with OpenMP for my MacBook Pro (18,2 M1 Max) running macOS Ventura 13.3 with only this for Makevars.

CC=clang
CFLAGS += -Xclang -fopenmp
LDFLAGS += -lomp

I did have to download the libomp. Specifically, LLVM 14.0.6.

Voilà!

> install.packages('data.table', type = 'source')
> library(data.table)
data.table 1.14.8 using 5 threads (see ?getDTthreads).  Latest news: r-datatable.com

PS...
CC=clang -Wno-nullability-completeness was nice.

[jangorecki]

@kevinushey could you please check if 1.13.2 resolves the problem for you? There was a change in #4735

4. User-supplied PKG_LIBS and PKG_CFLAGS are now retained and the suggestion in https://mac.r-project.org/openmp/; i.e.,
   PKG_CPPFLAGS='-Xclang -fopenmp' PKG_LIBS=-lomp R CMD INSTALL data.table_<ver>.tar.gz
   has a better chance of working on Mac.

[kevinushey]

Looks like this works for me as well. It sounds like the issue is ultimately resolved, then? Unless data.table wants to try and automagically enable OpenMP on macOS.

[mattdowle]

@kevinushey Great to hear it works.

Unless data.table wants to try and automagically enable OpenMP on macOS.

That's an interesting idea. I don't have macOS and was waiting to see how macOS users got on with the latest version. I got the impression from https://mac.r-project.org/openmp/ that there are so many problems with macOS and OpenMP that there is not really much we can do. But, now that you said this, perhaps we could have another go. That would mean that the CRAN binary would be OpenMP-enabled again, iiuc? That would be great. If so, what exactly works for you: simply passing -Xclang -fopenmp and that's all that's needed?

[kevinushey]

Right, and I can see arguments both ways:

1. On one hand, the use of OpenMP with Apple Clang isn't officially supported, and so for that reason data.table shouldn't try to enable it by default (since in theory Apple could pull the rug out from underneath us at any time);

2. On the other hands, as far as I know no users have reported issues when trying to use OpenMP with Apple Clang + data.table anyhow, so even if it's not officially supported it's still functional as far as it's used in data.table, so perhaps it's worth (trying to) enable by default.

I don't have a strong opinion on what the best way forward is, though.

[mattdowle]

If data.table makes an attempt in the configure script, and performs the attempt correctly by falling back to no OpenMP, then it looks like we're agreeing on option 2 then. It's the performing-the-attempt-correctly part that is tricky cross-platform. Did you see the comment I put into configure -- I'm not sure that attempt (testing to see if manually adding -fopenmp works) has ever worked as intended since it was put in a few versions ago.

[kevinushey]

I think you're right (sorry about that). It did ultimately work insofar that OpenMP was disabled by default on macOS, since the "default" OpenMP flags wouldn't work there. But it didn't help in turning OpenMP on.

Regardless, I think getting this right would mean redoing that part of the configure script, to keep track of + set the OpenMP compiler flags as appropriate in the Makevars file.

[mattdowle]

@kevinushey I merged #4707 but now I fear that change is going to break it on macOS as it will go back to using the "default" OpenMP flags. Do you know the appropriate way to get the "-fopenmp" passed through?

[kevinushey]

The main problem with the current approach, I think, is that $(SHLIB_OPENMP_CFLAGS) isn't actually defined to anything useful with CRAN R installations on macOS, so data.table's attempts to enable OpenMP will fail regardless.

To make things even more challenging, R CMD config does not allow you to query SHLIB_OPENMP_CFLAGS or its friends, so we don't have a straightforward way of asking R how it's configured to ask for OpenMP. 😞

I think the comments in my original post are still correct; to enable OpenMP support, data.table needs to adjust how OpenMP is requested specifically for macOS, which unfortunately requires relatively more configure gunk + perhaps some extra affordances in src/Makevars.in.

[mattdowle]

Thanks. Yes:

which unfortunately requires relatively more configure gunk + perhaps some extra affordances in src/Makevars.in

According to https://mac.r-project.org/openmp/, how about adding -Xclang -fopenmp to PKG_CFLAGS and adding -lomp to PKG_LIBS ?

[kevinushey]

That looks good to me, especially since it has (some form of) R-Core's official blessing.

[mattdowle]

Great. If I do the PR would you be able to test? (I don't have a Mac).

[kevinushey]

Sure, I'd be happy to.

[MichaelChirico]

@kevinushey any chance you're familiar enough with how the fix might go here to take up a PR here? It looks like it would benefit a lot of people.

[MichaelChirico]

I do see some other packages on CRAN with configure files reminiscent of the discussion above:

https://github.com/search?q=org%3Acran+path%3Aconfigure+%2F%5B-%5Dfopenmp%2F+%2F%5B-%5DXclang%2F&type=code

Picking one from the hat: @jameslamb I see {lightgbm} looks to possibly be attempting OpenMP setup in its configure, are you familiar with that / is it working well for you?

The PR introducing that part of configure was pretty heavy: microsoft/LightGBM#3188

[jameslamb]

👋🏻 I'm very familiar with that. It's been working well for me on my macbook and in LightGBM's continuous integration. And we haven't received user reports about OpenMP not being found, but that might just be because no one's noticed.

It does seem like OpenMP is not being found successfully in our CRAN builds:

* installing *source* package ‘lightgbm’ ...
...
checking whether OpenMP will work in a package... no
***********************************************************************************************
 OpenMP is unavailable on this macOS system. LightGBM code will run single-threaded as a result.
 To use all CPU cores for training jobs, you should install OpenMP by running

     brew install libomp
***********************************************************************************************
configure: creating ./config.status

https://www.r-project.org/nosvn/R.check/r-release-macos-x86_64/lightgbm-00install.html

So LightGBM's approach probably needs some tweaks to work with CRAN's setup.

Some other discussions you might find useful:

• [ci] fix R-package CI jobs and compatibility with OpenMP 15+ (fixes #5549, #5562) microsoft/LightGBM#5563
• OpenMP detection is still broken on MacOS dmlc/xgboost#8946

You can @ me any time here with questions, I'd be happy to try to help.

[kevinushey]

That reminds me that it's also possible to submit a package tarball for testing at https://mac.r-project.org/macbuilder/submit.html; perhaps that could be used to further verify whether or not the PR in #6034 successfully detected OpenMP support on CRAN?

[MichaelChirico]

Can anyone confirm this can be closed? Or do we need to wait for a CRAN release?

[ben-schwen]

I just uploaded it to macbuilder. See here for the results.

According to this chunk of main.Rout it looks good

> test.data.table()  # runs the main test suite of 5,000+ tests in /inst/tests/tests.Rraw
getDTthreads(verbose=TRUE):
  OpenMP version (_OPENMP)       201811
  omp_get_num_procs()            8
  R_DATATABLE_NUM_PROCS_PERCENT  unset (default 50)
  R_DATATABLE_NUM_THREADS        unset
  R_DATATABLE_THROTTLE           unset (default 1024)
  omp_get_thread_limit()         2147483647
  omp_get_max_threads()          8
  OMP_THREAD_LIMIT               unset
  OMP_NUM_THREADS                unset
  RestoreAfterFork               true
  data.table is using 4 threads with throttle==1024. See ?setDTthreads.
test.data.table() running: /Volumes/PkgBuild/work/1720650797-9b56f65b10b20c0e/packages/big-sur-arm64/results/4.4/data.table.Rcheck/data.table/tests/tests.Rraw

Thanks again @kevinushey