caret crashes R when data.table is loaded

[GeraldCNelson]

R crashes when running the example code in a shell with the following error message. In Rstudio, Rstudio itself crashes.

OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.
Abort trap: 6

Minimal, reproducible example:

library(data.table)
library(caret)

dt1 <- data.table(replicate(5,sample(0:100,50,rep=TRUE)))
dt2 <- data.table(replicate(5,sample(0:100,50,rep=TRUE)))
dtList <- c("dt1", "dt2")
dtList <- mget(dtList)
mergedDT <- Reduce(function(...) merge(..., all = TRUE), dtList)

Session Info:

> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.2

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.4.3

Before running this example, I deleted caret and all packages it depends on and then installed from source where possible. I also installed the OpenMP compiler directions for Mac at https://github.com/Rdatatable/data.table/wiki/Installation#openmp-enabled-compiler-for-mac.

And I replaced code in my Makevars file with the following.

LLVM_LOC = /usr/local/opt/llvm
CC=$(LLVM_LOC)/bin/clang -fopenmp
CXX=$(LLVM_LOC)/bin/clang++
# -O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/usr/local/opt/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib
CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include

[topepo]

Thanks for adding this here. It makes it easier to track.

Can you install the sessioninfo package and send the results of session_info() after loading caret and data.table (but before running the code)?

Also, I would suggest adding -fopenmp only when you know that the library can benefit from it. caret cannot but I don't know about data.table.

I cannot reproduce with:

> session_info()
─ Session info ───────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.4.3 (2017-11-30)
 os       macOS High Sierra 10.13.2   
 system   x86_64, darwin15.6.0        
 ui       X11                         
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       America/New_York            
 date     2018-01-06                  

─ Packages ───────────────────────────────────────────────────────────────────
 package      * version    date       source                         
 assertthat     0.2.0      2017-04-11 CRAN (R 3.4.0)                 
 bindr          0.1        2016-11-13 CRAN (R 3.4.0)                 
 bindrcpp       0.2        2017-06-17 CRAN (R 3.4.0)                 
 broom          0.4.3      2017-11-20 CRAN (R 3.4.3)                 
 caret        * 6.0-78     2017-12-10 CRAN (R 3.4.3)                 
 class          7.3-14     2015-08-30 CRAN (R 3.4.3)                 
 clisymbols     1.2.0      2017-05-21 CRAN (R 3.4.0)                 
 codetools      0.2-15     2016-10-05 CRAN (R 3.4.3)                 
 colorspace     1.3-2      2016-12-14 CRAN (R 3.4.0)                 
 CVST           0.2-1      2013-12-10 CRAN (R 3.4.0)                 
 data.table   * 1.10.4-3   2017-10-27 CRAN (R 3.4.2)                 
 ddalpha        1.3.1      2017-09-27 CRAN (R 3.4.2)                 
 DEoptimR       1.0-8      2016-11-19 CRAN (R 3.4.0)                 
 dimRed         0.1.0      2017-05-04 CRAN (R 3.4.0)                 
 dplyr          0.7.4      2017-09-28 CRAN (R 3.4.2)                 
 DRR            0.0.2      2016-09-15 CRAN (R 3.4.0)                 
 foreach        1.4.4      2017-12-12 CRAN (R 3.4.3)                 
 foreign        0.8-69     2017-06-22 CRAN (R 3.4.3)                 
 ggplot2      * 2.2.1      2016-12-30 CRAN (R 3.4.0)                 
 glue           1.2.0      2017-10-29 CRAN (R 3.4.2)                 
 gower          0.1.2      2017-02-23 CRAN (R 3.4.0)                 
 gtable         0.2.0      2016-02-26 CRAN (R 3.4.0)                 
 ipred          0.9-6      2017-03-01 CRAN (R 3.4.0)                 
 iterators      1.0.9      2017-12-12 CRAN (R 3.4.3)                 
 kernlab        0.9-25     2016-10-03 CRAN (R 3.4.0)                 
 lattice      * 0.20-35    2017-03-25 CRAN (R 3.4.0)                 
 lava           1.5.1      2017-09-27 CRAN (R 3.4.2)                 
 lazyeval       0.2.1      2017-10-29 CRAN (R 3.4.2)                 
 lubridate      1.7.1      2017-11-03 CRAN (R 3.4.2)                 
 magrittr       1.5        2014-11-22 CRAN (R 3.4.0)                 
 MASS           7.3-48     2017-12-25 CRAN (R 3.4.3)                 
 Matrix         1.2-12     2017-11-20 CRAN (R 3.4.3)                 
 mnormt         1.5-5      2016-10-15 CRAN (R 3.4.0)                 
 ModelMetrics   1.1.0      2016-08-26 CRAN (R 3.4.0)                 
 munsell        0.4.3      2016-02-13 CRAN (R 3.4.0)                 
 nlme           3.1-131    2017-02-06 CRAN (R 3.4.3)                 
 nnet           7.3-12     2016-02-02 CRAN (R 3.4.3)                 
 pillar         1.0.1      2017-11-27 CRAN (R 3.4.3)                 
 pkgconfig      2.0.1      2017-03-21 CRAN (R 3.4.0)                 
 plyr           1.8.4      2016-06-08 CRAN (R 3.4.0)                 
 prodlim        1.6.1      2017-03-06 CRAN (R 3.4.0)                 
 psych          1.7.8      2017-09-09 CRAN (R 3.4.3)                 
 purrr          0.2.4      2017-10-18 CRAN (R 3.4.2)                 
 R6             2.2.2      2017-06-17 CRAN (R 3.4.0)                 
 Rcpp           0.12.14    2017-11-23 CRAN (R 3.4.3)                 
 RcppRoll       0.2.2      2015-04-05 CRAN (R 3.4.0)                 
 recipes        0.1.1.9000 2017-12-28 Github (topepo/recipes@48a85f2)
 reshape2       1.4.3      2017-12-11 CRAN (R 3.4.3)                 
 rlang          0.1.6      2017-12-21 CRAN (R 3.4.3)                 
 robustbase     0.92-8     2017-11-01 CRAN (R 3.4.2)                 
 rpart          4.1-11     2017-03-13 CRAN (R 3.4.0)                 
 scales         0.5.0      2017-08-24 CRAN (R 3.4.1)                 
 sessioninfo  * 1.0.0      2017-06-21 CRAN (R 3.4.1)                 
 sfsmisc        1.1-1      2017-06-08 CRAN (R 3.4.0)                 
 stringi        1.1.6      2017-11-17 CRAN (R 3.4.2)                 
 stringr        1.2.0      2017-02-18 CRAN (R 3.4.0)                 
 survival       2.41-3     2017-04-04 CRAN (R 3.4.3)                 
 tibble         1.4.1      2017-12-25 CRAN (R 3.4.3)                 
 tidyr          0.7.2      2017-10-16 CRAN (R 3.4.2)                 
 tidyselect     0.2.3      2017-11-06 CRAN (R 3.4.2)                 
 timeDate       3042.101   2017-11-16 CRAN (R 3.4.2)                 
 withr          2.1.1      2017-12-19 CRAN (R 3.4.3) 

[GeraldCNelson]

Here's my output from session_info(). The -fopenmp option came directly from the openmp section of data.table. I know nothing about c code so am flying pretty blind.

I see a few differences between your output and mine. Other than recipes all of yours are from CRAN. I have a few from Github. In particular, your data.table is from CRAN and a slightly older version. I tried replacing my data.table with the one from CRAN but still got the same crash. I copied the list of packages with differences at the bottom of this comment.

> session_info()
─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.4.3 (2017-11-30)
 os       macOS High Sierra 10.13.2   
 system   x86_64, darwin15.6.0        
 ui       RStudio                     
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       America/Denver              
 date     2018-01-06                  

─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package      * version    date       source                            
 assertthat     0.2.0      2017-04-11 CRAN (R 3.4.0)                    
 bindr          0.1        2016-11-13 CRAN (R 3.4.0)                    
 bindrcpp       0.2        2017-06-17 CRAN (R 3.4.0)                    
 broom          0.4.3      2017-11-20 CRAN (R 3.4.3)                    
 caret        * 6.0-78     2018-01-06 Github (topepo/caret@b2b0d68)     
 class          7.3-14     2015-08-30 CRAN (R 3.4.3)                    
 clisymbols     1.2.0      2017-05-21 CRAN (R 3.4.0)                    
 codetools      0.2-15     2016-10-05 CRAN (R 3.4.3)                    
 colorspace     1.3-2      2016-12-14 CRAN (R 3.4.0)                    
 CVST           0.2-1      2013-12-10 CRAN (R 3.4.2)                    
 data.table   * 1.10.5     2018-01-06 local                             
 ddalpha        1.3.1      2017-09-27 CRAN (R 3.4.2)                    
 DEoptimR       1.0-8      2016-11-19 CRAN (R 3.4.2)                    
 dimRed         0.1.0      2017-05-04 CRAN (R 3.4.2)                    
 dplyr          0.7.4.9000 2018-01-06 Github (tidyverse/dplyr@cf54208)  
 DRR            0.0.2      2016-09-15 CRAN (R 3.4.2)                    
 foreach        1.4.4      2017-12-12 cran (@1.4.4)                     
 foreign        0.8-69     2017-06-22 CRAN (R 3.4.3)                    
 ggplot2      * 2.2.1.9000 2018-01-03 Github (tidyverse/ggplot2@4ee6c94)
 glue           1.2.0      2017-10-29 CRAN (R 3.4.2)                    
 gower          0.1.2      2017-02-23 CRAN (R 3.4.2)                    
 gtable         0.2.0      2016-02-26 CRAN (R 3.4.0)                    
 ipred          0.9-6      2017-03-01 CRAN (R 3.4.2)                    
 iterators      1.0.9      2017-12-12 CRAN (R 3.4.3)                    
 kernlab        0.9-25     2016-10-03 CRAN (R 3.4.2)                    
 lattice      * 0.20-35    2016-11-17 R-Forge (R 3.4.3)                 
 lava           1.5.1      2017-09-27 CRAN (R 3.4.2)                    
 lazyeval       0.2.1      2017-10-29 CRAN (R 3.4.2)                    
 lubridate      1.7.1      2017-11-03 CRAN (R 3.4.2)                    
 magrittr       1.5        2014-11-22 CRAN (R 3.4.0)                    
 MASS           7.3-48     2017-12-25 CRAN (R 3.4.3)                    
 Matrix         1.2-12     2017-11-20 CRAN (R 3.4.3)                    
 mnormt         1.5-5      2016-10-15 CRAN (R 3.4.0)                    
 ModelMetrics   1.1.0      2016-08-26 cran (@1.1.0)                     
 munsell        0.4.3      2016-02-13 CRAN (R 3.4.0)                    
 nlme           3.1-131    2017-02-06 CRAN (R 3.4.3)                    
 nnet           7.3-12     2016-02-02 CRAN (R 3.4.3)                    
 openxlsx       4.0.17     2017-03-23 CRAN (R 3.4.0)                    
 pillar         1.0.1      2017-11-27 CRAN (R 3.4.3)                    
 pkgconfig      2.0.1      2017-03-21 CRAN (R 3.4.0)                    
 plyr           1.8.4      2016-06-08 cran (@1.8.4)                     
 prodlim        1.6.1      2017-03-06 CRAN (R 3.4.2)                    
 psych          1.7.8      2017-09-09 CRAN (R 3.4.0)                    
 purrr          0.2.4      2017-10-18 CRAN (R 3.4.2)                    
 R6             2.2.2      2017-06-17 CRAN (R 3.4.0)                    
 Rcpp           0.12.14    2017-11-23 CRAN (R 3.4.3)                    
 RcppRoll       0.2.2      2015-04-05 CRAN (R 3.4.2)                    
 recipes        0.1.1      2017-11-20 cran (@0.1.1)                     
 reshape2       1.4.3      2017-12-11 cran (@1.4.3)                     
 rlang          0.1.6      2017-12-21 CRAN (R 3.4.3)                    
 robustbase     0.92-8     2017-11-01 CRAN (R 3.4.2)                    
 rpart          4.1-11     2017-03-13 CRAN (R 3.4.3)                    
 scales         0.5.0.9000 2017-11-27 Github (hadley/scales@d767915)    
 sessioninfo  * 1.0.0      2017-06-21 CRAN (R 3.4.1)                    
 sfsmisc        1.1-1      2017-06-08 CRAN (R 3.4.2)                    
 stringi        1.1.6      2017-11-17 CRAN (R 3.4.2)                    
 stringr        1.2.0      2017-02-18 CRAN (R 3.4.0)                    
 survival       2.41-3     2017-04-04 CRAN (R 3.4.3)                    
 tibble         1.4.1      2017-12-25 CRAN (R 3.4.3)                    
 tidyr          0.7.2      2017-10-16 CRAN (R 3.4.2)                    
 tidyselect     0.2.3      2017-11-06 CRAN (R 3.4.2)                    
 timeDate       3042.101   2017-11-16 CRAN (R 3.4.2)                    
 withr          2.1.1.9000 2018-01-03 Github (jimhester/withr@df18523)  
 yaml           2.1.16     2017-12-12 CRAN (R 3.4.2)                    
> 

Packages with differences

caret * 6.0-78 2018-01-06 Github (topepo/caret@b2b0d68) : caret * 6.0-78 2017-12-10 CRAN (R 3.4.3) 		
 CVST 0.2-1 2013-12-10 CRAN (R 3.4.2) : CVST 0.2-1 2013-12-10 CRAN (R 3.4.0) 		
 data.table * 1.10.5 2018-01-06 local : data.table * 1.10.4-3 2017-10-27 CRAN (R 3.4.2) 		
 DEoptimR 1.0-8 2016-11-19 CRAN (R 3.4.2) : DEoptimR 1.0-8 2016-11-19 CRAN (R 3.4.0) 		
 dimRed 0.1.0 2017-05-04 CRAN (R 3.4.2) : dimRed 0.1.0 2017-05-04 CRAN (R 3.4.0) 		
 dplyr 0.7.4.9000 2018-01-06 Github (tidyverse/dplyr@cf54208) : dplyr 0.7.4 2017-09-28 CRAN (R 3.4.2) 		
 DRR 0.0.2 2016-09-15 CRAN (R 3.4.2) : DRR 0.0.2 2016-09-15 CRAN (R 3.4.0) 		
 foreach 1.4.4 2017-12-12 cran (@1.4.4) : foreach 1.4.4 2017-12-12 CRAN (R 3.4.3) 		
 ggplot2 * 2.2.1.9000 2018-01-03 Github (tidyverse/ggplot2@4ee6c94) : ggplot2 * 2.2.1 2016-12-30 CRAN (R 3.4.0) 		
 gower 0.1.2 2017-02-23 CRAN (R 3.4.2) : gower 0.1.2 2017-02-23 CRAN (R 3.4.0) 		
 ipred 0.9-6 2017-03-01 CRAN (R 3.4.2) : ipred 0.9-6 2017-03-01 CRAN (R 3.4.0) 		
 kernlab 0.9-25 2016-10-03 CRAN (R 3.4.2) : kernlab 0.9-25 2016-10-03 CRAN (R 3.4.0) 		
 lattice * 0.20-35 2016-11-17 R-Forge (R 3.4.3) : lattice * 0.20-35 2017-03-25 CRAN (R 3.4.0) 		
 ModelMetrics 1.1.0 2016-08-26 cran (@1.1.0) : ModelMetrics 1.1.0 2016-08-26 CRAN (R 3.4.0) 		
 openxlsx 4.0.17 2017-03-23 CRAN (R 3.4.0) 							
 plyr 1.8.4 2016-06-08 cran (@1.8.4) : plyr 1.8.4 2016-06-08 CRAN (R 3.4.0) 		
 prodlim 1.6.1 2017-03-06 CRAN (R 3.4.2) : prodlim 1.6.1 2017-03-06 CRAN (R 3.4.0) 		
 psych 1.7.8 2017-09-09 CRAN (R 3.4.0) : psych 1.7.8 2017-09-09 CRAN (R 3.4.3) 		
 RcppRoll 0.2.2 2015-04-05 CRAN (R 3.4.2) : RcppRoll 0.2.2 2015-04-05 CRAN (R 3.4.0) 		
 recipes 0.1.1 2017-11-20 cran (@0.1.1) : recipes 0.1.1.9000 2017-12-28 Github (topepo/recipes@48a85f2)		
 reshape2 1.4.3 2017-12-11 cran (@1.4.3) : reshape2 1.4.3 2017-12-11 CRAN (R 3.4.3) 		
 rpart 4.1-11 2017-03-13 CRAN (R 3.4.3) : rpart 4.1-11 2017-03-13 CRAN (R 3.4.0) 		
 scales 0.5.0.9000 2017-11-27 Github (hadley/scales@d767915) : scales 0.5.0 2017-08-24 CRAN (R 3.4.1) 		
 sfsmisc 1.1-1 2017-06-08 CRAN (R 3.4.2) : sfsmisc 1.1-1 2017-06-08 CRAN (R 3.4.0) 		
 withr 2.1.1.9000 2018-01-03 Github (jimhester/withr@df18523) : withr 2.1.1 2017-12-19 CRAN (R 3.4.3) 		

[GeraldCNelson]

Didn't mean to close it. Clicked the wrong button.

[topepo]

I don't want to pass the buck, but caret doesn't use MPI but data.table does, as of late.

Can you cross-post there (or open an issue there that references this)? I think that they would be able to to figure this out. They have had a few openmp issues resolved but none seem to be the same as this.