Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Intel compiler v2016 support #691

Merged
merged 6 commits into from
Dec 14, 2015
Merged

Added Intel compiler v2016 support #691

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
15b48de
added support for Intel 2016 compilers
fmasselink Sep 18, 2015
782d164
Added support for installing Intel compilers v2016
fmasselink Sep 18, 2015
12c6707
Modified according to pull request #691
fmasselink Sep 18, 2015
01254a5
removed make_module_extra
fmasselink Sep 22, 2015
441d1b6
fixed paths for TBB, debugger. dirmap will now hold original items if…
fmasselink Oct 15, 2015
5e95c15
removed double modextravars and explicit gccversion = None
fmasselink Oct 15, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 108 additions & 37 deletions easybuild/easyblocks/i/icc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
from distutils.version import LooseVersion

from easybuild.easyblocks.generic.intelbase import IntelBase, ACTIVATION_NAME_2012, LICENSE_FILE_NAME_2012
from easybuild.tools.modules import get_software_root, get_software_version
from easybuild.tools.run import run_cmd


Expand Down Expand Up @@ -74,29 +75,35 @@ def install_step(self):
'license_file_name': LICENSE_FILE_NAME_2012,
}

super(EB_icc, self).install_step(silent_cfg_names_map=silent_cfg_names_map)
cfg_extras_map = {}
if LooseVersion(self.version) >= LooseVersion('2016'):
cfg_extras_map = {
'COMPONENTS': 'ALL',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a very good idea... This is icc, not "Intel world".

Is there documentation on what are valid values for COMPONENTS?

}
super(EB_icc, self).install_step(silent_cfg_names_map=silent_cfg_names_map, silent_cfg_extras=cfg_extras_map)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will crash for non-2016 versions, since cfg_extras_map is not defined

please change it to something like:

silent_cfg_extras = {}
if LooseVersion(self.version) >= LooseVersion('2016'):
    silent_cfg_extras.update{'COMPONENTS': 'ALL'}

super(EB_icc, self).install_step(silent_cfg_names_map=silent_cfg_names_map, silent_cfg_extras=silent_cfg_extras)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, makes sense.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we pass COMPONENTS=ALL, we get mkl, ipp, tbb, gdb and a whole other bunch of stuff installed. Do we want this? It actually makes the easyconfigs for the MKL, TBB, etc redundant. We can specify a subset of ALL:

intel-comp-l-all-vars__noarch
intel-comp-l-all-common__noarch
intel-comp-l-ps-common__noarch
intel-comp-l-all-devel__i486
intel-comp-l-ps-ss-devel__i486
intel-comp-l-all__i486
intel-comp-l-ps-ss-wrapper__i486
intel-comp-l-all-devel__x86_64
intel-comp-l-ps-devel__x86_64
intel-comp-l-ps-ss-devel__x86_64
intel-openmp-l-all__i486
intel-openmp-l-ps__i486
intel-openmp-l-all__x86_64
intel-openmp-l-ps-mic__x86_64
intel-openmp-l-ps__x86_64
intel-openmp-l-ps-ss__x86_64
intel-openmp-l-all-devel__i486
intel-openmp-l-all-devel__x86_64
intel-openmp-l-ps-mic-devel__x86_64
intel-openmp-l-ps-devel__x86_64
intel-openmp-l-ps-ss-devel__x86_64
intel-tbb-libs__noarch
intel-comp-all-doc__noarch
intel-comp-ps-ss-doc__noarch
intel-icc-doc__noarch
intel-icc-ps-doc__noarch
intel-icc-ps-ss-doc__noarch
intel-ifort-ps-doc__noarch
intel-icc-l-all__x86_64
intel-icc-l-ps-ss__x86_64
intel-icc-l-all-vars__noarch
intel-icc-l-all-common__noarch
intel-icc-l-ps-common__noarch
intel-icc-l-all__i486
intel-icc-l-ps-ss-wrapper__i486
intel-icc-l-all-devel__i486
intel-icc-l-all-devel__x86_64
intel-icc-l-ps-devel__x86_64
intel-icc-l-ps-ss-devel__x86_64
intel-ifort-l-ps__i486
intel-ifort-l-ps__x86_64
intel-ifort-l-ps-vars__noarch
intel-ifort-l-ps-common__noarch
intel-ifort-l-ps-devel__i486
intel-ifort-l-ps-devel__x86_64
intel-mpirt-l-ps__i486
intel-mkl__i486
intel-mkl-ps__i486
intel-mkl__x86_64
intel-mkl-ps__x86_64
intel-mkl-common__noarch
intel-mkl-ps-common__noarch
intel-mkl-devel__x86_64
intel-mkl-ps-mic-devel__x86_64
intel-mkl-ps-f95-devel__x86_64
intel-mkl-gnu-devel__x86_64
intel-mkl-ps-gnu-devel__x86_64
intel-mkl-ps-pgi-devel__x86_64
intel-mkl-sp2dp-devel__x86_64
intel-mkl-ps-cluster-devel__x86_64
intel-mkl-devel__i486
intel-mkl-ps-f95-devel__i486
intel-mkl-gnu-devel__i486
intel-mkl-ps-gnu-devel__i486
intel-mkl-ps-pgi-devel__i486
intel-mkl-ps-cluster-common__noarch
intel-mkl-ps-f95-common__noarch
intel-mkl-ps-cluster__x86_64
intel-mkl-gnu__x86_64
intel-mkl-ps-gnu__x86_64
intel-mkl-ps-pgi__x86_64
intel-mkl-sp2dp__x86_64
intel-mkl-ps-mic__x86_64
intel-mkl-gnu__i486
intel-mkl-ps-gnu__i486
intel-mkl-ps-pgi__i486
intel-mkl-doc__noarch
intel-mkl-ps-doc__noarch
intel-mkl-ps-tbb__i486
intel-mkl-ps-tbb-devel__i486
intel-mkl-ps-tbb__x86_64
intel-mkl-ps-tbb-devel__x86_64
intel-mkl-ps-tbb-mic__x86_64
intel-mkl-ps-tbb-mic-devel__x86_64
intel-ipp-l-common__noarch
intel-ipp-l-ps-common__noarch
intel-ipp-l-st-devel__i486
intel-ipp-l-st__i486
intel-ipp-l-st__x86_64
intel-ipp-l-mt__i486
intel-ipp-l-mt__x86_64
intel-ipp-l-st-devel__x86_64
intel-ipp-l-ps-st-devel__x86_64
intel-ipp-l-mt-devel__i486
intel-ipp-l-mt-devel__x86_64
intel-ipp-l-doc__noarch
intel-ipp-l-ps-doc__noarch
intel-crypto-ipp-st-devel__i486
intel-crypto-ipp-st-devel__x86_64
intel-crypto-ipp-ps-st-devel__x86_64
intel-crypto-ipp-st__x86_64
intel-crypto-ipp-st__i486
intel-crypto-ipp-mt-devel__i486
intel-crypto-ipp-mt-devel__x86_64
intel-crypto-ipp-mt__x86_64
intel-crypto-ipp-mt__i486
intel-crypto-ipp-ss-st-devel__i486
intel-crypto-ipp-ss-st-devel__x86_64
intel-crypto-ipp-common__noarch
intel-crypto-ipp-doc__noarch
intel-tbb-source__noarch
intel-tbb-devel__noarch
intel-tbb-common__noarch
intel-tbb-ps-common__noarch
intel-tbb-doc__noarch
intel-daal__i486
intel-daal__x86_64
intel-daal-common__noarch
intel-daal-doc__noarch
intel-gdb-gt__x86_64
intel-gdb-gt-src__noarch
intel-gdb-gt-libelfdwarf__x86_64
intel-gdb-gt-devel__x86_64
intel-gdb-gt-common__noarch
intel-gdb-gt-doc__noarch
intel-gdb-ps-cdt__x86_64
intel-gdb-ps-cdt-source__x86_64
intel-gdb-ps-mic__x86_64
intel-gdb-ps-mpm__x86_64
intel-gdb-ps-doc__noarch
intel-gdb-ps-doc-jp__noarch
intel-gdb__i486
intel-gdb__x86_64
intel-gdb-source__noarch
intel-gdb-python-source__noarch
intel-gdb-common__noarch
intel-gdb-doc__noarch
intel-icsxe__noarch
intel-ips__noarch
intel-ipsc__noarch
intel-ipsf__noarch
intel-compxe__noarch
intel-ccompxe__noarch
intel-fcompxe__noarch
intel-ipp-psxe__noarch
intel-mkl-psxe__noarch
intel-tbb-psxe__noarch
intel-daal-psxe__noarch
intel-mpi-psxe__x86_64
intel-mpi-rt-psxe__x86_64
intel-psf-ia__i486
intel-psf-intel__x86_64
intel-gdb-ps-common__noarch
intel-psxe-common__noarch
intel-psxe-doc__noarch
intel-ccomp-doc__noarch
intel-fcomp-doc__noarch
intel-icsxe-doc__noarch
intel-ips-doc__noarch
intel-ipsc-doc__noarch
intel-ipsf-doc__noarch
intel-compxe-doc__noarch
intel-ccompxe-doc__noarch
intel-fcompxe-doc__noarch
intel-icsxe-pset

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

omfg... how the hell da we figure out what we need from this mess? O_o

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trail and error. But most are clear what they mean.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seriously? There's no proper documentation for this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None that I have found?
See https://software.intel.com/en-us/articles/intel-composer-xe-2015-silent-installation-guide under COMPONENTS


def sanity_check_step(self):
"""Custom sanity check paths for icc."""

binprefix = "bin/intel64"
libprefix = "lib/intel64/lib"
libprefix = "lib/intel64"
if LooseVersion(self.version) >= LooseVersion("2011"):
if LooseVersion(self.version) <= LooseVersion("2011.3.174"):
binprefix = "bin"
elif LooseVersion(self.version) >= LooseVersion("2013_sp1"):
binprefix = "bin"
libprefix = "lib/intel64/lib"
if LooseVersion(self.version) >= LooseVersion("2016"):
libprefix = "lib/intel64_lin"
else:
libprefix = "compiler/lib/intel64/lib"
libprefix = "compiler/lib/intel64"

binfiles = ["icc", "icpc"]
if LooseVersion(self.version) < LooseVersion("2014"):
binfiles += ["idb"]

custom_paths = {
'files': ["%s/%s" % (binprefix, x) for x in binfiles] +
["%s%s" % (libprefix, x) for x in ["iomp5.a", "iomp5.so"]],
'files': [os.path.join(binprefix, x) for x in binfiles] +
[os.path.join(libprefix, 'lib%s' % x) for x in ['iomp5.a', 'iomp5.so']],
'dirs': [],
}

Expand All @@ -105,53 +112,117 @@ def sanity_check_step(self):
def make_module_req_guess(self):
"""Customize paths to check and add in environment.
"""
debuggerpath = None
prefix = None
if self.cfg['m32']:
# 32-bit toolchain
libpaths = ['lib', 'lib/ia32'],
dirmap = {
'PATH': ['bin', 'bin/ia32', 'tbb/bin/ia32'],
'LD_LIBRARY_PATH': ['lib', 'lib/ia32'],
'LIBRARY_PATH': ['lib', 'lib/ia32'],
'LD_LIBRARY_PATH': libpaths,
'LIBRARY_PATH': libpaths,
'MANPATH': ['man', 'share/man', 'man/en_US'],
'IDB_HOME': ['bin/intel64']
}
else:
# 64-bit toolit
dirmap = {
'PATH': ['bin', 'bin/intel64', 'tbb/bin/emt64'],
'LD_LIBRARY_PATH': ['lib', 'lib/intel64'],
'LIBRARY_PATH': ['lib', 'lib/intel64'],
'MANPATH': ['man', 'share/man', 'man/en_US'],
'IDB_HOME': ['bin/intel64']
}
# 64-bit toolkit

# using get_software_version('GCC') won't work, while the compiler toolchain is dummy:dummy, which does not
# load dependencies.
gccversion = get_software_version('GCC')
# manual approach to at least have the system version of gcc
if not gccversion:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a bit fat warn here. We don't really support this anymore. icc should always be installed alongside an EB GCC.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Ward, not sure what you mean.
My icc.eb file does have a dependency on GCC
name = 'icc'
version = '2016.0.109'
toolchain = {'name': 'dummy', 'version': 'dummy'}gcc = 'GCC'
gccver = '5.2.0'
versionsuffix = '-%s-%s' % (gcc, gccver)

though during the install phase it is not available. Likely because the toolchains is dummy, dummy and it doesn't load the GCC version listed.
One thing I was thinking about, (based on earlier discussion with dependencies for Intel-toolchains), was that if the FFTW libraries are built, they will currently be built with the system-GCC, while the dependency is not active during install.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, icc always needs a GCC. I was referring to your fallback to system GCC. I think you can do the following:

gccversion = get_software_version('GCC')
if not gccversion:
  raise EasyBuildError("GCC module is not loaded?")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wpoely86: no, please don't, you'll break the installation of icc in the ictce toolchain, where we don't include our own GCC underneath.

We should encourage using an EB-built GCC when installing/using icc, but not enforce it.

I wouldn't go further than adding a self.log.warning is the system GCC is being relied upon.

cmd = 'gcc --version'
(out, _) = run_cmd(cmd, log_all=True, simple=False)
ver_re = re.compile("^gcc \(GCC\) (?P<version>[0-9.]+) [0-9]+", re.M)
gccversion = ver_re.search(out).group('version')

gccversion = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this line should not be here?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, test from my side I think....

# TBB directory structure
# https://www.threadingbuildingblocks.org/docs/help/tbb_userguide/Linux_OS.htm
tbbgccversion = 'gcc4.4' # gcc version 4.4 or higher that may or may not support exception_ptr
if gccversion and LooseVersion(gccversion) >= LooseVersion("4.1") and LooseVersion(gccversion) < LooseVersion("4.4"):
tbbgccversion = 'gcc4.1' # gcc version number between 4.1 and 4.4 that do not support exception_ptr
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, this may be relevant for older versions?


if LooseVersion(self.version) < LooseVersion("2016"):
prefix = "composer_xe_%s" % self.version

# Debugger is dependent on INTEL_PYTHONHOME since version 2015 and newer
if LooseVersion(self.version) >= LooseVersion("2015"):
# Debugger requires INTEL_PYTHONHOME, which only allows for a single value
debuggerpath = os.path.join('composer_xe_%s' % self.version.split('.')[0], 'debugger')
if os.path.isdir(os.path.join(self.installdir, debuggerpath, 'python/intel64')):
self.cfg['modextravars'] = { 'INTEL_PYTHONHOME': os.path.join('$root',debuggerpath, 'python/intel64') }

libpaths = [os.path.join('tbb/lib/intel64', tbbgccversion),
'ipp/lib/intel64',
'debugger/ipt/intel64/lib',
'lib/intel64',
'compiler/lib/intel64',
]
dirmap = {
'PATH': ['debugger/gdb/intel64/bin', 'ipp/bin/intel64', 'tbb/bin/intel64', 'bin/intel64'],
'LD_LIBRARY_PATH': libpaths,
'LIBRARY_PATH': libpaths,
'MANPATH': ['debugger/gdb/intel64/share/man', 'man/en_US', 'share/man', 'man'],
'CPATH': ['ipp/include', 'tbb/include'],
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IDB_HOME is no longer set?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope. Intel dropped IDB in favor of GDB.
Though only these two got set:
GDBSERVER_MIC=/glb/home/s_hpcssw/easybuildinstall/software/rhel/6/icc/2016.0.109/debugger_2016/gdb/targets/mic/bin/gdbserver
GDB_CROSS=/glb/home/s_hpcssw/easybuildinstall/software/rhel/6/icc/2016.0.109/debugger_2016/gdb/intel64_mic/bin/gdb-mic

else:
# New Directory Layout for Intel Parallel Studio XE 2016
# https://software.intel.com/en-us/articles/new-directory-layout-for-intel-parallel-studio-xe-2016
prefix = "compilers_and_libraries_%s/linux" % self.version
# Debugger requires INTEL_PYTHONHOME, which only allows for a single value
debuggerpath = 'debugger_%s' % self.version.split('.')[0]

libpaths = ['daal/../compiler/lib/intel64_lin',
os.path.join('daal/../tbb/lib/intel64_lin', tbbgccversion),
'daal/lib/intel64_lin',
os.path.join(debuggerpath, 'libipt/intel64/lib'),
os.path.join('tbb/lib/intel64', tbbgccversion),
'mkl/lib/intel64',
'ipp/lib/intel64',
'ipp/../compiler/lib/intel64',
'mpi/intel64',
'compiler/lib/intel64',
'lib/intel64_lin',
]
dirmap = {
'PATH': ['mpi/intel64/bin',
'ipp/bin/intel64',
os.path.join(debuggerpath, 'gdb/intel64/bin'),
'bin/intel64',
'bin',
],
'LD_LIBRARY_PATH': libpaths,
'LIBRARY_PATH': libpaths,
'MANPATH': ['man/common', 'man/en_US', 'debugger/gdb/intel64/share/man'],
'CPATH': ['ipp/include', 'mkl/include', 'tbb/include', 'daal/include'],
'DAALROOT': ['daal'],
'TBBROOT': ['tbb'],
'IPPROOT': ['ipp'],
'CLASSPATH': ['daal/lib/daal.jar']
}

# set debugger path
if debuggerpath:
if os.path.isdir(os.path.join(self.installdir, debuggerpath, 'python/intel64')):
self.cfg['modextravars'] = { 'INTEL_PYTHONHOME': os.path.join('$root', debuggerpath, 'python/intel64') }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is added twice?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First one is not necessary. Was work-in-progess that was left behind. Thanx for the keen eye.


# in recent Intel compiler distributions, the actual binaries are
# in deeper directories, and symlinked in top-level directories
# however, not all binaries are symlinked (e.g. mcpcom is not)
if os.path.isdir("%s/composerxe-%s" % (self.installdir, self.version)):
prefix = "composerxe-%s" % self.version
# more recent versions of the Intel Compiler (2013.sp1 and newer)
if os.path.isdir(os.path.join(self.installdir, prefix)):
oldmap = dirmap
dirmap = {}
for k, vs in oldmap.items():
dirmap[k] = []
if k == "LD_LIBRARY_PATH":
prefix = "composerxe-%s/compiler" % self.version
else:
prefix = "composerxe-%s" % self.version
for v in vs:
v2 = "%s/%s" % (prefix, v)
dirmap[k].append(v2)

elif os.path.isdir("%s/compiler" % (self.installdir)):
prefix = "compiler"
oldmap = dirmap
dirmap = {}
for k, vs in oldmap.items():
dirmap[k] = []
prefix = ''
if k == "LD_LIBRARY_PATH":
prefix = "compiler/"
for v in vs:
v2 = "%s%s" % (prefix, v)
dirmap[k].append(v2)
v2 = os.path.join(prefix, v)
if os.path.exists(os.path.join(self.installdir, v2)):
dirmap[k].append(v2)
elif os.path.isdir(os.path.join(self.installdir, v)):
dirmap[k].append(v)

return dirmap