Support the concept of .ebp patch file or similar, for local site modifications #544

fgeorgatos · 2013-03-19T19:26:28Z

This in effect will allow to store patchfiles at local HPC sites as opposed to complete easyconfigs, which has the important advantage of allowing to benefit from upstream updates & bugfixes (think of MPI stacks that require local tuning, about this).

I think the following will help understand the use-case:

sw@gaia-13:~$ diff uni.lu-customizations/OpenMPI-1.4.5-GCC-4.6.3-no-OFED.eb /opt/apps/HPCBIOS.20130301/software/EasyBuild/1.2.0/lib/python2.6/site-packages/easybuild_easyconfigs-1.2.0.0-py2.6.egg/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.4.5-GCC-4.6.3-no-OFED.eb
13c13
< configopts = '--with-threads=posix --enable-shared --enable-mpi-threads '

---
> configopts = '--with-threads=posix --enable-shared --enable-mpi-threads'
16,19d15
<
< # FG, 20130306; customization for University of Luxembourg systems
< patches += ['openmpi-mca-params.conf-unilu.patch']
< configopts += '--with-openib --enable-mpirun-prefix-by-default '
sw@gaia-13:~$ cat uni.lu-customizations/openmpi-mca-params.conf-unilu.patch
--- opal/etc/openmpi-mca-params.conf.orig   2009-12-08 21:36:08.000000000 +0100
+++ opal/etc/openmpi-mca-params.conf    2013-03-06 08:13:21.216969244 +0100
@@ -56,3 +56,8 @@

 # See "ompi_info --param all all" for a full listing of Open MPI MCA
 # parameters available and their default values.
+
+# FG, 20130306; customization for University of Luxembourg systems
+btl_openib_if_exclude=mlx4_1
+btl_openib_warn_nonexistent_if=0
+plm_rsh_agent=oarsh

It is important to support this in the framework, to allow automated pick-up of such diffs during robotic builds with dependencies etc.

The text was updated successfully, but these errors were encountered:

boegel · 2013-03-19T20:16:31Z

To be more clear: this is intended to support easyconfig-specific customizations easily.

When EasyBuild finds a .ebp file alongside the .eb file, or in some directory that is specified, it would first alter the .eb file with the .ebp 'patch' before performing the build.

stdweird · 2013-03-20T13:07:55Z

what do you want to save in the local repo? both files, or the result sfer patching?

boegel · 2013-03-20T16:27:21Z

The result, imho, since it's a local repo.

When we add the feature to automagically start pushing easyconfigs that worked upstream, that's another issue...

fgeorgatos · 2013-03-20T21:34:13Z

I can see 3 levels of patches:

diff on the easyconfig itself
a parameter for patches=bla (ie. patch the sources of a package) # this may be doable manually with --amend.
diff on the produced modulefile # OK, this is basically hack you should rather avoid

(1) allows also for (2) and has the advantage that you don't miss the upgrades/bugfixes from upstream, so easily.
I'd opt for storing these diffs as .epb, along with patchfiles to help in the mission of (2)...

My gut feeling says that the above is generic enough to cover plenty of needs, but time will tell us what's right!

boegel · 2013-09-25T10:59:27Z

Having this in place would also allow to scratch the ignoring of CPU throttling in the ATLAS easyconfigs we ship, and simply use .ebp easyconfig patches that enable the ignoring when required (cfr. easybuilders/easybuild-easyconfigs#441).

fgeorgatos · 2013-10-09T07:34:53Z

fyi. this is extracted as-is from the customization of MPI stacks and TotalView on this end, to prove the issue:
(important: notice that eg. different OpenMPI stack versions may need different customization or patch)

[583][fotis.ctfwork_: /Users/fotis/uni.lu/easybuild-easyconfigs/easybuild/easyconfigs]$ git diff
diff --git a/easybuild/easyconfigs/i/impi/impi-4.0.0.028-32bit.eb b/easybuild/easyconfigs/i/impi/impi-4.0.0.028-32bit.eb
index c1fe77f..800e093 100644
--- a/easybuild/easyconfigs/i/impi/impi-4.0.0.028-32bit.eb
+++ b/easybuild/easyconfigs/i/impi/impi-4.0.0.028-32bit.eb
@@ -20,3 +20,7 @@ import os
 license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")

 m32 = True
+
+
+# added by EasyBuild as dictated by command line options
+modextravars = {"I_MPI_MPD_RSH": "oarsh"}
diff --git a/easybuild/easyconfigs/i/impi/impi-4.0.0.028.eb b/easybuild/easyconfigs/i/impi/impi-4.0.0.028.eb
index 02fa80a..60ac55a 100644
--- a/easybuild/easyconfigs/i/impi/impi-4.0.0.028.eb
+++ b/easybuild/easyconfigs/i/impi/impi-4.0.0.028.eb
@@ -17,3 +17,7 @@ dontcreateinstalldir = 'True'
 # license file
 import os
 license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")
+
+
+# added by EasyBuild as dictated by command line options
+modextravars = {"I_MPI_MPD_RSH": "oarsh"}
diff --git a/easybuild/easyconfigs/i/impi/impi-4.0.2.003.eb b/easybuild/easyconfigs/i/impi/impi-4.0.2.003.eb
index 3e513a0..748eec6 100644
--- a/easybuild/easyconfigs/i/impi/impi-4.0.2.003.eb
+++ b/easybuild/easyconfigs/i/impi/impi-4.0.2.003.eb
@@ -19,3 +19,7 @@ dontcreateinstalldir = 'True'
 # license file
 import os
 license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")
+
+
+# added by EasyBuild as dictated by command line options
+modextravars = {"I_MPI_MPD_RSH": "oarsh"}
diff --git a/easybuild/easyconfigs/i/impi/impi-4.1.0.027.eb b/easybuild/easyconfigs/i/impi/impi-4.1.0.027.eb
index 72b72e7..bde28cb 100644
--- a/easybuild/easyconfigs/i/impi/impi-4.1.0.027.eb
+++ b/easybuild/easyconfigs/i/impi/impi-4.1.0.027.eb
@@ -17,3 +17,7 @@ dontcreateinstalldir = 'True'
 # license file
 import os
 license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")
+
+
+# added by EasyBuild as dictated by command line options
+modextravars = {"I_MPI_MPD_RSH": "oarsh"}
diff --git a/easybuild/easyconfigs/i/impi/impi-4.1.0.030.eb b/easybuild/easyconfigs/i/impi/impi-4.1.0.030.eb
index 5d906bc..ae37dd6 100644
--- a/easybuild/easyconfigs/i/impi/impi-4.1.0.030.eb
+++ b/easybuild/easyconfigs/i/impi/impi-4.1.0.030.eb
@@ -17,3 +17,7 @@ dontcreateinstalldir = 'True'
 # license file
 import os
 license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")
+
+
+# added by EasyBuild as dictated by command line options
+modextravars = {"I_MPI_MPD_RSH": "oarsh"}
diff --git a/easybuild/easyconfigs/i/impi/impi-4.1.1.036.eb b/easybuild/easyconfigs/i/impi/impi-4.1.1.036.eb
index a6158d1..3f81883 100644
--- a/easybuild/easyconfigs/i/impi/impi-4.1.1.036.eb
+++ b/easybuild/easyconfigs/i/impi/impi-4.1.1.036.eb
@@ -17,3 +17,7 @@ dontcreateinstalldir = 'True'
 # license file
 import os
 license_file = os.path.join(os.getenv('HOME'), "licenses", "intel", "license.lic")
+
+
+# added by EasyBuild as dictated by command line options
+modextravars = {"I_MPI_MPD_RSH": "oarsh"}
diff --git a/easybuild/easyconfigs/m/MPICH/MPICH-3.0.3-ClangGCC-1.1.3.eb b/easybuild/easyconfigs/m/MPICH/MPICH-3.0.3-ClangGCC-1.1.3.eb
index 99806a5..1ed6634 100644
--- a/easybuild/easyconfigs/m/MPICH/MPICH-3.0.3-ClangGCC-1.1.3.eb
+++ b/easybuild/easyconfigs/m/MPICH/MPICH-3.0.3-ClangGCC-1.1.3.eb
@@ -21,3 +21,5 @@ sanity_check_paths = {
 }

 moduleclass = 'mpi'
+
+modextravars = {"HYDRA_LAUNCHER_EXEC": "oarsh", "HYDRA_IFACE": "ib0"}
diff --git a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.7-GCC-4.6.3-hwloc-chkpt.eb b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.7-GCC-4.6.3-hwloc-chkpt.eb
index 8d81812..9ab7122 100644
--- a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.7-GCC-4.6.3-hwloc-chkpt.eb
+++ b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.7-GCC-4.6.3-hwloc-chkpt.eb
@@ -31,3 +31,5 @@ builddependencies = [('Bison', '2.5')]
 parallel = 1

 moduleclass = 'mpi'
+
+modextravars = {"HYDRA_LAUNCHER_EXEC": "oarsh"}
diff --git a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.7-GCC-4.6.3.eb b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.7-GCC-4.6.3.eb
index 19e6c4d..be06f5d 100644
--- a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.7-GCC-4.6.3.eb
+++ b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.7-GCC-4.6.3.eb
@@ -7,12 +7,10 @@ It is based on MPICH2 and MVICH."""

 toolchain = {'name': 'GCC', 'version': '4.6.3'}

-sources = ['%(namelower)s-%(version)s.tgz']
+sources = ['%s-%s.tgz' % (name.lower(), version)]
 # note: this URL will only work for the most recent version (previous versions no longer available?)
 source_urls = ['http://mvapich.cse.ohio-state.edu/download/mvapich2']

-builddependencies = [('Bison', '2.5')]
-
 rdma_type = "gen2" # 'gen2' or 'udapl'

 ## enable building of MPE routines
@@ -22,3 +20,5 @@ withmpe = True
 parallel = 1

 moduleclass = 'mpi'
+
+modextravars = {"HYDRA_LAUNCHER_EXEC": "oarsh"}
diff --git a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.8.1-GCC-4.7.2.eb b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.8.1-GCC-4.7.2.eb
index 2b6e8df..1ef48df 100644
--- a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.8.1-GCC-4.7.2.eb
+++ b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.8.1-GCC-4.7.2.eb
@@ -7,11 +7,11 @@ description = "This is an MPI-2 implementation (conforming to MPI 2.1 standard)
 toolchain = {'name': 'GCC', 'version': '4.7.2'}

 source_urls = ['http://mvapich.cse.ohio-state.edu/download/mvapich2/']
-sources = ['%(namelower)s-%(version)s.tgz']
-
-builddependencies = [('Bison', '2.7')]
+sources = ['mvapich2-%s.tgz' % version]

 # parallel build doesn't work
 parallel = 1

 moduleclass = 'mpi'
+
+modextravars = {"HYDRA_LAUNCHER_EXEC": "oarsh"}
diff --git a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-ClangGCC-1.1.3.eb b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-ClangGCC-1.1.3.eb
index fd7d023..875a9bd 100644
--- a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-ClangGCC-1.1.3.eb
+++ b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-ClangGCC-1.1.3.eb
@@ -12,3 +12,5 @@ sources = ['mvapich2-%s.tgz' % version]
 dependencies = [('Bison', '2.7')]

 moduleclass = 'mpi'
+
+modextravars = {"HYDRA_LAUNCHER_EXEC": "oarsh"}
diff --git a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-ClangGCC-1.2.3.eb b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-ClangGCC-1.2.3.eb
index 229478e..18603ff 100644
--- a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-ClangGCC-1.2.3.eb
+++ b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-ClangGCC-1.2.3.eb
@@ -12,3 +12,5 @@ sources = ['mvapich2-%s.tgz' % version]
 dependencies = [('Bison', '2.7')]

 moduleclass = 'mpi'
+
+modextravars = {"HYDRA_LAUNCHER_EXEC": "oarsh"}
diff --git a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-GCC-4.7.3.eb b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-GCC-4.7.3.eb
index 9ff3ca9..77ddf70 100644
--- a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-GCC-4.7.3.eb
+++ b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-GCC-4.7.3.eb
@@ -12,3 +12,5 @@ sources = ['%(namelower)s-%(version)s.tgz']
 builddependencies = [('Bison', '2.7')]

 moduleclass = 'mpi'
+
+modextravars = {"HYDRA_LAUNCHER_EXEC": "oarsh"}
diff --git a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-iccifort-2011.13.367.eb b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-iccifort-2011.13.367.eb
index b140cf3..49d3f15 100644
--- a/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-iccifort-2011.13.367.eb
+++ b/easybuild/easyconfigs/m/MVAPICH2/MVAPICH2-1.9-iccifort-2011.13.367.eb
@@ -16,3 +16,5 @@ builddependencies = [('Bison', '2.7', '', ('ictce', '4.1.13'))]
 preconfigopts = 'LDFLAGS="-Wl,-rpath,$EBROOTICC/compiler/lib/intel64 $LDFLAGS"'

 moduleclass = 'mpi'
+
+modextravars = {"HYDRA_LAUNCHER_EXEC": "oarsh"}
diff --git a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.4.5-GCC-4.6.3-no-OFED.eb b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.4.5-GCC-4.6.3-no-OFED.eb
index 368b930..35aadca 100644
--- a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.4.5-GCC-4.6.3-no-OFED.eb
+++ b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.4.5-GCC-4.6.3-no-OFED.eb
@@ -13,7 +13,7 @@ source_urls = ['http://www.open-mpi.org/software/ompi/v%s/downloads' % '.'.join(
 configopts = '--with-threads=posix --enable-shared --enable-mpi-threads --without-openib '
 configopts += '--enable-mpirun-prefix-by-default '  # suppress failure modes in relation to mpirun path

-patches = ['pax_disable.patch']
+patches = ['openmpi-mca-params.conf-1.4.5-local.patch', 'pax_disable.patch']

 sanity_check_paths = {
     'files': ["bin/%s" % binfile for binfile in ["ompi_info", "opal_wrapper", "orterun"]] +
diff --git a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.4.5-GCC-4.6.3.eb b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.4.5-GCC-4.6.3.eb
index 9e7974c..b000920 100644
--- a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.4.5-GCC-4.6.3.eb
+++ b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.4.5-GCC-4.6.3.eb
@@ -12,12 +12,7 @@ source_urls = ['http://www.open-mpi.org/software/ompi/v%s/downloads' % '.'.join(
 configopts = '--with-threads=posix --enable-shared --enable-mpi-threads --with-openib '
 configopts += '--enable-mpirun-prefix-by-default '  # suppress failure modes in relation to mpirun path

-# needed for --with-openib
-if OS_NAME in ['debian', 'ubuntu']:
-    osdependencies = ['libibverbs-dev']
-else:
-    # OK for OS_NAME == redhat, fedora, RHEL, SL, centos
-    osdependencies = ['libibverbs-devel']
+patches = ['openmpi-mca-params.conf-1.4.5-local.patch', 'pax_disable.patch']

 sanity_check_paths = {
     'files': ["bin/%s" % binfile for binfile in ["ompi_info", "opal_wrapper", "orterun"]] +
diff --git a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.3-iccifort-2011.13.367.eb b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.3-iccifort-2011.13.367.eb
index 368fd7f..09abaf2 100644
--- a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.3-iccifort-2011.13.367.eb
+++ b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.3-iccifort-2011.13.367.eb
@@ -36,3 +36,5 @@ sanity_check_paths = {
 }

 moduleclass = 'mpi'
+
+patches = ['openmpi-mca-params.conf-1.6.4-local.patch', 'pax_disable.patch']
diff --git a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-ClangGCC-1.1.3.eb b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-ClangGCC-1.1.3.eb
index 1a944e1..46a9d08 100644
--- a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-ClangGCC-1.1.3.eb
+++ b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-ClangGCC-1.1.3.eb
@@ -35,3 +35,5 @@ sanity_check_paths = {
 }

 moduleclass = 'mpi'
+
+patches = ['openmpi-mca-params.conf-1.6.4-local.patch', 'pax_disable.patch']
diff --git a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-GCC-4.6.4.eb b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-GCC-4.6.4.eb
index 663d940..c04a29e 100644
--- a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-GCC-4.6.4.eb
+++ b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-GCC-4.6.4.eb
@@ -35,3 +35,5 @@ sanity_check_paths = {
 }

 moduleclass = 'mpi'
+
+patches = ['openmpi-mca-params.conf-1.6.4-local.patch', 'pax_disable.patch']
diff --git a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-GCC-4.7.2.eb b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-GCC-4.7.2.eb
index e142825..d32bd08 100644
--- a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-GCC-4.7.2.eb
+++ b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.4-GCC-4.7.2.eb
@@ -35,3 +35,5 @@ sanity_check_paths = {
 }

 moduleclass = 'mpi'
+
+patches = ['openmpi-mca-params.conf-1.6.4-local.patch', 'pax_disable.patch']
diff --git a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.5-GCC-4.8.1-no-OFED.eb b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.5-GCC-4.8.1-no-OFED.eb
index d3b9362..af0bd04 100644
--- a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.5-GCC-4.8.1-no-OFED.eb
+++ b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.5-GCC-4.8.1-no-OFED.eb
@@ -10,7 +10,7 @@ toolchain = {'name': 'GCC', 'version': '4.8.1'}
 sources = [SOURCELOWER_TAR_GZ]
 source_urls = ['http://www.open-mpi.org/software/ompi/v%(version_major_minor)s/downloads']

-patches = ['OpenMPI-1.6.5-vt_cupti_events.patch']
+patches = ['openmpi-mca-params.conf-1.6.4-local.patch', 'OpenMPI-1.6.5-vt_cupti_events.patch']

 configopts = '--with-threads=posix --enable-shared --enable-mpi-thread-multiple --without-openib '
 configopts += '--enable-mpirun-prefix-by-default '  # suppress failure modes in relation to mpirun path
diff --git a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.5-GCC-4.8.1.eb b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.5-GCC-4.8.1.eb
index ec79b10..f68b0f1 100644
--- a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.5-GCC-4.8.1.eb
+++ b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.6.5-GCC-4.8.1.eb
@@ -9,7 +9,7 @@ toolchain = {'name': 'GCC', 'version': '4.8.1'}
 sources = [SOURCELOWER_TAR_GZ]
 source_urls = ['http://www.open-mpi.org/software/ompi/v%(version_major_minor)s/downloads']

-patches = ['OpenMPI-1.6.5-vt_cupti_events.patch']
+patches = ['openmpi-mca-params.conf-1.6.4-local.patch', 'OpenMPI-1.6.5-vt_cupti_events.patch']

 configopts = '--with-threads=posix --enable-shared --enable-mpi-thread-multiple --with-openib '
 configopts += '--enable-mpirun-prefix-by-default '  # suppress failure modes in relation to mpirun path
diff --git a/easybuild/easyconfigs/t/TotalView/TotalView-8.11.0-0-linux-x86-64.eb b/easybuild/easyconfigs/t/TotalView/TotalView-8.11.0-0-linux-x86-64.eb
index 3a01987..91d69dd 100644
--- a/easybuild/easyconfigs/t/TotalView/TotalView-8.11.0-0-linux-x86-64.eb
+++ b/easybuild/easyconfigs/t/TotalView/TotalView-8.11.0-0-linux-x86-64.eb
@@ -35,3 +35,6 @@ import os
 license_file = os.path.join(os.getenv('HOME'), "licenses", "roguewave", "license.lic")

 moduleclass = 'debugger'
+
+# needed for uni.lu
+modextravars = {"TVDSVRLAUNCHCMD": "oarsh"}
diff --git a/easybuild/easyconfigs/t/TotalView/TotalView-8.11.0-2-linux-x86-64.eb b/easybuild/easyconfigs/t/TotalView/TotalView-8.11.0-2-linux-x86-64.eb
index f619dfb..510cf37 100644
--- a/easybuild/easyconfigs/t/TotalView/TotalView-8.11.0-2-linux-x86-64.eb
+++ b/easybuild/easyconfigs/t/TotalView/TotalView-8.11.0-2-linux-x86-64.eb
@@ -35,3 +35,6 @@ import os
 license_file = os.path.join(os.getenv('HOME'), "licenses", "roguewave", "license.lic")

 moduleclass = 'debugger'
+
+# needed for uni.lu
+modextravars = {"TVDSVRLAUNCHCMD": "oarsh"}
diff --git a/easybuild/easyconfigs/t/TotalView/TotalView-8.12.0-0-linux-x86-64.eb b/easybuild/easyconfigs/t/TotalView/TotalView-8.12.0-0-linux-x86-64.eb
index 7147f39..0447e73 100644
--- a/easybuild/easyconfigs/t/TotalView/TotalView-8.12.0-0-linux-x86-64.eb
+++ b/easybuild/easyconfigs/t/TotalView/TotalView-8.12.0-0-linux-x86-64.eb
@@ -35,3 +35,6 @@ import os
 license_file = os.path.join(os.getenv('HOME'), "licenses", "roguewave", "license.lic")

 moduleclass = 'debugger'
+
+# needed for uni.lu
+modextravars = {"TVDSVRLAUNCHCMD": "oarsh"}

boegel · 2013-11-08T19:39:10Z

@fgeorgatos: Thanks for sharing this, this will help us in figuring out how to properly support site customizations.

boegel · 2016-01-28T10:18:57Z

@jas02: you may be interested in this idea/concept w.r.t. managing site customisations of easyconfigs

fgeorgatos · 2016-01-28T13:02:44Z

discussed this a bit now: it would be a conditional step of retrieving an .ebp file and,
if it exists, merge original .eb together with .epb and then evaluate that result instead
(this idea comes with both features and issues, yet it may have its merit for certain uses)

On 28 January 2016 at 11:18, Kenneth Hoste [email protected] wrote:

@jas02 https://github.com/jas02: you may be interested in this
idea/concept w.r.t. managing site customisations of easyconfigs

fgeorgatos · 2016-01-28T13:05:17Z

btw. the other viable alternative is to maintain your diffs in a git branch

that works fine!;
this is about implementing a python-aware way of pushing modifications,
avoiding to touch the original easyconfig.

On 28 January 2016 at 14:02, Fotis Georgatos wrote:

discussed this a bit: it would be a conditional step of retrieving an .ebp file and,
if it exists, merge original .eb together with .epb and then evaluate that result instead
(this idea comes with both features and issues, yet it may have its merit for certain uses)

boegel · 2016-03-16T17:24:07Z

as mentioned by @besserox during today's EB conf call: maybe there's no need for this at all, since maintaining patches on top of the mainline easyconfigs in separate branch in a clone of the easybuild-easyconfigs repo, and updating it with master on every EasyBuild release is good enough to maintain site customisations?

akesandgren · 2016-10-21T08:25:02Z

The concept of .ebp does indeed look interesting and i can immediately see lots of uses for it.

What's the status?

fgeorgatos · 2016-10-25T18:15:13Z

I think this was in stalled state until today ;)

Basically, people got convenienced with git commits and workflows, so this
ended up in the backburner...

On Friday, October 21, 2016, Åke Sandgren [email protected] wrote:

The concept of .ebp does indeed look interesting and i can immediately see
lots of uses for it.

What's the status?

boegel · 2016-10-25T18:18:29Z

Indeed, it got stuck to being just an idea...

Even though people have resorted to git instead, I could still see this being useful though.

@akesandgren up for giving it a try? I could give you some pointers...

akesandgren · 2016-10-25T18:21:14Z

Not at the moment, way to much things to deal with first...

geimer · 2017-02-11T16:24:01Z

Doesn't adding support for Easyconfig patch files come down to something like "amend from file"?

EasyBuild would have to look for a file with the same name as the easyconfig but with an '.ebp' extension and parse it after the real easyconfig. In the 'ebp' file one would be able to use = to overwrite values and += to extend them. The stored easyconfig would then simply be the concatenation of the two (just like an easyconfig used with --try-amend). These "amend files" would IMHO be much easier to maintain than real patch files in case of changes in the upstream easyconfig.

What do you think?

vanzod · 2017-02-11T16:37:33Z

@geimer There may be a use case for storing the .ebp files separately instead of the concatenated version easyconfig. If I want to apply the same patch to a new version of the easyconfig, I need to store it somewhere else or strip it off the concatenated stored file. Instead, if the .ebp is stored separately the user could easily retrieve it.
Maybe instead of concatenation EB could simply add a line in the build report that says which patch file has been used.

fgeorgatos · 2017-02-13T08:25:40Z

So, as regards the pythonic version of easyconfigs, the --amend-from-file approach may well make sense; since it allows more fine manipulation of existing settings, including using conditionals (which could be thought as evil by some). My take is, that it might not be the final answer, yet it appears reasonable for the near future! F.

…

On Sat, 11 Feb 2017 at 17:24, geimer ***@***.***> wrote: Doesn't adding support for Easyconfig patch files come down to something like "amend from file"? EasyBuild would have to look for a file with the same name as the easyconfig but with an '.ebp' extension and parse it after the real easyconfig. In the 'ebp' file one would be able to use = to overwrite values and += to extend them. The stored easyconfig would then simply be the concatenation of the two (just like an easyconfig used with --try-amend). These "amend files" would IMHO be much easier to maintain than real patch files in case of changes in the upstream easyconfig. What do you think?

ocaisa · 2017-02-13T11:18:29Z

This discussion is moving in the direction of what we are already discussing (and @rjeschmi
and I are starting to implement) for the .yeb format. What we are implementing boils down to making the .yeb format fat with toolchains so that a single <software>-<version>-<versionsuffix>.yeb is equivalent to the entire set of .eb for the same software. The implementation we have planned would:

Have common information in the first yaml document

Each subsequent .yaml document is then toolchains and customisations on this common information, for example:

%YAML 1.2
---
<common stuff>
---
toolchain: {name: foss, version: 2015b}
---
toolchain: {name: intel, version: 2015b}
add:
  - patches: ['intel.patch']
replace:
  - postinstallcmds: ["echo TOY > %(installdir)s/README"]

We could consider an additional .yebp format that would just contain the modifications (one could even consider a <software>.yebp that applied to all versions of <software> (for documentation changes) and a <software>-<version>-<versionsuffix>.yeb for specific things).

rjeschmi · 2017-02-13T17:22:40Z

I have committed some of the test cases and very rough functions into a branch here:

https://github.com/rjeschmi/easybuild-framework/tree/fat-yaml

boegel mentioned this issue Aug 29, 2013

NCL file out of date? Dependencies missing? easybuilders/easybuild-easyconfigs#395

Closed

boegel mentioned this issue Sep 25, 2013

goalf 1.5.12 easybuilders/easybuild-easyconfigs#441

Merged

boegel mentioned this issue Oct 1, 2013

set ignorethrottling to True in ATLAS v3.10.1 easyconfigs wpoely86/easybuild-easyconfigs#4

Merged

fgeorgatos mentioned this issue Oct 7, 2013

Easyconfig: allow configs to be overridden with env vars #712

Closed

boegel mentioned this issue Dec 31, 2013

Binding of processes to hardware resources by MPI easybuilders/easybuild-easyconfigs#581

Open

boegel modified the milestone: v1.X Jun 24, 2015

boegel mentioned this issue Mar 3, 2016

Updated psmpi easyconfig files to use the new psmpi easyblock easybuilders/easybuild-easyconfigs#2619

Merged

boegel modified the milestones: v2.7.0, v2.8.0 Mar 3, 2016

boegel modified the milestones: v2.8.0, v2.x May 10, 2016

boegel mentioned this issue May 23, 2018

share extensions between R, Python, Ruby, etc... having the same version but different toolchains. easybuilders/easybuild-easyconfigs#6355

Open

boegel modified the milestones: 3.x, 4.x Feb 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support the concept of .ebp patch file or similar, for local site modifications #544

Support the concept of .ebp patch file or similar, for local site modifications #544

fgeorgatos commented Mar 19, 2013

boegel commented Mar 19, 2013

stdweird commented Mar 20, 2013

boegel commented Mar 20, 2013

fgeorgatos commented Mar 20, 2013

boegel commented Sep 25, 2013

fgeorgatos commented Oct 9, 2013

boegel commented Nov 8, 2013

boegel commented Jan 28, 2016

fgeorgatos commented Jan 28, 2016 •

edited

Loading

fgeorgatos commented Jan 28, 2016 •

edited

Loading

boegel commented Mar 16, 2016

akesandgren commented Oct 21, 2016

fgeorgatos commented Oct 25, 2016 •

edited

Loading

boegel commented Oct 25, 2016

akesandgren commented Oct 25, 2016

geimer commented Feb 11, 2017

vanzod commented Feb 11, 2017

fgeorgatos commented Feb 13, 2017 via email

ocaisa commented Feb 13, 2017 •

edited

Loading

rjeschmi commented Feb 13, 2017

Support the concept of .ebp patch file or similar, for local site modifications #544

Support the concept of .ebp patch file or similar, for local site modifications #544

Comments

fgeorgatos commented Mar 19, 2013

boegel commented Mar 19, 2013

stdweird commented Mar 20, 2013

boegel commented Mar 20, 2013

fgeorgatos commented Mar 20, 2013

boegel commented Sep 25, 2013

fgeorgatos commented Oct 9, 2013

boegel commented Nov 8, 2013

boegel commented Jan 28, 2016

fgeorgatos commented Jan 28, 2016 • edited Loading

fgeorgatos commented Jan 28, 2016 • edited Loading

boegel commented Mar 16, 2016

akesandgren commented Oct 21, 2016

fgeorgatos commented Oct 25, 2016 • edited Loading

boegel commented Oct 25, 2016

akesandgren commented Oct 25, 2016

geimer commented Feb 11, 2017

vanzod commented Feb 11, 2017

fgeorgatos commented Feb 13, 2017 via email

ocaisa commented Feb 13, 2017 • edited Loading

rjeschmi commented Feb 13, 2017

fgeorgatos commented Jan 28, 2016 •

edited

Loading

fgeorgatos commented Jan 28, 2016 •

edited

Loading

fgeorgatos commented Oct 25, 2016 •

edited

Loading

ocaisa commented Feb 13, 2017 •

edited

Loading