Easyconfig: allow configs to be overridden with env vars #712
Conversation
It's possible to override an Easyconfig option with an environment variable. The name should follow EB_EC_PARAM_<eb name>_<config>, for example: EB_EC_PARAM_ZLIB_HOMEPAGE. Options can be strings, list and dicts. Specify like: EB_EC_PARAM_ZLIB_PATCHES="file1:file2" EB_EC_PARAM_ZLIB_TOOLCHAINOPTS="pic=False:optarch=True" EB_EC_PARAM_ZLIB_HOMEPAGE="www.somesite.com"
|
reviewed and very nice! I really like snippets of code that give good orthogonality and have immense potential reuse across different cases! |
|
(@wpoely86 i can be a bit direct sometimes, don't take this personal. you have been contributing great stuff lately, much appreciated) a few remarks: but finally, someone still needs to explain me what problem is being solved here (really, the homepage as an environment variable? what's next, the content of the patchfiles?), and how to avoid the new one created, the cleanup of variables to garantee a clean build environment (to make it reproducable). something is clearly annoying users, i'd like to understand what. |
|
Hi Stijn, speaking for myself, I had commented at some point in #700 something like: ie. OK, the homepage wasn't the most convincing example but the general idea to be able In fact, OpenMPI requires some patching for us (so the above idea gets a hit), So, if ps. |
|
so a more realistic use case would be: you install EasyBuild latest and greatest and want to run eg and you want this also in environment variables and maybe also commandline, and it all has to make sense (eg typo checker). welll, it will depend a bit on the parser in v2. if that one is extended to more keys, it should be not that hard. |
|
what is the most important feature, from my biased PoV, is the ability to benefit from upstream patches (eg. by somebody improving on the configopts of OpenMPI), yet still retaining the additional tuning that is required in a certain HPC environment (for some it may be the ssh transport, for some others the queueing platform, for some more the arch etc) Granted, not everything is possible, but the ability to reduce the long list on our side to just a subset of exceptions, would be of great value: In fact, the ideas of fat easyconfigs & this PR together make it fairly elegant to meet the need. ps. |
|
can you post a diff of your .eb with the released ones? just to get an idea what needs customisation. and i don't think your gromacs website url example is valid. it does not point to the gromacs website, only to some documentation. however, it makes a lot of sense to introduce a parameter in the easyconfigs to hold a (list of) urls with relevant documentation/usage examples/... for the software that ios being installed. |
|
@stdweird I have no knowledge about the v2 format/parser so I can't comment on any of that. But you are correct that this code is too limited (the true/false is a good point). I would find it useful to quickly change something, like a toolchain option. And things like |
|
Furthermore, I happened to pass today around this easyconfig and here is a good example why the offered code can reduce effort elsewhere: |
|
the problem with this sort of code is that EB has no control over it, and thus can not check anything (is it set or not?) nor produce help to see what can be set. |
|
@stdweird: I have amended now #544 with the customization examples you asked (there are some more we do here, such as for MATLAB tuning but they don't add up much more, as regards this discussion). It is AS-IS, with the human divergence included, to prove the need for these two issues to get addressed ;-) Notice that different OpenMPI stacks (1.4.5 vs 1.6.x varieties) may require different patchfiles, therefor the variable approach is interesting because someone could script that externally, meanwhile the [openmpi] style approach could work if you can specify versions. What is more tricky is: how could we generalize the patchsets etc., for all users on a system? |
|
Automatic reply from Jenkins: Can I test this? |
|
ok to test |
| self[new_config] = os.environ[env_var].split(os.pathsep) | ||
| elif isinstance(self[new_config], dict): | ||
| try: | ||
| self[new_config] = {x.split('=')[0] : x.split('=')[1] for x in os.environ[env_var].split(os.pathsep)} |
There was a problem hiding this comment.
There is no dict comprehension in python 2.4
|
hi, i'm more and more convinced about the need of this PR; Having the feature would allow to have one more path to integrate easybuild with puppet and other CM systems. |
|
Automatic reply from Jenkins: Can I test this? |
|
test this please |
|
@fgeorgatos: I agree it would be very useful to have a good way to easily allow site customizations soon, but rushing things in like this is not the way. Although what @wpoely86 proposes here is an initial step, supporting this differently and more consistently (across command line, environment variables and config file entries) will be a much better solution. Like @stdweird already mentioned, the new easyconfig format also plays an important role in this. We're currently focusing on that (the new easyconfig format), and I feel it's pretty clear that supporting site customizations in a nice & clean way is going to be the next big thing we look into. So, for now, this won't go in as is. I'll leave the PR open for reference, but I doubt it'll go in in this form. I'll discuss this with @stdweird in due time, and maybe we should briefly discuss this during an upcoming EB conf call (although the next one would probably be a bit soon). If you want to help out, try clearly documenting the various options we discussed during the hackathon. We did so on the whiteboard, and you and me both have a picture of it, but I don't think there's much of this in the hackathon notes (yet). Documenting what we discussed there will definitely help in focusing the effort. |
|
Hi, just for the record, we have had use-cases for this PR on our end |
|
@boegel : ping ;-) |
|
@fgeorgatos: I have some concerns w.r.t. combo with We need a better solution here imho, probably via a configuration option like |
|
This has been inactive for 3 years. I guess there isn't much interest on it since the PR dates from 5 years ago. I'll close it up now guys in an effort to tidy up things, but if you think it is still relevant please open it again! |
Based on the discussion in #700
It's possible to override an Easyconfig option with an environment
variable. The name should follow EB_EC_PARAM__, for
example: EB_EC_PARAM_ZLIB_HOMEPAGE. Options can be strings, list and
dicts. Specify like:
EB_EC_PARAM_ZLIB_PATCHES="file1:file2"
EB_EC_PARAM_ZLIB_TOOLCHAINOPTS="pic=False:optarch=True"
EB_EC_PARAM_ZLIB_HOMEPAGE="www.somesite.com"
@boegel, @fgeorgatos please have a look.
Currently, only EB specific overrides are possible. Something like EB_EC_PARAM_MAXPARALLEL would make sense but EB_EC_PARAM_NAME doesn't. Do we allow a limited set of general options? In principle, all options can be general, only name would give problems (I think).