number of available cpus (in cpuset) #700

wpoely86 · 2013-09-30T11:59:07Z

On Linux, try to find out if we're a member of a cpuset and find the
number of CPUs in it. Use this number to set the parallelism. If we're not in a cpuset, continue with get_core_count().

This should also be done on BSD...

On Linux, try to find out if we're a member of a cpuset and find the number of CPUs in it. If not, continue with the old behaviour. This should also be done on BSD...

boegel · 2013-09-30T12:09:25Z

easybuild/tools/systemtools.py

+                _log.info("In cpuset with %s CPUs" % numofcpus)
+                return numofcpus
+        except IOError, err:
+            _log.warning("Failed to read /proc/%s/status to determine the cpuset: %s" % (mypid, err))


at least add an else: with a warning log message that attempting get to determine number of available CPUs is not even attempted

Right!

I think adding a pass is beter here? It's not a problem when this fails, as long as get_core_count() still works.

OK, I don't follow. When an exception is raised, it prints a warning and continues to get_core_count(). Where do you want an else:?

an else: on the same level as if os_type == LINUX:, that's run for e.g. OS X.

It's OK that it just falls through to use get_core_count, but it should be explicit in the log.

I think adding a pass is beter here? It's not a problem when this fails, as long as get_core_count() still works.

A simple pass is silent, a warning log message also passes through in the end, but at least there's something in the logs...

wpoely86 · 2013-09-30T12:45:05Z

All done and tested.

boegel · 2013-09-30T13:07:27Z

easybuild/tools/systemtools.py

+            cpuset = re.match("^Cpus_allowed_list:\s*([0-9,-]+)", txt, re.M)
+            if cpuset is not None:
+                cpuset_list = cpuset.group(1).split(',')
+                numofcpus = 0


please use num_of_cpus (improved readability)

boegel · 2013-09-30T13:09:32Z

@wpoely86: If you collapse the 7 lines of code into the (ugly ;-)) list comprehension line I proposed, then I consider this ready, but it'll have to wait until EB v1.9, since the feature freeze in in effect since last Friday.

Do you need this urgently for something particular?

As soon as I've branched off 1.8.x (soon) I can merge this into develop...

wpoely86 · 2013-09-30T13:11:46Z

I don't need it any time soon. The current code works, it only gives more process than cpu's.

boegel · 2013-10-01T07:35:33Z

Verified that unit tests work, this should be good to go in as soon as 1.8.x is branched off. Thanks @wpoely86!

fgeorgatos · 2013-10-01T14:08:00Z

Hi there,

how is this one being used?
I try to grok if it relates in any way to BC_CORES_PER_NODE from here:
http://hpcbios.readthedocs.org/en/latest/HPCBIOS_2012-98.html#hpcbios-2012-98

boegel · 2013-10-01T14:12:09Z

@fgeorgatos: it is used to determine the default setting for parallel, i.e. the default value for n in Make -j n.
Before this patch, we simply counted the number of available cores, which is of course wrong if your locked inside a cpuset.

fgeorgatos · 2013-10-01T15:34:14Z

OK! Could we "invent" some kind of BC_ variable that allows to override
this externally from EasyBuild,
since that would be a convenient handle in configuration management
systems, user startup files etc?

wpoely86 · 2013-10-01T15:40:14Z

I don't understand what you want to do. Putting EasyBuild in a cpuset is done externally?

You can set maxparallel in EasyConfig, if you want to control it?

fgeorgatos · 2013-10-01T20:14:59Z

I was considering the case of provisioning for something like
BC_MAXPARALLEL,
so that there is a way to control the automation externally.
(fi. we have a node with 16 decacore CPUs, this would allow for some
interesting combinations)

If you don't see the need for it, I'm fine to postpone for when the moment
makes it arise.

wpoely86 · 2013-10-02T08:32:57Z

Well, it's easy to do. We only need to check $BC_MAXPARALLEL in set_parallelism and compare it to maxparallel and take the minimum of those two.

I can add it if you find it usefull.

boegel · 2013-10-02T08:34:46Z

That should be fairly easy to support... But maybe this should be done in a more generic way, i.e. to allow that various easyconfig parameters can be easily overridden. I'm thinking $EB_EC_PARAM_MAXPARALLEL, $EB_EC_PARAM_PATCHES, $EB_EC_PARAM_IGNORETHROTTLING for ATLAS, etc.

Basically, the equivalent of the $EASYBUILD_X variables that can be set to override EasyBuild configuration parameters (installpath, buildpath, ...).

wpoely86 · 2013-10-02T08:58:56Z

Should we do it generally or eb specific? $EB_EC_PARAM_PATCHES or $EB_EC_PARAM_ATLAS_PATCHES. Things like $EB_EC_PARAM_PATCHES are only useful for a specific eb.

boegel · 2013-10-02T09:06:24Z

Hmm, good point @wpoely86... I think the $EB_EC_PARAM_ATLAS_PATCHES makes more sense, it gives more fine-grained control in case of large multi-easyconfig builds.

fgeorgatos · 2013-10-02T10:26:42Z

+1, really like the direction this is taking; giving control externally,
will require more easy tuning
via configuration management or modules, without forcing to hack the
defaults. Nice.

I'd suggest that the each time a variable is defined, the setting is
"enforced" (as opposed
to taking the minimum or such, in order to defer more control externally.
But I agree that
the builder may have a reason to check both and make a choice on that basis)

On Wed, Oct 2, 2013 at 11:06 AM, Kenneth Hoste [email protected]:

Hmm, good point @wpoely86 https://github.com/wpoely86... I think the
$EB_EC_PARAM_ATLAS_PATCHES makes more sense, it gives more fine-grained
control in case of large multi-easyconfig builds.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/700#issuecomment-25524054
.

boegel · 2013-10-02T11:11:20Z

The order of preference should be the same as with the EasyBuild configuration: i.e. command line (is this relevant in this case? --try-X doesn't count) has preference over environment has preference over easyconfig file.
Enforcing rather than combining values makes more sense to me, it certainly avoids potentially nasty surprises.

wpoely86 · 2013-10-02T11:57:33Z

command line is not relevant IMHO. That leaves environment > easyconfig.

I will have a look at where and how to code this. The init of the easyconfig class seems a logical place.

boegel · 2013-10-02T11:59:57Z

@wpoely86: Yes, this should be tackled in EasyConfig.__init__ in easybuild/framework/easyconfig/easyconfig.py, probably just after this (and before the validation part):

# parse easyconfig file
self.parse(path)

boegel · 2013-10-18T13:44:56Z

The required update for vsc.utils is available in wpoely86#4.

But, this seems to break the unit tests on OS X (and the get_(avail_)core_count() functionality), which is a regression...

So, maybe we should only use the sched_getaffinity approach on Linux, and fall back to the previous way on OS X?

@stdweird: How portable is sched_getaffinity supposed to be (on Linux, and on *nix)?

wpoely86 · 2013-10-18T15:11:48Z

According to https://developer.apple.com/library/mac/releasenotes/Performance/RN-AffinityAPI/ there is no equivalent of sched_getaffinity in darwin. Maybe we better stick to the old approach for darwin?

@boegel can you check what sysctl hw.logicalcpu tells us?

boegel · 2013-10-18T16:33:32Z

I would stick to the old approach on Darwin, yes. Getting this 100% right (w.r.t. taskset, cpusets, etc.) is more important on Linux, but we should break this on OS X (and not only because my Mac laptop is my main development platform).

On a MacBook Pro with (dual-core) Intel Core 2 Duo:

$ uname -a
Darwin Kenneths-MacBook-Pro.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64
$ sysctl hw.logicalcpu
hw.logicalcpu: 2

include latest FULL version of vsc-base

Tcl module fixes

…o numcpus

…dn't work

Conflicts: vsc/README.md vsc/__init__.py vsc/utils/generaloption.py

…debug info

… avaiable number of cores

…e purging *all* loaded modules

boegel · 2013-11-01T00:12:58Z

This is awaiting the merge of wpoely86#4, in which the open issues are fixed, and which makes this ready for merging in.

long has unlimited precision. EB needs to be ready for systems with 64 or more cores ;-)

wpoely86 · 2013-11-03T13:56:55Z

Merged wpoely86#4

I've added one commit to use long instead of int. We want easybuild to be ready for >32 core systems afterall... 😉

boegel · 2013-11-04T08:40:49Z

@wpoely86: I see why you've added to extra commit, it makes sense, but it's not really needed.

With Python 2.6:

>>> long('fffffffffffffffffffffffffff', 16)
324518553658426726783156020576255L
>>> int('fffffffffffffffffffffffffff', 16)
324518553658426726783156020576255L

Nevertheless, I like the change, it makes it more explicit that it should be robust for > 32 cores.

This one is now ready to go in, thanks @wpoely86!

number of available cpus (in cpuset)

Keep cpuset in mind with setting parallelism

80bd425

On Linux, try to find out if we're a member of a cpuset and find the number of CPUs in it. If not, continue with the old behaviour. This should also be done on BSD...

boegel reviewed Sep 30, 2013
View reviewed changes

wpoely86 added 2 commits September 30, 2013 14:28

Small improvements to get_avail_core_count()

b5a2d2e

Print warning when not in Linux in get_avail_core_count()

2dfd627

boegel reviewed Sep 30, 2013
View reviewed changes

get_avail_core_count(): made code more compact

2cb27c5

Merge branch 'develop' into numcpus

55300fd

wpoely86 and others added 2 commits October 18, 2013 15:38

Added deprecated warning for get_core_count()

a7f0ea2

deprecate get_core_count, update vsc.utils

871bd1d

boegel and others added 16 commits October 28, 2013 14:30

include latest FULL version of vsc-base

81d0ae3

disable declaring of vsc namespace

5618806

Merge pull request easybuilders#740 from boegel/full_vsc_base

5e5b2bc

include latest FULL version of vsc-base

Merge pull request easybuilders#739 from boegel/tcl_module_fixes

bb647d5

Tcl module fixes

improve comments

e107198

Merge branch 'numcpus' of github.com:wpoely86/easybuild-framework int…

b635418

…o numcpus

readd sysctl approach for non-Linux systems, and fail hard it that di…

af40b65

…dn't work

Merge branch 'numcpus' into test

dbd0ee6

Conflicts: vsc/README.md vsc/__init__.py vsc/utils/generaloption.py

check both get_core_count and get_avail_core_count in unit test, add …

0206394

…debug info

fix syntax error in test_core_count

041e75e

fix another syntax error in test_core_count

99315ae

provide previous code as fallback in case sched_getaffinity is broken

b2bb259

rework fallback in get_avail_core_count, cpuset ranges is a filter on…

1b0999e

… avaiable number of cores

clean up implementation via proper mask and counting bits

46c324f

restore environment after each modules.py unit test since they involv…

c787619

…e purging *all* loaded modules

restore environment after purging modules in options.py unit tests

7ce51f9

boegel mentioned this pull request Nov 1, 2013

deprecate get_core_count, update vsc.utils (+ update develop) wpoely86/easybuild-framework#4

Merged

boegel and others added 2 commits November 2, 2013 15:00

fix remark

22059ad

get_avail_core_count(): use long instead of int

6ae37cf

long has unlimited precision. EB needs to be ready for systems with 64 or more cores ;-)

boegel added a commit that referenced this pull request Nov 4, 2013

Merge pull request #700 from wpoely86/numcpus

4f18d16

number of available cpus (in cpuset)

boegel merged commit 4f18d16 into easybuilders:develop Nov 4, 2013

wpoely86 deleted the numcpus branch November 4, 2013 08:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

number of available cpus (in cpuset) #700

number of available cpus (in cpuset) #700

wpoely86 commented Sep 30, 2013

boegel Sep 30, 2013

wpoely86 Sep 30, 2013

wpoely86 Sep 30, 2013

boegel Sep 30, 2013

boegel Sep 30, 2013

wpoely86 commented Sep 30, 2013

boegel Sep 30, 2013

boegel commented Sep 30, 2013

wpoely86 commented Sep 30, 2013

boegel commented Oct 1, 2013

fgeorgatos commented Oct 1, 2013

boegel commented Oct 1, 2013

fgeorgatos commented Oct 1, 2013

wpoely86 commented Oct 1, 2013

fgeorgatos commented Oct 1, 2013

wpoely86 commented Oct 2, 2013

boegel commented Oct 2, 2013

wpoely86 commented Oct 2, 2013

boegel commented Oct 2, 2013

fgeorgatos commented Oct 2, 2013

boegel commented Oct 2, 2013

wpoely86 commented Oct 2, 2013

boegel commented Oct 2, 2013

boegel commented Oct 18, 2013

wpoely86 commented Oct 18, 2013

boegel commented Oct 18, 2013

boegel commented Nov 1, 2013

wpoely86 commented Nov 3, 2013

boegel commented Nov 4, 2013

number of available cpus (in cpuset) #700

number of available cpus (in cpuset) #700

Conversation

wpoely86 commented Sep 30, 2013

boegel Sep 30, 2013

Choose a reason for hiding this comment

wpoely86 Sep 30, 2013

Choose a reason for hiding this comment

wpoely86 Sep 30, 2013

Choose a reason for hiding this comment

boegel Sep 30, 2013

Choose a reason for hiding this comment

boegel Sep 30, 2013

Choose a reason for hiding this comment

wpoely86 commented Sep 30, 2013

boegel Sep 30, 2013

Choose a reason for hiding this comment

boegel commented Sep 30, 2013

wpoely86 commented Sep 30, 2013

boegel commented Oct 1, 2013

fgeorgatos commented Oct 1, 2013

boegel commented Oct 1, 2013

fgeorgatos commented Oct 1, 2013

wpoely86 commented Oct 1, 2013

fgeorgatos commented Oct 1, 2013

wpoely86 commented Oct 2, 2013

boegel commented Oct 2, 2013

wpoely86 commented Oct 2, 2013

boegel commented Oct 2, 2013

fgeorgatos commented Oct 2, 2013

boegel commented Oct 2, 2013

wpoely86 commented Oct 2, 2013

boegel commented Oct 2, 2013

boegel commented Oct 18, 2013

wpoely86 commented Oct 18, 2013

boegel commented Oct 18, 2013

boegel commented Nov 1, 2013

wpoely86 commented Nov 3, 2013

boegel commented Nov 4, 2013