Allow specifying /proc and other needed folders

[wojons]

The use case of this is running monitoring tools or scripts built around psutil that run inside docker continents. With docker the admin can specify that the /proc of the host machine is read only and mounted in a location on the sub continent the issue for this is that /proc room the host maybe /proc_host and then you need to read things from there for monitoring. An issue that comes up is CoreOS and a few other docker OS dont have package managers so there is no way to really install tools there its expected that the user installs the tools in a docker container.

[giampaolo]

Mmm... it's not clear to me why anyone would want to change the default /proc location as I suppose a lot of different system tools makes the assumption that /proc lives in "/proc", not only psutil. Is this something you do explicitly or is it Docker which does it?

[wojons]

Yes I agree in 99.99% of the time /proc is /proc. So CoreOS which is a OS made for docker it does not have a package manager so there is no way to install new software on the server it assumes that everything you need to do can run from another docker container. And if that docker continer needs access to /proc for getting monitoring data then you mount that /proc from the host OS to some other location on the container.

[giampaolo]

Yes. I would say that another possible approach here in case /proc is /proc_host or whatever is to just create a symlink to /proc. This has the downside that it requires root privileges though. Is that something you can do by default with CoreOS? What is the default /proc location in CoreOS? I'm wondering whether that can be determined at runtime.

[wojons]

Not sure if you used docker before or openvz. but each container is acting like its own operating system but shares the same kernel as the host but uses cgroups to limit the scope of processes that the container can see. I can mount the /proc easily to /proc_host but its hard to get the system to work correctly if i mount /proc of the host over the /proc of the container.

[ggueret]

+1 with the following use case : mount the host rootfs in a docker container and get some metrics from psutil, like cAdvisor "--volume=/:/rootfs:ro".

[giampaolo]

OK, I'm convinced there's an use case which justifies this change.
The problem is how to expose this feature in terms of API.
The Linux implementation lives in the psutil._pslinux namespace:
https://github.com/giampaolo/psutil/blob/master/psutil/_pslinux.py
...which psutil imports as "_psplatform":

psutil/psutil/__init__.py

Line 62 in 999dc85

from . import _pslinux as _psplatform

The first thing which pops into my mind is to define a PROCFS_PATH constant in _pslinux.py which the user can change like this:

import psutil
psutil._psplatform.PROCFS_PATH

I'm not too happy about this though because we are accessing a private module. I can't think of any other solution though (also renaming the module doesn't fully convince me).
Thoughts? @jloden @josiahcarlson

[josiahcarlson]

I hate to say it, but what about a module property in psutil? Dirty, but then you aren't hiding it in a private module. Make it do the right thing on all platforms (like an exception on platforms where it doesn't exist).

Or don't even make it a property, make it a constant, and make psutil._psplatform.* check psutil.PROCFS_PATH whenever it is looking for the value, and it can just be a global like any other. You can import psutil in _psplatform functions that need it, or reference it directly from sys.modules based on __name__. Stuff the module property in _psplatform.PROCFS_PATH, which checks the attribute in psutil.

GIampaolo, you always have fun programming problems :)

P.S. At least one of those suggestions is a joke, can you guess which one(s)? Trick or Treat ;)

[giampaolo]

Thanks for chiming in Josiah. =)

I hate to say it, but what about a module property in psutil?

I knew you'd say that. =)

You can import psutil in _psplatform functions that need it.

Not possible. This would create a circular import because it's psutil (aka psutil/__init__.py) which imports _pslinux.

reference it directly from sys.modules based on __name__

This is more interesting. I have tried to import psutil; psutil.PROCFS_PATH = 'foo' then print sys.modules['psutil'].PROCFS_PATH from _pslinux.py but it doesn't get updated (it prints the old value). Is this what you were suggesting? Or was this the joke? =)

Another possitility would be to provide a psutil.set_procfs_path() function in psutil namespace which changes _psplatform.PROCFS_PATH value. Perhaps this is the cleanest solution, even though I hate to provide a brand new function just for this. :-(

[josiahcarlson]

Funnily enough, the module property actually answers the problem pretty well, and wasn't the joke.

If sys.modules['psutil'].PROCFS_PATH doesn't get updated when you update psutil.PROCFS_PATH, then you really need to check the __name__ and __file__ values in sys.modules['psutil'] and _pslinux.py, because I bet they are referring to different packages.

When I pull down the git repo, do a quick make, add the attribute to __init__.py, and add a simple function to _pslinux.py:

def do_stuff():
    import sys
    return sys.modules[__name__.rpartition('.')[0]].PROCFS_PATH

I get the following behavior:

>>> import psutil
>>> psutil.PROCFS_PATH
'/proc'
>>> from psutil import _pslinux
>>> _pslinux.do_stuff()
'/proc'
>>> psutil.PROCFS_PATH = '/other/path'
>>> _pslinux.do_stuff()
'/other/path'

So that one should work, if you are using the right __name__ in your references.

You can import psutil in _psplatform functions that need it.

Not possible. This would create a circular import because it's psutil (aka psutil/init.py) which imports _pslinux.

Importing psutil inside a function (like I do with sys above in the do_stuff() function) is one way to avoid exceptions caused by circular imports without injecting stuff into __builtins__ or using sys.modules as I did above in do_stuff(). The sys.modules reference may be the cleanest without resorting to a module property.

[giampaolo]

If sys.modules['psutil'].PROCFS_PATH doesn't get updated when you update psutil.PROCFS_PATH,
then you really need to check the name and file values in sys.modules['psutil'] and
_pslinux.py, because I bet they are referring to different packages.

This was the issue indeed. Cool! I will stick with psutil.PROCFS_PATH as to me it seems the cleanest API from the user standpoint. Thanks Josiah. I will not tell you I owe you another beer because with all the beers I owe you'd probably need a liver transplant. :D

[josiahcarlson]

Always happy to help. Save the beer money for your travel, and we'll run into each other again somewhere between LA and Turin :D

[ggueret]

Works well, thanks a lot : )

[giampaolo]

;-)