psutil cannot create Process object, OSX #135

giampaolo · 2014-05-23T14:40:04Z

From [email protected] on November 16, 2010 17:17:04

What steps will reproduce the problem?  
1. Build and install from psutil-0.2.0.tar.gz
2. Get current process pid  
3. create a psutil.Process object 

What is the expected output?  


What do you see instead?  
Here is cut/paste from my terminal:

bash-3.2$ python -c "import psutil; import os; p=os.getpid(); print 
psutil.Process(p)"
python(97058) malloc: *** mmap(size=140733193650176) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python(97058) malloc: *** mmap(size=140733193650176) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
psutil.Process (pid=97058, name='Python') 

What version of psutil are you using? What Python version?  
psutil version  0.2.0
Python 2.6.1 

On what operating system? Is it 32bit or 64bit version?  
64bit OS X 10.6.4 Please provide any additional information below.

Original issue: http://code.google.com/p/psutil/issues/detail?id=135

The text was updated successfully, but these errors were encountered:

giampaolo · 2014-05-23T14:40:07Z

From g.rodola on November 16, 2010 08:36:23

I've seen this already and I remember it occurred with Python 2.6.1 only.
Those messages does not come from psutil but most likely from Python itself.
Could you please try to update your Python version (say 2.6.6 or 2.7.0) and try again?

You might want to use virtualenv if you want to avoid messing with default 
Python: http://pypi.python.org/pypi/virtualenv

Labels: OpSys-OSX

giampaolo · 2014-05-23T14:40:07Z

From [email protected] on November 16, 2010 09:27:24

I think this error is coming from the commandline code for OS X, when we 
allocate memory for the process argument space. Can you try running something 
like this: 

    python -c "import psutil; import os; p=os.getpid(); print 
psutil.Process(p).name"

and see if you still get any errors? Also you can try 


    python -c "import psutil; import os; p=os.getpid(); print 
psutil.Process(p).cmdline"

I suspect you'll get an empty list ( Issue #83 )

Thanks

giampaolo · 2014-05-23T14:40:08Z

From [email protected] on November 16, 2010 09:49:08

Seems it is a problem limited to Python 2.6.1.  Also, the subject of this 
report isn't correct - a process *is* created, even under 2.6.1. FWIW, the 
output of the same commands with two different python versions:

/usr/bin/python -V; /usr/bin/python -c "import psutil; import os; 
p=os.getpid(); print psutil.Process(p).cmdline; print psutil.Process(p).pid"; 
Python 2.6.1
python(45212) malloc: *** mmap(size=140733193650176) failed (error code=12)

*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
None
45212

/opt/local/bin/python -V; /opt/local/bin/python -c "import psutil; import os; 
p=os.getpid(); print psutil.Process(p).cmdline; print psutil.Process(p).pid"; 
Python 2.6.6
['/opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python',
 '-c', 'import psutil; import os; p=os.getpid(); print 
psutil.Process(p).cmdline; print psutil.Process(p).pid']
45219

giampaolo · 2014-05-23T14:40:09Z

From g.rodola on November 16, 2010 09:57:15

I'm not sure, but it seems to be related: http://bugs.python.org/issue2901

giampaolo · 2014-05-23T14:40:09Z

From [email protected] on November 16, 2010 10:30:19

Similar error but I can't see how it would be related since that was in the 
socket module. 

I can't imagine why it's only showing up in Python 2.6.1 on 64bit systems 
though (I'm running the stock 2.6.1 but on 32bit and there's no issue here). In 
any case, if we've verified it works under python 2.6.6 and 2.7 then I'm not 
sure if we should chase this down any further. 

I'm planning to rewrite the code for cmdline on OS X anyway, so this may go 
away with the re-org too.

giampaolo · 2014-05-23T14:40:10Z

From [email protected] on November 16, 2010 14:36:36

I committed completely rewritten cmdline code as r823 ; if you have the 
time/inclination you can try this with the latest trunk and see if that makes a 
difference as well.

giampaolo · 2014-05-23T14:40:11Z

From [email protected] on November 21, 2010 08:35:17

Status: WaitingForReview

giampaolo · 2014-05-23T14:40:11Z

From [email protected] on January 27, 2011 03:58:30

I am also facing similar issue with psutil. Machine details : OSX 10.6.3 
(64bit) having python 2.6.1.

I just executed following line of code 
for i in psutil.process_iter():
      print i

Result: Code a number of 'malloc' related errors. Here is the snippet from terminal.


"python(204) malloc: *** mmap(size=140733193650176) failed (error code=12)

*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
psutil.Process (pid=1, name='launchd')
psutil.Process (pid=0, name='kernel_task')
Traceback (most recent call last):
File "", line 1, in
File "/Library/Python/2.6/site-packages/psutil/init.py", line 437, in
process_iter
for pid in pids:
MemoryError: getcmdargs(): insufficient memory for procargs

Due to project requirement I can't switch to another version of Python. Any 
help in this regard highly appreciated !!!

Thanks,
Anil

giampaolo · 2014-05-23T14:40:12Z

From g.rodola on January 27, 2011 04:06:29

I still think this is something wrong with Python 2.6.1.
In comment #6: https://code.google.com/p/psutil/issues/detail?id=135#c6 ...Jay 
told he rewritten cmdline code in r823 .
If you can't upgrade to a newer 2.6.* version you can check whether the problem 
still occurs with the latest version from trunk, altough I think it makes 
difference.

giampaolo · 2014-05-23T14:40:13Z

From [email protected] on January 27, 2011 07:17:37

My development machine is 10.6.6 with the default/stock Python 2.6.1 so this is 
not likely a general problem with 2.6.1 itself. 

Are you using the binary download, or building from source? What psutil version 
are you using - is it the latest? If you're not already, can you try building 
from the latest SVN source and see if the problem still occurs?

What do the system and python resource usage counts look like during this time? 
Is there any possibility it's running out of memory? The error message simply 
indicates "cannot allocate memory" so it could mean any number of things, but 
let's try and at least rule out an out of memory condition. 

-Jay

giampaolo · 2014-05-23T14:40:13Z

From [email protected] on August 03, 2011 08:20:10

On OS X 10.7 with Python 2.7.2 and psutil compiled from svn, running

python -c "import psutil; import os; p=os.getpid(); print 
psutil.Process(p).cmdline"

gives:

python(17681) malloc: *** mmap(size=140733193650176) failed (error code=12)

*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "", line 1, in
File
"/Users/wnielson/Projects/Envs/chronograph/src/psutil/psutil/init.py", line
195, in cmdline
return self._platform_impl.get_process_cmdline()
File "/Users/wnielson/Projects/Envs/chronograph/src/psutil/psutil/_psosx.py",
line 85, in wrapper
return callable(self, _args, *_kwargs)
File "/Users/wnielson/Projects/Envs/chronograph/src/psutil/psutil/_psosx.py",
line 129, in get_process_cmdline
return _psutil_osx.get_process_cmdline(self.pid)
OSError: [Errno 12] Cannot allocate memory

I have ~1GB of free memory.

giampaolo · 2014-05-23T14:40:14Z

From g.rodola on August 19, 2011 04:57:48

I still cannot reproduce the problem.
Is cmdline property the only one causing this problem?
Could you please try to run the test suite (cd psutil; python setup install; 
python test/test_psutil.py) and paste the result?

giampaolo · 2014-05-23T14:40:15Z

From [email protected] on August 21, 2011 08:16:48

I installed psutil like this:
svn checkout http://psutil.googlecode.com/svn/trunk/ psutil-read-only
cd psutil-read-only
sudo python setup.py 

All these give me the same exception:
python -c "import psutil; import os; p=os.getpid(); po = psutil.Process(p); 
print po.name"
python -c "import psutil; import os; p=os.getpid(); po = psutil.Process(p); 
print po.cmdline"
python -c "import psutil; import os; p=os.getpid(); po = psutil.Process(p); 
print po.exe"

python(42634) malloc: *** mmap(size=140733193650176) failed (error code=12)

*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "", line 1, in
File "psutil/init.py", line 183, in exe
cmdline = self.cmdline
File "psutil/init.py", line 195, in cmdline
return self._platform_impl.get_process_cmdline()
File "psutil/_psosx.py", line 85, in wrapper
return callable(self, _args, *_kwargs)
File "psutil/_psosx.py", line 129, in get_process_cmdline
return _psutil_osx.get_process_cmdline(self.pid)
OSError: [Errno 12] Cannot allocate memory

while these works:
python -c "import psutil; import os; p=os.getpid(); po = psutil.Process(p); 
print po.create_time" 
     ==> 1313939596.09
python -c "import psutil; import os; p=os.getpid(); po = psutil.Process(p); 
print po.uids" 
    ==> user(real=501, effective=501, saved=501)

Here's the output of running python test/test_psutil.py: 
http://pastebin.com/nYWMbCYQ Hope this helps, feel free to request more info.

giampaolo · 2014-05-23T14:40:15Z

From g.rodola on August 21, 2011 09:14:52

Attaching test results in case the link to pastebin.com won't work in the future.

Attachment: nYWMbCYQ.txt

giampaolo · 2014-05-23T14:40:16Z

From [email protected] on August 21, 2011 09:23:07

Also, I'm running osx 10.7 (64-bit kernel) and python 2.7.1

giampaolo · 2014-05-23T14:40:16Z

From [email protected] on September 13, 2011 08:08:31

I've the same problem on OSX Lion 10.7 (64-bit) and Python 2.7.1 psutil 0.3.0

giampaolo · 2014-05-23T14:40:17Z

From [email protected] on September 15, 2011 18:27:30

I did a little debugging with this issue and found that it may be because 
KERN_ARGMAX is quite large in OSX 10.7 (and apparently 10.6.4 based on the 
first post).

To be exact the maximum argument size is set to 140733193650176 bytes or 
approximately 140 TB. I have no idea why the size is so high.

The code in process_info.c:get_arg_list() then mallocs that size of memory.

As a temporary workaround I change process_info.c:get_arg_max() to limit the 
maximum size returned.

size_t
get_argmax()
{
    size_t argmax;
    int mib[] = { CTL_KERN, KERN_ARGMAX };
    size_t size = sizeof(argmax);

    if (sysctl(mib, 2, &argmax, &size, NULL, 0) == 0) {
        if(argmax > 65737) {   // Added this check to make this work
           return 65737;           // Temporarily return an arbitrary limit so 
we don't try to allocate tons of memory
        }
        return argmax;
    }
    return 0;
}

Long term, I don't think it's good to pre-allocate that much memory. ARGMAX can 
be set very, very high depending on what UNIX flavor is in use, but I'm not 
sure of the best way to handle this.

giampaolo · 2014-05-23T14:40:18Z

From [email protected] on September 15, 2011 18:46:43

KERN_ARGMAX is supposed to be the maximum size of command line arguments, so 
that doesn't make much sense to allow TBs worth of data, which seems incredibly 
excessive. There must be something else going on here, but it's hard to debug 
better without access to Lion.

giampaolo · 2014-05-23T14:40:18Z

From [email protected] on September 15, 2011 18:50:09

Maybe there is a way to determine the size of the actual command line argument 
space for the process and allocate the actual amount needed rather than trying 
to allocate the argmax before calling sysctl() with KERN_PROCARGS2. Will have 
to investigate a little bit and try some test code.

giampaolo · 2014-05-23T14:40:19Z

From [email protected] on September 16, 2011 06:06:48

Sorry, at least part of my last post was inaccurate.
mymacbook:psutil$ sysctl kern.argmax
kern.argmax: 262144

This means argmax *should be* 262144 which is much more reasonable than 140TB! 
I'm on Mac OSX 10.7.

It looks like maybe something's wrong with the call to sysctl that's making it 
get the wrong value, but it doesn't jump out at me right now what that might 
be. I can probably look at it again sometime this weekend if no one else has 
figured it out by then.

giampaolo · 2014-05-23T14:40:20Z

From [email protected] on September 16, 2011 06:26:37

Ahh thanks, that's what I suspected since that looked a lot like an overflow or 
cast conversion error. I think the sysctl() call is structured differently for 
Lion, maybe a size_t isn't the right type of variable to use for argmax on 10.7.

giampaolo · 2014-05-23T14:40:20Z

From [email protected] on September 16, 2011 07:55:52

Yes, I suspect you're right. The Mac documentation uses an int in their example 
when retrieving a different value (KERN_MAXPROC). KERN_ARGMAX is also listed as 
being an integer type. 

I'm at work so don't have my Mac. Later I'll try it using an int and see what happens.

giampaolo · 2014-05-23T14:40:21Z

From [email protected] on September 16, 2011 08:10:19

If I change the sysctl() call to use an int instead of a size_t for KERN_ARGMAX 
in get_argmax it works fine here (10.6.8 64bit), but then it was working here 
to begin with :) 

Let me know if it works ok on Lion and if so then I'll commit the changes and 
hopefully that will resolve this issue for you and the other folks running into it.

giampaolo · 2014-05-23T14:40:22Z

From [email protected] on September 16, 2011 10:33:14

Changing the type to int fixes the issue for me, running OSX 10.7.1.

giampaolo · 2014-05-23T14:40:22Z

From [email protected] on September 16, 2011 10:41:51

I've committed changes as r1122 , it'd be great if anyone running OS X (all 
versions) can test the current SVN build and ensure it works properly for them. 
I can only test on my 10.6.8 64bit box so I'd like to hear from other folks to 
make sure the change didn't break other platforms.

giampaolo · 2014-05-23T14:40:23Z

From [email protected] on September 16, 2011 11:28:48

looks good, but after running tests I got the following output:

Attachment: test_output.txt

giampaolo · 2014-05-23T14:40:23Z

From [email protected] on September 16, 2011 11:35:30

The python != Python thing happens on OS X because of the way the process is 
launched, it's not a bug in psutil just a peculiarity of OS X's python 
installations. I could have sworn I fixed it in the test case previously but I 
guess not, or the change was reverted at some point. 

In any event other than that the only other issue in the test case is in 
test_get_connections_ipv6 which is unrelated to this issue, so as far as the 
original bug report everything looks good.

giampaolo · 2014-05-23T14:40:24Z

From [email protected] on September 16, 2011 11:48:19

committed test suite changes as r1123 so that should take care of the name 
mismatches on OS X

giampaolo · 2014-05-23T14:40:25Z

From [email protected] on September 16, 2011 11:48:29

Status: FixedInSVN

giampaolo · 2014-05-23T14:40:25Z

From g.rodola on September 19, 2011 11:06:02

Jay if this is fixed please update HISTORY file in accordance.

giampaolo · 2014-05-23T14:40:26Z

From [email protected] on September 21, 2011 13:33:33

Can anybody test this on other OS X versions? Giampaolo, if you've got 10.5 can 
you test there and make sure the changes didn't break anything else? If we're 
good to go then I'll update the HISTORY file and call it done.

giampaolo · 2014-05-23T14:40:27Z

From g.rodola on September 22, 2011 04:15:19

Everything's fine on 10.5.8 but note that I've never bumped into the original 
issue in the first place.

giampaolo · 2014-05-23T14:40:27Z

From g.rodola on September 23, 2011 08:31:25

Labels: Milestone-0.3.1

giampaolo · 2014-05-23T14:40:28Z

From g.rodola on October 21, 2011 16:44:17

Labels: -Milestone-0.3.1

giampaolo · 2014-05-23T14:40:29Z

From g.rodola on October 21, 2011 16:45:26

Labels: Milestone-0.4.0

giampaolo · 2014-05-23T14:40:29Z

From g.rodola on October 28, 2011 20:44:14

Status: Fixed

giampaolo · 2014-05-23T14:40:30Z

From g.rodola on March 02, 2013 03:56:57

Updated csets after the SVN -> Mercurial migration: r823 == revision 
31161e959cff r1122 == revision 3e5cc84772dc r1123 == revision c1df2f5d8211

giampaolo self-assigned this May 23, 2014

giampaolo closed this as completed May 23, 2014

pyup-bot mentioned this issue Jun 2, 2017

Pin psutil to latest version 5.2.2 crowddynamics/crowddynamics#123

Closed

pyup-bot mentioned this issue Jul 21, 2017

Pin psutil to latest version 5.2.2 mozilla-services/autopush#975

Closed

pyup-bot mentioned this issue Aug 16, 2017

Pin psutil to latest version 5.2.2 springermac/pyinstaller#37

Closed

giampaolo added the macos label Nov 15, 2020

psutil cannot create Process object, OSX #135

psutil cannot create Process object, OSX #135

Comments

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014

giampaolo commented May 23, 2014