Apparent memory leak when calling python from rust

[ajprax]

Nature of the bug: Values returned from python functions called from rust are not freed (refcount goes up but does not come down).

🌍 Environment

• Your operating system and version: ubuntu 18.04
• Your python version: 3.7
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: pyenv
• Your Rust version (rustc --version): rustc 1.50.0-nightly (349b3b324 2020-11-29)
• Your PyO3 version: 0.13.2
• Have you tried using latest PyO3 main (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?: no. I have rust-numpy in my dependencies and it turns on default-features including auto-initialize which prevents me from compiling.

💥 Reproducing

python code:

def get_py_obj():
    return object()

rust code:

pub fn leaks() -> PyResult<()> {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let module = py.import("module")?;
    module.call_method1("get_py_obj", ())?;
    Ok(())
}

Calling leaks in a loop will grow memory usage unboundedly (the values I'm working with are much larger than empty objects so this becomes a problem very quickly). This example does nothing with the returned values because I found it wasn't necessary to reproduce the leak, but it also leaks if you use the value for something (e.g. return some property of it).

this formulation also leaks.

pub fn leaks() -> PyResult<()> {
    Python::with_gil(|py| {
        let module = py.import("module")?;
        module.call_method1("get_py_obj", ())?;
        Ok(())
    })
}

this too, along with a few other variations I tried just in case.

module.getattr("get_py_obj")?.call0()?;

a python variation that prints the refcount before and after the rust call (requires that you call leaks multiple times)

import sys
prev = None
def get_py_obj():
    global prev
    obj = object()
    if prev is not None:
        # subtract one for the argument to getrefcount per the documentation
        print("after passing through rust:", sys.getrefcount(prev) - 1) # should be 1 for the variable prev but is 2
    print("before passing through rust:", sys.getrefcount(obj) - 1) # 1 for the variable obj
    prev = obj
    return obj

[birkenfeld]

I cannot reproduce this bug with Python 3.7.10 / pyo3 0.13.2 / rustc 1.53.0-nightly (07e0e2ec2 2021-03-24). I also tried with your specific rustc nightly and that did not reproduce either.

I used only code from the issue (except for a fn main() that just calls leaks() in a a loop).

Are you sure you've minimized your test case properly?

[ajprax]

One possible difference is that the entry point to my program is python. Tomorrow I'll put together and end to end repro and test a rust entry point.

…

On Sun, Apr 11, 2021, 2:09 AM Georg Brandl ***@***.***> wrote: I cannot reproduce this bug with Python 3.7.10 / pyo3 0.13.2 / rustc 1.53.0-nightly (07e0e2ec2 2021-03-24). I also tried with your specific rustc nightly and that did not reproduce either. I used only code from the issue (except for a fn main() that just calls leaks() in a a loop). Are you sure you've minimized your test case properly? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPCTEABIXHFHGMJ7LKNPBLTIFRMRANCNFSM42NV2MGQ> .

[davidhewitt]

If your entry point is Python then #1556 sounds very similar. The repro would be very helpful, thank you!

[ajprax]

Ok, I think #1556 does explain the issue. Ultimately it comes down to an oversight on my part (that a #[pyfunction] acquires the GIL even if it's not explicitly using it, a thing which I should have known since I use allow_threads in other places to avoid blocking other python code but didn't think about).

What was happening was, I was acquiring the GIL on crossing from python into rust, then inside the rust calling back into python in a loop. In each pass of the loop I was apparently acquiring and dropping the GIL, but those were no-ops because it was already acquired and therefor the reference counts were not decremented (because the GIL and its pool were still held) causing the apparent leak. Adding allow_threads seems to fix the issue.

I will say, it's a bit of a gotcha that reference counts aren't decremented until the GIL is properly dropped even if it appears dropped locally, and also that a pyfunction acquires the GIL even if it isn't using it.

[birkenfeld]

You're right, it is definitely unexpected. The guide kind-of mentions it in "Advanced topics", but it is far from obvious.