docs of deprecated C function should propose replacement in C, not in Python

[jwuttke]

Documentation

In the 3.13 Python/C API Reference Manual, page "Initialization, Finalization, and Threads", Py_GetProgramName() and several other C functions are declared deprecated. To replace them, it is suggested "Get sys.executable instead".

As maintainer of a C program, I find this not helpful. I would need a drop-in replacement in C, not a reference to a Python variable.

Linked PRs

• gh-129342: Explain how to replace Py_GetProgramName() in C #129361

[serhiy-storchaka]

Like PySys_GetObject("executable")?

[jwuttke]

    PyObject* obj = PySys_GetObject("executable");
    PyObject* repr = PyObject_Repr(obj);
    PyObject* str = PyUnicode_AsEncodedString(repr, "utf-8", "~E~");
    const std::string s = PyBytes_AS_STRING(str);

kind of works, but it returns the full path, not just the executable name.

[serhiy-storchaka]

Don't use PyObject_Repr(), it does not do what you think it does.

PyUnicode_AsUTF8AndSize() can be more convenient than PyUnicode_AsEncodedString() + PyBytes_AS_STRING() to you.

I suggested PySys_GetObject("executable") as a way to get the sys.executable value. If it is not equivalent to the Py_GetProgramName() result, the suggestion in the documentation is not correct. cc @vstinner

[jwuttke]

Thanks.

These replacements for Py_GetProgramFullPath and Py_GetPath work for me:

    {
	PyObject* obj = PySys_GetObject("executable");
	PyObject* repr = PyObject_Str(obj);
	const std::string s = PyUnicode_AsUTF8AndSize(repr, NULL);
	std::cout << "  - executable: " + s + "\n";
    }
    {
	PyObject* obj = PySys_GetObject("path");
	PyObject* repr = PyObject_Str(obj);
	const std::string s = PyUnicode_AsUTF8AndSize(repr, NULL);
	std::cout <<  "  - Python PATH: " + s + "\n";
    }

Replacement of Py_GetPythonHome seems to require a somewhat different solution.

[jwuttke]

https://docs.python.org/3/c-api/init.html#c.Py_GetPythonHome says: "Get PyConfig.home or PYTHONHOME environment variable instead."

Retrieving a Python enviroment variable from C seems feasible but lengthy. So I tried PyConfig.home:

    {
	PyConfig config;
	PyConfig_InitPythonConfig(&config);
	auto status = PyConfig_Read(&config);
	if (PyStatus_Exception(status))
	    throw std::runtime_error("Cannot access PyConfig");
	std::cout << "  - Python home: " + toString(config.home) + "\n";
	PyConfig_Clear(&config);
    }

Seems to work. Is there a simpler solution?

[jwuttke]

@serhiy-storchaka I just see on PEP731 that you are member of the C API working group. So you are the right person to ask: WHY these deprecations? They break production code, and now it turns out that there is not even a 1:1 replacement...

[vstinner]

Retrieving a Python enviroment variable from C seems feasible but lengthy. So I tried PyConfig.home: (...)

I added PyConfig_Get() from PEP 741 "Python Configuration C API" to Python 3.14. The pythoncapi-compat project makes the function available on Python 3.13 and older. Would it be a good replacement for your use case?

Py_GetPythonHome() can be replaced with PyConfig_Get("home").

[jwuttke]

Of course a 1:1 replacement is much better than a multi-line workaround.

Is there a similarly simple solution for Py_GetProgramFullPath and Py_GetPath?

Nonethess, my question goes: WHY these deprecations? Have you really well pondered aginst the damage done (not least to Python's reputation) by breaking things?

[vstinner]

• Py_GetProgramFullPath() can be replaced with PyConfig_Get("executable") or PySys_GetObject("executable").
• Py_GetPath() can be replaced with PyConfig_Get("module_search_paths") or PySys_GetObject("path").

Would you mind to elaborate on your use case? You call Py_GetProgramFullPath(), Py_GetPath() and Py_GetPythonHome() after Python initialization? What is the purpose of reading these variables?

Nonethess, my question goes: WHY these deprecations? Have you really well pondered against the damage done (not least to Python's reputation) by breaking things?

In Python 3.8, I designed and implemented PEP 587 "Python Initialization Configuration" which unifies the "Python initialization". Slowly (since Python 3.8), I deprecated the old way to configure Python.

How can I help you to migrate your code towards newer APIs which are not deprecated? Does PySys_GetObject() and/or PyConfig_Get() answer to your question?

[serhiy-storchaka]

Sorry, it was discussed before I became the member of the C API working group, so I am not aware of arguments. I guess this is related to handling global state at different stages of Python initialization and shutdown, and maybe to supporting multiple interpreters.

But we should be more conservative with deprecating the public C API. For every deprecated function we should document an alternative that works in all maintained Python versions. If PyConfig_Get("home") is the only alternative to Py_GetPythonHome(), the latter should not be deprecated until all maintained Python versions have PyConfig_Get(), i.e. not earlier than 3.17.

[jwuttke]

My use case: in a complex software with C++ core and Python API [1,2], we want to introspect the state of an embedded Python interpreter for various debugging purposes.

Anyway, functions Py_GetProgramName() etc have been part of the public API, and you should not be surprised that they have been used and that production code depends on them.

I had hoped that the Python community had learned from the forced upgrade from Py2 to Py3. But no, you are breaking things within major 3, with relatively short transition periods. This is in stark contrast with the emphasis C++ is putting on backwards compatibility, and it raises serious doubts whether C++ and Python is a good combination in a software project that is meant to be maintained for decades to come.

This said, a well-documented 1:1 drop-in replacement like PyConfig_Get("home") for Py_GetPythonHome() is much much better than my above three-line replacement for Py_GetProgramFullPath.

But why not leave Py_GetPythonHome() in the API for the lifetime of Python major 3? You may still reimplement it internally as a trivial wrapper of PyConfig_Get("home").

[1] https://www.bornagainproject.org/
[2] https://jugit.fz-juelich.de/mlz/bornagain

[vstinner]

I just ran a code search on PyPI top 8,000 projects using regex Py_GetPythonHome|Py_GetProgramFullPath|Py_GetPath. I found a single project using these functions:

• pythonnet (3.0.4)

I recently postponed these deprecations from Python 3.14 to Python 3.15 since PEP 741 landed in Python 3.14. If the delay is too short, we can postpone the deprecation even later.

[jwuttke]

Regarding my own needs, the above workaround just works. Thank you for your kind offer, but no need to postpone deprecation just for me. If I keep insisting here, then it's not for my own immediate needs but for the sake of the user community at large.

I am not convinced by your statistics. Top 8k is not enough if there are 600k projects. And more importantly, C projects with embedded Python interpreter don't typically go to PyPI. I just did a quick GitHub code search. 100s of hits for Py_GetPythonHome() and PyGetPath().

[vstinner]

I wrote documentation PR gh-129361 to explain how to replace Py_GetProgramName() in C, rather than in Python.

Example:

Py_GetProgramName(): Use PySys_GetObject("executable") (sys.executable) or PyConfig_Get("executable") instead.

---

I am not convinced by your statistics. Top 8k is not enough

It's just to get a coarse idea on the ratio of impacted projects. The answer on top 8k is: "more than zero" :-) but the ratio is low.

PEP 741 lists a few projects embedding Python. But usually these projects only "set" the configuration, they don't need to "get" the configuration.

[vstinner]

But why not leave Py_GetPythonHome() in the API for the lifetime of Python major 3? You may still reimplement it internally as a trivial wrapper of PyConfig_Get("home").

The problem of this API is that the returned string lifetime is undefined: the caller doesn't have to call free() on it. The string has to remain valid "forever". It's not possible to reimplement it using PyConfig_Get("home").

Moreover, since we have to store a copy of this string (to respect the API lifetime), the string can become inconsistent if the related sys attribute / PyConfig member is updated. For example, if you modify sys.executable to make the path absolute (or any other change), Py_GetProgramFullPath() will return the old executable unchanged. Fixing this issue by updating the string copy can lead to crash if the API result is used after the change (dangling pointer).

The implementation is non-trivial because it has to use a specific memory allocator (_PyMem_SetDefaultAllocator(PYMEM_DOMAIN_RAW, &old_alloc);) and we have to sync PyConfig with these copies at some points.

I did my best to maintain backward compatibility when I implemented PEP 587 (PyConfig). But the backward compatibility has a cost in term of maintenance burden. Deprecating (and removing) a few functions should reduce this cost.

I'm sorry that you're application is impacted and that the documentation on how to update wasn't great.

I'm aware of these problems and that's why I designed PEP 741 which might enter the stable ABI in the future. The stable ABI prevents these annoyance since the ABI is guaranteed to stay unchanged. In Python 3.14, it's not in the stable ABI yet.

[jwuttke]

Thank you, Victor, for these explanations and for the perspective that things will become more stable in the future.

[jwuttke]

One more question please: is my replacement code correct, or do I need to add some free statements to prevent dangling pointers and memory leaks?

    {
        PyConfig config;
        PyConfig_InitPythonConfig(&config);
        auto status = PyConfig_Read(&config);
        if (PyStatus_Exception(status))
            throw std::runtime_error("Cannot access PyConfig");
        std::cout << "  - program name: " + wToString(config.program_name) + "\n";
        PyConfig_Clear(&config);
    }
    {
        PyObject* obj = PySys_GetObject("executable");
        PyObject* repr = PyObject_Str(obj);
        const std::string s = PyUnicode_AsUTF8AndSize(repr, NULL);
        std::cout << "  - executable: " + s + "\n";
    }

[vstinner]

You should not call PyConfig_InitPythonConfig() to get the configuration: this API is to set the configuration. Use PyConfig_Get() or PySys_GetObject() instead.

[serhiy-storchaka]

Don't call PyObject_Str(). The result is normally a string, and if it is not a string, you will likely need to handle it in other way (raise a C++ exception, or use special placeholder, or omit adding this value to the debug info). You need also to check if the results of PySys_GetObject() and PyUnicode_AsUTF8AndSize() not NULL. This can happen if sys.executable was deleted (or not yest set) or the string contains lone surrogates.

Normally, you should check every C API call for errors.

[jwuttke]

Use PyConfig_Get() or PySys_GetObject() instead.

How? Where are the docs? https://docs.python.org/3/c-api/index.html offers no way to read the manual in one page so that one could do a full-text search. It has a search widget near the bottom though. However, search for PyConfig_Get yields no hit.

[jwuttke]

Don't call PyObject_Str()

How to do without? Calling PyUnicode_AsUTF8AndSize directly on obj doesn't work.

[serhiy-storchaka]

It should work if obj is a Python string (use PyUnicode_Check() to check this, but first check for NULL).

[jwuttke]

    {
        PyObject* obj = PySys_GetObject("executable");
	if (!obj)
	    throw std::runtime_error("Cannot identify Python executable");
	if (!PyUnicode_Check(obj))
	    throw std::runtime_error("PySys_GetObject('executable') does not return string");
        const std::string s = PyUnicode_AsUTF8AndSize(obj, NULL);
        result += "  - executable: " + s + "\n";
    }
    {
        PyObject* obj = PySys_GetObject("path");
	if (!obj)
	    throw std::runtime_error("Cannot identify Python path");
	if (!PyUnicode_Check(obj))
	    throw std::runtime_error("PySys_GetObject('path') does not return string");
        const std::string s = PyUnicode_AsUTF8AndSize(obj, NULL);
        result += "  - Python PATH: " + s + "\n";
    }

works for "executable" but not for "path". For the latter, PyUnicode_Check returns 0.

[serhiy-storchaka]

Yes, because sys.path is a list, not a string.

If you only want to format a string containing some Python settings, it may be easier to write that code in Python and then execute it with PyRun_String(). It may need much less intermediate error checking and reference counters management.

[AA-Turner]

in the API for the lifetime of Python major 3

Note, Python does not use SemVer or any similar versioning scheme (it predates them), we have a policy on deprecations and removals, which describes our commitments to backwards (in)compatibility.

[jwuttke]

@AA-Turner PEP 387 is excellent reading indeed. It says "the incompatibility should be easy to resolve in affected code". Where "easy" probably means less than: spending a whole working day without arriving at a valid replacement in spite of substantial help from two volunteeers here.

[vstinner]

Fixed by change 632ca56. Thanks for the bug report.