Windows python3 executable #99185
Comments
|
@zooba do you have any proposals for this? On unix, we have symlinks up to the executable with the minor version:
pip's case is a bit different, all the executables are launchers for a Python module. |
Hmm, I checked about 10 of the top Google results for |
|
Thanks for the reply and attention.
True, though I believe on Windows, python.exe simply invokes functions in python3X.dll. Here's a directory listing: The executable stubs are even smaller than the pip ones: Yes, symbolic links exist on Windows, but only for NTFS, not FAT32. IMO, perhaps just following pip's lead and copying the executable stubs might be the lowest friction solution.
Well, the first result...python.org...currently uses The REAL problem is that there is inconsistency between platforms and documentation. Likewise, Anyway, point is there is inconsistency across platforms. And while some inconsistency is unavoidable, these seem unnecessary and do cause confusion for students...and distress for teachers. If we can just get ...I guess asking for a default |
You mean 100 kB. The python.org tutorial explains what to use for different systems, showing |
|
Pedantry aside, yes my CS majors must of course learn the intricacies, though you'd be surprised at how little motivation many show to learn even the basics of terminal operations. I can and will hold these students accountable as it is a learning outcome. HOWEVER, students in other majors take classes or are simply given ancillary assignments in Python to provide exposure and understanding (and recruiting), not to master it. They copy/paste blocks of code that they find online or that I write. Having to constantly provide "translations" and then having them still encounter roadblocks and frustrations as they have to seek and wait potentially hours for assistance due to the need to add or remove a '3'...is avoidable pain. Pain that can needlessly drive away potentially talented technology majors, which is counter to our mission. This is adding little to this issue. |
AFAIK it parses the command line and envvars and invokes the interpreter via
This is actually a Debian (and naturally Ubuntu and anything else Debian-based) specific thing btw, other distributions don't have that.
And IIRC there's a toggle setting to enable support for them 🙃
Well, pip's executable stubs (aka packaging entrypoint launchers) are a bit different, it's not as straight-forward here. Like I said above, I am pretty sure the executable is more than just a stub, so we need to create stubs.
No, if we are fixing the inconsistencies, I think it should be done for all platforms. On macOS it can be done via symlinks though, I am actually surprised you mentioned it doesn't already do this.
I agree, it's unfortunate. If you are teaching introduction to Python, I would consider using IPython or Jupyter though. Anyway, I am glad to see you are advocating for your students, I wish my teachers did that when I was in college 😅 |
|
Thanks for the clarifications, @FFY00!
Surprised me, too. I just installed the latest from python.org just to make sure: Would also be nice to have it do the same for pip as there's only a /usr/local/bin/pip3 and pip3.x. Debian/Ubuntu .deb package seems also not to add a /usr/bin/python or /usr/local/bin/python symbolic link. python3-pip DOES however have pip, pip3 and pip3.x script replicas. Different teams making the installers, I guess. FWIW, I'll go to my grave not understanding the real distinction between /usr/bin and /usr/local/bin. 😆
Agreed!! We include Python in a (very) intro course, a "scripting and automation" course, database, and data science. Each kinda lends itself to different environments. Intro uses some kind of simple IDE (currently IDLE maybe?). Automation I think uses Mu, Data Science uses Jupyter. And my latest pain is in Database, where I need them to "experience" accessing a database from an application...here I use VSCode and CLI since they are connecting to a local MariaDB daemon. I guess the challenge of having such a powerful, general purpose scripting language is that it is used in so many different ways... |
|
I've been hesitant to add more executables to the Python install directory, because enough users already have messed up PATH that adding more is only going to be a bigger problem. Any changes at all are going to cause churn, which is going to break people, which is going to use up any remaining goodwill we have left. I'd rather save the churn until we actually have a good, long-term fix. But I also don't know what that fix will be. My best guess right now is that all the scenarios that would benefit from having "python3" be slightly more consistent (it's still not going to behave the same) would benefit far more from using a distro (who can easily do this themselves) or a preinstalled/configured environment such as via Jupyter or Minecraft. The python.org install of Python is such a long way from being a beginner friendly option that I'd rather leave that space clearly open for someone who wants to focus on that audience. The Store install is a significant concession, but also properly handles the PATH modifications so I'm far less concerned about multiple executables (and the stub that launches the store was added by Microsoft in support - we don't control that from here, though I personally have the contacts to make changes if needed). That said, if they don't fix up the execution alias UI some time soon I'll seriously consider dropping some of the entry points. |
To offer a really concrete example here:
|
|
I understand, and agree with your point regarding churn, however if we go that route I think we should decide what is our stance regarding installing versioned executable in general and document it, along with whatever exceptions we might have (eg. Windows if we decide that we want to install versioned executables as a general practice, or Linux if we decide we don't). Having somewhere in the documentation where we can point people to learn what exactly is the difference in behavior between the different platforms would go a long way to mitigate this issue. |
and As a "convenience" maybe coming up with a managed "standard" method of setting the current version that is consistent across platforms? Perhaps pysetver or something. It could present a list of known installs (in the path on Windows and common install locations elsewhere) and allow you to pick one to set as active? Would modify the path on Windows and update the symlink elsewhere? Kinda sorta like venv, but for the base install? Dunno. Might just be more churn...
Doesn't fix the copy/paste issue, but I agree that it is a good middle ground if no universal fix can be envisioned. |
This would be great, but unfortunately doesn't work well (on Windows at least). A process can't change the environment of its parent process, which would mean it would have to be a batch file and a PowerShell script and a Bash script to handle the three most popular shells (on Windows - Git Bash gets a lot of use), and nobody has signed up for that.
This is actually my ideal design, but it requires OS changes to work properly. I'm trying to argue those into existence before I give up and look for an alternative (potentially renaming the A non-terminal interface such as Jupyter, Mu or VS Code, is always going to avoid this level of confusion. It's a shame, but also pretty irreversible, that terminal skills are a prerequisite to using Python. |
True...but...to change the path permanently, you're modifying the registry using setx or similar. Which will require closing and reopening the command prompt anyway. So using C/Win32 or python + the winreg module could do this. You'd just need to print something like "Changes won't be effective until you close and reopen the command prompt" at the end. Would be happy to throw together some proof-of-concept code for this if interested. But, yes, a more elegant "work immediately" option would need to BOTH modify the registry and refresh the current environment from the registry. Sadly, an age-old issue since...ages old. Like me. :D And one that would indeed require a bat/ps1/sh script.
Interesting. Would be the "most similar" to symlinks on other platforms. But I can imagine this would almost have to be an "opt in" option. At least if other Python installs are detected. No perfect solutions here, huh? At least without a Tardis. |
|
maybe python can ship with a "manage with pyenv" as the standard installation, so users who want some facility with managing versions and the whole python3, 3.X executable thing can have that managed for them, and we can decouple the "manage python version stuff" repo from "cpython". be nice if cpython wasn't even in the business of managing the active version, paths, etc. |
|
I've proposed such a project at work, and there's a bit of interest in doing it. I do think the best path out of here is to let python.org Python keep doing its thing, but introduce something new for managing installs that works with the exact same runtimes, but not the original installers. That way, people who keep downloading the same installer they always have been won't break, but we can start moving people who want a better experience onto the new thing, hopefully without forcing the burden of a totally new project onto the volunteer team. |
On Windows, the Python version 3 installation from https://www.python.org/downloads/ does not come with a python3.exe file. See here: python/cpython#99185 Added some Windows specific instructions (call python instead)
This is a valid concern but frankly I think it's less annoying than not having a cross platform way to run Python 3!! E.g. in GitHub CI I can't use |
|
I came here after we ran into the same cross-platform problem as many others --
I was curious too 🙂 so had a good rummage through the issues as I know there are StackOverflow questions about this from years ago, because I've just read 'em. I found the following previous discussion from 2014 useful: #65705: [Windows] installations should include pythonX.exe and pythonX.Y.exe executables (originally titled " Windows MSI installer should mklink (symlink) python.exe to python2.7.exe" 😮.) The earliest mention of this problem I could find here was in this comment in 2009: #51134 (comment) Just when I think I'm close to understanding the trade-offs I discover #72872 py.exe ignored PATH when using python3 shebang and this comment:
So now I'm really not sure what the best approach is even for running a script on Windows, let alone cross-platform wise 🤔. |
Unfortunately, the best way is for the person running it to know how they launch Python. This one also works cross-platform 😉 The next best way is for you to include a full copy of Python with your script and include a shortcut/launcher that makes it appear like a normal executable. (Pynsist and Briefcase both generate suitable installers, and some other tools like Pyoxidizer pack it into a single executable. The embeddable distro is meant for this purpose and is only ~8MB) If you're in a situation where you have a trivial script (no installed dependencies) and you can't rely on your users to know how to launch Python, then yeah, this situation isn't great. But I'd contend that this is also the situation most likely to be harmed by inconsistent PATH, dependency conflicts, and such. One day someone may contribute the support to concatenate a zip file to the py.exe launcher, which would then let you distribute an executable containing your script that assumes your users have already installed Python. I personally would never inflict that upon my users, but people seem to want to do it 🤷♂️ |
|
The "reverse" solution to this is to add a common This is basically how MS does it with Slightly changing the subject, the current windows store stub+install I find to be exceptionally confusing. Because it's the only Windows In other words, Installing a new Python3 version updates Plus an "aspiring to be savvy" user will get extra confused finding that, not only is the executable in an entirely different directory tree than the actual install, but the It's a bit of a mess if I'm being honest (and obviously I am). If there's a way to just wrap the regular installer into the MSIX package (I think there is), then that might be more desirable here. Anything to help remove inconsistency seems a worthy project. |
I agree, but I don't want to go breaking the world by changing stuff without being confident we're landing in a good spot. The regular installer has no shortage of issues, as does an MSIX install, as does using redirector executables, as does using symlinks, as does using PATH, as does having an annual incompatible release, etc. Solving one problem risks making others worse (evidence: half the things I just listed were added to solve problems). And a lot of things are in motion that could fundamentally change everything again (e.g. proposed PEP 711). There's a growing consensus that globally installed runtimes are a bad design, which I've believed for a while but it's way more widely accepted now, especially among those who are the focal points for complaints about them. While there are papercuts in the current situation, as long as it doesn't change people can figure out how to work within it. If we're changing it slightly every year, nobody will have a chance. So we're basically staying put until it's clearer what they way forward should be. |
|
I don't think you will find a perfect solution for this. Your options are:
Python installation is already enough of a total mess that adding The best time to plant a tree... |
|
Personally what I'm struggling to understand is why, if the Windows Store package includes a I freely admit I know very little about the Python ecosystem on Windows or Linux so appreciate that things are complicated but I don't believe you can ignore how people are invoking Python on Linux simply because this issue is about Python on Windows. Python is a cross-platform tool and teams are expecting fundamental stuff like As an aside what brought me to this conversation was someone adding mypy as one of the linters to our (cross-platform) build pipeline. It's [1] I'm basing that entirely on this comment in the Windows Store issue: #79158 (comment) so maybe things changed after that and it got dropped. I fully understand why it's needed in the Windows Store package -- to replace the stub from Microsoft. |
|
tl;dr: We assume people installing from python.org know how to launch Python.
The main assumption is that people who choose to go to python.org to download an installer won't bother to get it from the Store. And a secondary assumption is that those who know to go to python.org also know how to launch Python (and so won't use Clearly this second assumption is occasionally incorrect, as people who have installed from python.org still follow instructions that (apart from Microsoft's stub) won't ever have worked for them. One other mitigating factor that makes it safer to add the extra executable to the Store install is that there can be only one So assuming that most people who install the Store package have not installed from python.org, and are simply following instructions to get stuff to run, it makes sense and is safe to provide both
I generally agree, but my experience of tree planting in Python installers so far has been a huge wave of literal hatred and demands for things to change back (or for me to lose my job, etc.). Please excuse me for being hesitant to volunteer to face that alone again. |
Beating a dead horse, but I think this is probably not a 100% valid assumption. People get started with Python using a number of paths. For example, my students are given the instructions for using WinGet, Homebrew, or downloading/installing. All install the Python.org version. And again, despite every effort to explain and differentiate instructions for Mac & Windows users, I will be hit with "why doesn't this command work?" questions multiple times on every assignment. Sure, maybe a few aren't destined to be superstar coders (or even very good at following instructions), but they shouldn't be frustrated and discouraged by this (see especially the pip issues bellow). Other users might start a journey of learning Python on their own from one source. Stop. Restart after a while from another source. And both may give different "getting started" instructions. I don't think a person installing from Python.org can be assumed to be experienced. I'll also just clarify current Windows installer behavior for those not as familiar. Not Installed (clean Windows install)Both WindowsStore Python Install
No changes are made to PATH, so the executables in user\WindowsApps are found if no other Python installs exist. Note also that the Python.org InstallThis one installs Both user\Python and user\Python\Scripts are added to PATH immediately above user\WindowsApps. This DOES install the handy py launcher, placing So now, EXCEPT now both And it's further exacerbated if the more common novice mistake happens and both Python.org and WindowsStore versions are identical. In that case only The above it obviously just one example. But similar chaos happens any time the WindowsStore and Python.org installers are mixed and matched. Which for a novice is FAR from unlikely. It's happened quite a number of times with my students. Usually in the opposite direction, having installed from python.org but unwittingly installing the WindowsStore version by typing The multiple ways of launching python is confusing and troubling, but the issues that arise from pip and pip3 ambiguities might be the most impactful. Certainly the hardest for a novice (or even a professor) to diagnose.
I TRULY hate that for you!! The pressure to please everyone when it is impossible is...terrible. And I can definitely relate to having your a$$ handed to you simply for doing what you thought was right or best. And it must be an order of magnitude worse given the size of the Python install base and how many are impacted by your decisions. I also hear you that fixing the inconsistencies might break a small minority of users' code. And that even though this minority are also the ones most capable of diagnosing and fixing the issues (or maybe reading a README before jumping on an upgrade), they are the vocal ones who'll be coming for you with tar and feathers in hand. And I COMPLETELY respect the desire to take your time to get it right. I won't pretend to know the best/right answer. This thread has certainly shown why it's tricky. But I do hope the team/community will consider "planting a tree" or "ripping off the band-aid" at a point in the near future. I don't know if the formal PEP process can cover this. But perhaps a public vetting of a solution might help relieve some of the personal pressures. And at least allow the tar and feathers to be distributed more broadly. |
I'd just like to clarify that this isn't a case of "newbies following Linux instructions that obviously won't work on Windows".
Interesting, but doesn't that mean you can get the |
Granted, but what alternative do we have? Assume that people using the installer don't know how to launch Python? What then? Add a message to the installer (oh look, it's already there)? Add links to getting started docs (also already there)? Restrict the ways Python can be launched to avoid conflicts (but all the ways are currently used by someone)? The vast majority of users of the installer do know how to launch Python, because they already have been. Those who are completely new must have used some kind of reference to find the site, and so whoever pointed them that way can also mention it (or they can look at the site or the installer, which recommend
The same issue arises frequently on Linux, and so the general recommendation for a few years now has been to use
That's the theory behind the distutils removal. We'll see how much fallout comes my way when 3.12 ships (we all agreed it needed to be done, but my name's on the PEP, even though others did the actual removal).
There are a million other ways to get into this situation though, and the only good way out of it is for a user to learn how to launch the Python they want, or to learn to install a different tool and launch a different command. Adding I acknowledge that it may be better after that transition period. But given the sheer amount of discussion going on right now about changes like these and on similar timescales, I'm not going to make any changes right now without that side of things settling. |
Counterpoint: I have the MSYS2 bin path in my In contrast, your example is only ephemeral until those old versions go out of support. Also, I would consider the issue that a security update modifies Footnotes |
This strikes me as an odd thing to do. Why would the 3.10 installer move the position of itself in the |
|
A non newbie/tutorial use case is that in conda python is packaged very minimally meaning even in conda there is no python3 alias (eventough it is a virtual env). There are then other packages packaged with scripts that use a python3 shebang as they were primarily developed on Linux and it just works. However when called from an env on Windows it either doesn't work or calls the global install. Then the only real option (without manually modifying the environment) is to wrap the call with python to the qualified path of the script within the env install rather just being able to call the cmdlet from within the environment. However I spotted above, that shebangs potentially bypass the path entirely on windows which also seems like a big problem. |
|
Giving this some more thought, I think we can probably make venv create a Because a venv isn't active by default, and when you activate one it puts itself at the front of PATH, this misses my concerns about having the installer modify PATH. And it also overrides a Store install (or integrates with one if you create the venv from a Store install), which is otherwise potentially going to cause confusion when it "works" but launches the wrong thing. |
|
I also just posted #114881 which I think could help cover some of the gap for users who want safer PATH settings. Interested in people's thoughts over there on whether it would be usable enough (one day it could become the default, but it's one of these scenarios where we need to inform people who made their code work 5 years ago and stopped listening to anything we say, so they won't find out until they - or more likely, their users - break...) |
|
I still think your concerns about backwards compatibility are unwarranted. The current behaviour is a total mess. What's the point of being backwards compatible with that? IF there was no risk of mixing up Python versions currently, and adding Here are your options:
I mean, fixing it in |
|
The problem with that table is that Python Adding Adding |
|
Pardon the drive-by comment, but IMO the |
Well no, because the launcher is only available on Windows. That's the whole point of this issue. I suppose if you can convince all the Mac and Linux distros (Homebrew, Ports, Debian, Fedora, Arch, etc. etc.) to add a
Absolutely, but this problem is already awkward enough. There's no way it's worth adding additional difficulty by trying to use a "nice" name. |
I think Brett's work on python launcher gets pretty close. It's already part of homebrew, linuxbrew, Fedora, and Arch, leaving only Ports and Debian from your list, but it's also trivially installable on Linux distros. Moreover, the project has ambitions to support Windows at which point it could be a drop in for the Python Launcher for Windows and possibly distributed with CPython. Maybe our energy would best be spent focusing on delivering that more easily to more platforms and completing the Windows support. |
(Though I don't really see why it couldn't tbh.) If it could be distributed with CPython then that would be great, but that's a whole lot of ifs and work that isn't even planned... If it were me I would just add python3.exe now, as Microsoft has already done, and then try and do a long term fix. But I've banged on about this enough now; I think you get the message! Let's reconvene in 5 years 😄 |
How does this work with virtual environments? From what I understand, the current windows launcher will also override python versions from a virtual environment when not directly called (shebangs, via tooling etc). OS wide installs are the least controlled way to use python (or many other tools) and therefore virtual environments are used more and more to create this certainty. Pushing solutions that continue to work around the problem and undermine virtual environments don’t really help. |
See the searching for interpreters guidance for details, but tl;dr, it has virtualenv support. |
Afraid not, Brett's launcher has about 1% of the functionality of our current one, and he has no desire to add the rest (nor would I recommend adding the rest - shebang emulation turns out to be nearly impossible to get right). It could certainly be distributed on other platforms though. |
ISSUE:
On Windows, a Python 3 install directory contains
python.exe. However, there is nopython3.exe,python3.cmdor similar. Thus, a user following almost any online tutorial, copy/pasting installation scripts, etc. will fail because all are written for *nix platforms that standardize onpython3as the name of the executable.Worse, there is a
python3.exein a defaultAppData\Local\Microsoft\WindowsAppsdirectory which is a stub to open the Windows Store to download Python 3. So the unsuspecting user is greeted with the option to install Python 3 from the Windows Store, which could break an existing installation and certainly will not do what they intended.This is obviously only an issue for novice users. But I teach novice users. And this comes up numerous times each semester despite posting FAQ's and warnings related to this issue. And it seems very easy to fix in the default install.
This seems like it should be an issue long since reported, but I cannot find it by searching. So I apologize if this is a duplicate.
REQUESTED FIX:
One option is simply to add a copy of
python.exenamedpython3.exein the base install directory. This appears to be how PIP is handled, withpip.exe,pip3.exeandpip3.X.exeall being copies of the same executable located in a Scripts subdirectory.Alternatively, adding a
python3.cmdscript that calls python.exe and forwards all args would serve the same function. Though I'm unsure that saving 100MB would be worth the potential confusion.Linked PRs
The text was updated successfully, but these errors were encountered: