Confusion around python package management with mise. Maybe room for improvement in the docs?

[gaufde]

I've used mise for a while, but at an extremely basic level. So far, all I've been doing is basically this:

mise use [email protected]
python3 -m venv .venv
source .venv/bin/activate
(.venv) pip install scikit-image

However, as a PyCharm user the GUI even takes care of some of those last steps for me.

I've been reading the docs to try and understand what else mise can do for me, and I was pretty confused about the package management features. I think I've figured it out now, but I'll explain my journey since that might be useful.

For example, one question I had, was can mise help me manage packages installed in my venv? There is a custom backend for pipx that says you can use it to install python packages. I was not already familiar pipx, but it seems like it can manage some but not all python packages. For example, because scikit-image is not a CLI-type application, it doesn't appear to work:

Example running `mise use pipx:scikit-image --verbose` throws an error

gaufde:~/test/ > mise use pipx:scikit-image --verbose
DEBUG ARGS: /run/current-system/sw/bin/mise use pipx:scikit-image --verbose
DEBUG EnvResults
DEBUG config: ~/test/mise.toml
DEBUG config: ~/.config/mise/config.toml
DEBUG install_some_versions: pipx:scikit-image@latest
INFO  pipx:[email protected] install
INFO  pipx:[email protected] pipx install scikit-image==0.25.0
DEBUG EnvResults
DEBUG EnvResults
DEBUG $ pipx install scikit-image==0.25.0
creating virtual environment...
creating shared libraries...
upgrading shared libraries...
installing scikit-image from spec 'scikit-image==0.25.0'...
No apps associated with package scikit-image. Try again with '--include-deps'
to include apps of dependent packages, which are listed above. If you are
attempting to install a library, pipx should not be used. Consider using pip
or a similar tool instead.
INFO  pipx:[email protected] Note: Dependent package 'numpy' contains 2 apps
INFO  pipx:[email protected]   - f2py
INFO  pipx:[email protected]   - numpy-config
INFO  pipx:[email protected] Note: Dependent package 'imageio' contains 2 apps
INFO  pipx:[email protected]   - imageio_download_bin
INFO  pipx:[email protected]   - imageio_remove_bin
INFO  pipx:[email protected] Note: Dependent package 'tifffile' contains 4 apps
INFO  pipx:[email protected]   - lsm2bin
INFO  pipx:[email protected]   - tiff2fsspec
INFO  pipx:[email protected]   - tiffcomment
INFO  pipx:[email protected]   - tifffile
ERROR pipx failed
Error:
   0: failed to install pipx:[email protected]
   1: pipx exited with non-zero status: exit code 1

Location:
   src/cmd.rs:462

Version:
   2024.12.17 macos-arm64 (1980-01-01)

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

I guess this fits with the idea that mise is neither a system package manager like brew, nor a language package manager like pip. Then, I noticed that buried in the docs for the pipx backend, it is mentioned that one can have the pipx backend use uv under the hood. Again, I was not already familiar with uv, but it seems to be a replacement for both pipx as well as pip. So, that re-ignited the idea that maybe mise can fully set up the venv with required packages. So, I ran mise use -g uv to install it. The docs say that it should automatically replace pipx, so I try mise use pipx:scikit-image --verbose again:

Example running `mise use pipx:scikit-image --verbose` throws an error when using uv

gaufde:~/test/ > mise use pipx:scikit-image --verbose
DEBUG ARGS: /run/current-system/sw/bin/mise use pipx:scikit-image --verbose
DEBUG EnvResults
DEBUG config: ~/test/mise.toml
DEBUG config: ~/.config/mise/config.toml
DEBUG install_some_versions: pipx:scikit-image@latest
INFO  pipx:[email protected] install
INFO  pipx:[email protected] uv tool install scikit-image==0.25.0
DEBUG EnvResults
DEBUG EnvResults
DEBUG $ uv tool install scikit-image==0.25.0
Resolved 9 packages in 53ms
Installed 9 packages in 85ms
 + imageio==2.36.1
 + lazy-loader==0.4
 + networkx==3.4.2
 + numpy==2.2.1
 + packaging==24.2
 + pillow==11.1.0
 + scikit-image==0.25.0
 + scipy==1.15.0
 + tifffile==2024.12.12
INFO  pipx:[email protected] No executables are provided by `scikit-image`
ERROR uv failed
Error:
   0: failed to install pipx:[email protected]
   1: uv exited with non-zero status: exit code 1

Location:
   src/cmd.rs:462

Version:
   2024.12.17 macos-arm64 (1980-01-01)

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

It looks like mise is indeed using uv as expected, but it still isn't able to install the scikit-image package for me.

The python cookbook shows this section of code:

[tasks.install]
description = "Install dependencies"
alias = "i"
run = "uv pip install -r requirements.txt"

Which makes me think that uv (via mise ?) could be doing a bit more with regards to package management, but it also shows that uv is working alone for project-specific package management. I was pretty confused about what I should expect from these tools right now, but my guess now is that the cookbook is the best example at the moment.

One thing I haven't tried yet is the python default packages feature. That feature specifically calls out pip, so I'm assuming it works as expected.

Lastly, because this became a bit of a stream-of-thought, here are the questions that the docs didn't immediately answer for me:

• What level of package management is the pipx backend aimed at? What is the use case for this feature for people who aren't already familiar with pipx?
• What is the level of support for uv? Why does it appear to be another supported backend that doesn't have it's own documentation?
• Does the hidden uv backend do exactly the same thing as the pipx one, considering uv seems to have a broader feature set than pipx?
• Can mise handle managing project-specific python packages inside a venv? It can create and activate venvs. It can install default packages. Beyond that, it seems like tasks are probably the real answer, but I went down some confusing trains of thought to sort this out.

[egnor]

See more discussion here: #323

For automatic setup, the current thinking is that one can use the postinstall hook to set up a venv and prepopulate it with packages as desired -- and that furthermore, yes, the documentation needs to be updated to give examples of how to do that.

For everyday package management once things are setup, there's no particular reason mise needs to be involved? Just run your favorite package manager as you see fit? Use tasks if you want to make shortcut commands? I'm not sure I understand what you're asking about? Mise certainly isn't intended to wrap every aspect of every development tool?

[gaufde]

See more discussion here: #323

For automatic setup, the current thinking is that one can use the postinstall hook to set up a venv and prepopulate it with packages as desired -- and that furthermore, yes, the documentation needs to be updated to give examples of how to do that.

Thanks, I'll take a look at that!

For everyday package management once things are setup, there's no particular reason mise needs to be involved? Just run your favorite package manager as you see fit? Use tasks if you want to make shortcut commands? I'm not sure I understand what you're asking about? Mise certainly isn't intended to wrap every aspect of every development tool?

Yeah, this was what I understood when I first started using mise, and it is what I understand now. There was a brief period where I thought mise might be able to do more though, and I think that my confusion arose from the docs saying things like: "You may install python packages directly" on the pipx backend page. That coupled with mise handling default package installs made me think that maybe there were more features I didn't understand yet. Does that help explain the source of confusion?

I think one other part I found confusing was that the docs seem to be written with the assumption that one already understands the use case for projects like pipx or uv. Without that context, it is pretty hard to tell what the mise wrapper is supposed to do, especially once I was already down the train of thought that mise might be able to wrap more aspects of development that I was using it for.

[egnor]

(Note, I'm not authoritative about anything here, just a happy user of mise!)

What you say makes sense to me, and there is significant philosophical subtlety, and the documentation could for sure be better about spelling out the intent of the system, especially in places where there's some gray area overlap like using pipx as a mise backend.

After all all these things are "package managers" that install and uninstall bundles of software?

• Operating system package managers like apt, yum, flatpak, snap, etc.
• User-installed package managers like brew, conda, etc.
• Language specific package managers like pip, npm, cargo, etc.
• Tool managers like mise and asdf (and nix?)
• Language-specific tool managers like NVM and pyenv
• Container managers like Docker

Mise (and asdf) live in the "language agnostic tool manager" niche I think?

• Package installs are scoped to a project (working directory), creating "micro environments" in the same computer
• Package installs can be driven by a manifest allowing those "micro environments" to be reproduced
• Mostly focused on software development tools, but not specific to a particular language or ecosystem
• Mostly focused on reliably "bootstrapping" subsystems that often have their own internal package management

The last one is key I think? If you're developing in Python, you should be using pip (or pipenv or poetry or ...) and pyproject.toml (or requirements.txt or ...) to manage Python packages the Python way. If you're developing in Node, you should be using npm (or yarn or ...) and package.json to manage Node packages the Node way. Mise's role here will mainly be to get you set up with the right version of Python (and pip) or Node (and npm) so you can go from there.

The use case for pipx/uvx backends (I think??) is more "I'm not working in Python per se, but my basic toolset starting point includes something that happens to be written in Python and distributed with pip, and by using this backend I can have mise install that tool in a consistent way for anyone who uses this repository". Otherwise, having mise in the picture would just be an awkward layer of unnecessary indirection, if you've already got (say) Python set up in a project.

I'm sure @jdx would be open to PRs for documentation to clarify design intent and expected use cases!

[jdx]

I think we probably could explain a bit what pipx is on the backend page. I think there are a few gaps:

• I used the word "packages" when I should've used "tool"
• It should explain that it can use uvx or pipx in the beginning
• It should briefly cover what pipx/uvx are and how they are different than pipx/uv generally

It seems the real issue that OP had was this sentence:

You may install python packages directly from

which I understand, using the word "package" there is misleading. mise is not a package manager, mise is a tool manager. It manages CLIs only. If I were to start mise today I would never have implemented default packages but that was behavior inherited from asdf. The postinstall hook is a better boundary since it's the user that's building package management functionality, not mise.

I want to be clear about one thing though, it's not mise's role to explain how the tools mise provides work. mise provides them but if you want to learn about them you need to go to the vendor for that information. It seems the bulk of this was misunderstanding what pipx is and not noticing that the docs refer mainly to uvx not uv which is the equivalent of pipx to pip.

[egnor]

[super tangential]

I'm sensitive to this because I'm doing a dive into the world of web frameworks, and much like OP finding myself constantly confused because everyone explains why their gizmo is the best and explains the detailed method calls but never explains where things fit in their view of the Big Picture and just assume you're up on the rapidly evolving terminology.

I'd say mise does way better than usual here, this page is a decent Big Picture intro, and the beginner's guide it links to provides more of a walkthrough. The only thing I might propose adding (or even write a PR, if people agree) is a brief bit on "what mise is not". I'm not sure how many people would care, though, I think I approach understanding things differently than most users seem to.

[gaufde]

I think we probably could explain a bit what pipx is on the backend page. I think there are a few gaps:

• I used the word "packages" when I should've used "tool"
• It should explain that it can use uvx or pipx in the beginning
• It should briefly cover what pipx/uvx are and how they are different than pipx/uv generally

I think you hit the nail on the head here! Giving more complete context doesn't have to take many words. Something as simple as re-phrasing "You may install python packages directly from..." —> "You may install tools written in python..." makes a huge difference in my mind.

I want to be clear about one thing though, it's not mise's role to explain how the tools mise provides work. mise provides them but if you want to learn about them you need to go to the vendor for that information. It seems the bulk of this was misunderstanding what pipx is and not noticing that the docs refer mainly to uvx not uv which is the equivalent of pipx to pip.

I think I understand what you are saying. There is no way mise could or should cover how each tool works. Mise can manage the tools, but the user still has to decide what tools they want/need. That said, I do think that one (maybe two) sentence(s) to provide context can be appropriate, and based on your bullets above, it looks like we agree. It can be as simple as "pipx and uvx allow you to install and run end-user CLI tools written in python. With this backend, you can use mise to install such tools from PyPI, Git, etc...."

everyone explains why their gizmo is the best and explains the detailed method calls but never explains where things fit in their view of the Big Picture and just assume you're up on the rapidly evolving terminology.

@egnor I think this is an eternal problem. As someone who also produces content on technical topics, it is so easy to assume context is clear when you are the one living and breathing within that context. I have this issue all the time in my job; it can be really hard to know what background interested users might have. From the other side, when I'm reading such docs, it can sometimes take me mulling on things for a day or two before I'm able to assemble the Big Picture in my mind. Other times I don't get the Big Picture until I've learned about something completely different and all of a sudden it helps me understand a different project that didn't make sense before. Lastly, and frustratingly, the Big Picture always seems kinda obvious to me in retrospect, which can make it difficult for me to figure out what didn't make sense the first time around.

Anyways, @jdx thanks for being receptive to this feedback! I really appreciate it, and hopefully things will be clearer for the next person.