PEP440 following CI Versions

[daxpryce]

Maybe I missed this somewhere, but as far as I can tell pyo3-pack has no support for it:

I'm trying to set up a CI pipeline, both for our in-works stuff with snapshot builds, as well as our release pipeline. Unfortunately, I can't seem to make both pip and Cargo happy at the same time.

To start with, let's stick with a definition of a CI generated build ID. It'll be in the format of {YYYY}{MM}{DD}{MonotonicallyIncreasingDailyBuildNumber}, with the build number being left padded to 3 places, e.g. "20190422003", where today's date is 2019-04-22 and we're on the 3rd build of the day. I'll call this {BuildID} in my examples down below.

From what I can tell, Cargo is adamantly opposed to a {Major}.{Minor}.{Patch}.dev{BuildId}, which is a fair enough pre-release format that complies with PEP440 and pip's --pre flag. Example: 2.1.3.dev20190422003

Unfortunately, the Cargo way of doing CI builds is very strictly in the form of {Major}.{Minor}.{Patch}-{SomeAllowableString}{BuildId} , e.g. 2.1.3-dev20190422003. The hyphen is most important, and it's something Cargo does not seem capable of budging on. And since our version is the came as our version in our Cargo.toml, our wheel gets generated as my_artifact_2.1.3-dev20190422003-cp36-cp36m-manylinux1_x86_64.whl. I can publish this to a snapshot repo, but my_artifact_2.1.3-dev... is not going to work with pip's --pre flag; it just ignores it.

If we were in a mono repo this wouldn't even be close to a problem, but we aren't, so it is. It would be great if we could specify some version metadata that would get appended to the version prior to constructing our wheels. I superficially thought I could just rename the wheel and maybe update some metadata in a pinch, but with the sha256 checksums it looked like it wasn't going to be that straight forward.

Is there any way that you can think of to make pip --pre happy AND Cargo? If not, can we add a flag that simply appends a "fixed" string to the version that pyo3-pack is going to generate with (fixed in that it wouldn't need to be anything special to pyo3-pack - just a string append operation, but something I can construct in our build pipeline and pass in)?

[konstin]

What platform are you on and what version of pyo3-pack are you using?

I've tried to build the get-fourtytwo example with version = "2.1.0-dev2" on linux with pyo3-pack 0.5.0 (the latest release) on got a wheel called get_fourtytwo-2.1.0_dev2-cp36-cp36m-manylinux1_x86_64.whl (notice the underscore in the version). pyo3-pack should automatically escape version (and other metadata) with underscores:

https://github.com/PyO3/pyo3-pack/blob/96b0187bfd5fc8cfe3f880807a9bbf08e2eb673f/src/metadata.rs#L227-L233

I've uploaded the wheel to TestPyPI (https://test.pypi.org/project/get-fourtytwo/2.1.0-dev2/) and could successfully install it with --pre:

$ pip install -i https://test.pypi.org/simple/ --pre get-fourtytwo
Collecting get-fourtytwo
  Downloading https://test-files.pythonhosted.org/packages/18/a9/3f91553ea53341518398287ef99daf450d6d2443ef0ae7fda3d5b7298350/get_fourtytwo-2.1.0_dev2-cp36-cp36m-manylinux1_x86_64.whl (111kB)
    100% |████████████████████████████████| 112kB 851kB/s 
Installing collected packages: get-fourtytwo
Successfully installed get-fourtytwo-2.1.0.dev2

PEP 440 also explicitely allows using ., - and _ as separator (https://www.python.org/dev/peps/pep-0440/#development-release-separators) , so cargo's stricter semver shouldn't lead to any problem.

[daxpryce]

You know what? You're absolutely right.

For the record, I am using pyo3-pack 0.6.0-alpha2; as for platform, my build pipelines are for all 3 major OSes, 64 bit only, python 3.6 and python 3.7.

I think the reason I ran into problems is I started off by building versions as "1.2.3-SNAPSHOT" -- that's the maven/gradle still lurking in me, I guess. I could publish this easily enough, but pip install --pre would always ignore it unless I specified an exact version, which was counter to my CI goals.

I mistakenly presumed that meant that 1.2.3-{anything} would fail as well, so I didn't even try a 1.2.3-dev1234. Instead I tried to massage it into 1.2.3.dev1234, which Cargo wasn't too keen on:

(essex-graphs-rust) dwayne@valinhall:~/source/essex-graphs-rust/packages/pyo3$ cat Cargo.toml 
[package]
name = "essex-layouts-pyo3"
version = "1.2.0.dev1234"
authors = ["Dwayne Pryce <[email protected]>"]
edition = "2018"

[lib]
name = "essexlayoutspyo3"
crate-type = ["rlib", "cdylib"]

[dependencies]
pyo3 = ">= 0.6.0-alpha.2, <0.6.0-alpha.3"
essex-graphs-core = { path = "../core" }
chrono = "0.4"


[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]
(essex-graphs-rust) dwayne@valinhall:~/source/essex-graphs-rust/packages/pyo3$ python --version
Python 3.6.7
(essex-graphs-rust) dwayne@valinhall:~/source/essex-graphs-rust/packages/pyo3$ pyo3-pack build --release -i python3.6
Cargo metadata failed: error during execution of `cargo metadata`: error: failed to parse manifest at `/home/dwayne/source/essex-graphs-rust/packages/pyo3/Cargo.toml`

Caused by:
  Extra junk after valid version: .dev1234 for key `package.version`

Since you said that 2.1.0-dev2 worked with pip install --pre just fine, I gave it a try too - and it worked. My guess is that pip install --pre is happy enough with semver-dash-string as long as that string also has a number in it? It was a red herring in the end.

Anyway, we can close this - I now have a working pre-release version working that makes cargo, pyo3-pack, and pip happy, so that means I'm happy. Well, as happy as one can be before they try to wrangle a bunch of CI build agents into doing what they want, I guess.

Thanks!