Migrate from bump2version to tbump #1756

matthewfeickert · 2022-02-02T03:46:17Z

Summary

At the moment, trying to do more complicated things, like release candidates, with bump2version is difficult if not basically impossible in an automated way (Example: The bump2version docs Semantic versioning example requires that you create a release before you can create the release candidate for it — this is backwards.).

The bump2version suggested alternatives lists tbump which seems to be a good fit. The main difference is that instead of providing the SemVer part (major, minor, patch) to bump, you provide the entire version (though you can verify the version is valid in advance).

On my fork I have it working so that with a tbump.toml based workflow I can bump the version to any major, minor, patch, or release candidate from the command line, and with the addition of a recast-atlas inspired bump-version.yml workflow we can create tags through GitHub Actions workflow dispatch events that will get pushed to master, which will set off TestPyPI publication workflows. If those pass and behave as expected, we can then make a GitHub release that can also take a note from recast-atlas and have the deployment to PyPI get triggered from there.

This would effectively remove the need for the tag.yml workflow, which has served us well, making it a bit easier to step through things slower for testing (and to roll back tags before things get to PyPI) and to switch in the future to having release branches. I believe that @kratsg was okay with that idea given that he approved and merged PR #1632, but if not we can discuss this further.

Additional Information

Example

Use Example:

$ docker run --rm -ti python:3.9 /bin/bash
root@5e614876d9ad:/# python -m venv venv && . venv/bin/activate
(venv) root@5e614876d9ad:/pyhf# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@5e614876d9ad:/pyhf# git config --global user.email "[email protected]"
(venv) root@5e614876d9ad:/pyhf# git config --global user.name "Your Name"
(venv) root@5e614876d9ad:/# git clone https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 270d59d3900fdef7549015db45260e85f83e33b6
(venv) root@5e614876d9ad:/pyhf# apt update && apt install -y vim
(venv) root@5e614876d9ad:/pyhf# python -m pip --quiet install tbump
(venv) root@5e614876d9ad:/pyhf# cat tbump.toml 
# Uncomment this if your project is hosted on GitHub:
github_url = "https://github.com/scikit-hep/pyhf/"

[version]
current = "0.6.3"

# Example of a semver regexp.
# Make sure this matches current_version before
# using tbump
regex = '''
  (?P<major>\d+)
  \.
  (?P<minor>\d+)
  \.
  (?P<patch>\d+)
  (rc
    (?P<candidate>\d+)
  )?
  '''

[git]
# The current version will get updated when tbump is run
message_template = "Bump version: 0.6.3 →  {new_version}"
tag_template = "v{new_version}"

# For each file to patch, add a [[file]] config
# section containing the path of the file, relative to the
# tbump.toml location.
[[file]]
src = "tbump.toml"

[[file]]
src = "src/pyhf/utils.py"

[[file]]
src = "README.rst"

[[file]]
src = "src/pyhf/data/citation.bib"

[[file]]
src = ".zenodo.json"

[[file]]
src = "codemeta.json"

[[file]]
src = "CITATION.cff"

[[field]]
# the name of the field
name = "candidate"
# the default value to use, if there is no match
default = ""

# You can specify a list of commands to
# run after the files have been patched
# and before the git commit is made

#  [[before_commit]]
#  name = "check changelog"
#  cmd = "grep -q {new_version} Changelog.rst"

# Or run some commands after the git tag and the branch
# have been pushed:
#  [[after_push]]
#  name = "publish"
#  cmd = "./publish.sh"
(venv) root@5e614876d9ad:/pyhf# git add tbump.toml && git commit -m "Add tbump config"
(venv) root@5e614876d9ad:/pyhf# tbump --help
Usage:
  tbump [options] <new_version>
  tbump [options] init [--pyproject] <current_version>
  tbump --help
  tbump --version

Options:
   -h --help          Show this screen.
   -v --version       Show version.
   -C --cwd=<path>    Set working directory to <path>.
   -c --config=<path> Use specified toml config file. When not set, `tbump.toml` is assumed.
   --non-interactive  Never prompt for confirmation. Useful for automated scripts.
   --dry-run          Only display the changes that would be made.
   --only-patch       Only patches files, skipping any git operations or hook commands.
   --no-tag           Do not create a tag
   --no-push          Do not push after creating the commit and/or tag
   --no-tag-push      Create a tag, but don't push it
(venv) root@5e614876d9ad:/pyhf# tbump --non-interactive --no-push 0.7.0rc1
:: Bumping from 0.6.3 to 0.7.0rc1
=> Updating current version
* Set current version to 0.7.0rc1 in tbump.toml
=> Patching files
- tbump.toml:4 current = "0.6.3"
+ tbump.toml:4 current = "0.7.0rc1"
- tbump.toml:22 message_template = "Bump version: 0.6.3 →  {new_version}"
+ tbump.toml:22 message_template = "Bump version: 0.7.0rc1 →  {new_version}"
- src/pyhf/utils.py:135 '@software{pyhf,  author = {Lukas Heinrich and Matthew Feickert and Giordon Stark},  title = "{pyhf: v0.6.3}",  version = {0.6.3},  doi = {10.5281/zenodo.1169739},  url = {https://doi.org/10.5281/zenodo.1169739},  note = {https://github.com/scikit-hep/pyhf/releases/tag/v0.6.3}}@article{pyhf_joss,  doi = {10.21105/joss.02823},  url = {https://doi.org/10.21105/joss.02823},  year = {2021},  publisher = {The Open Journal},  volume = {6},  number = {58},  pages = {2823},  author = {Lukas Heinrich and Matthew Feickert and Giordon Stark and Kyle Cranmer},  title = {pyhf: pure-Python implementation of HistFactory statistical models},  journal = {Journal of Open Source Software}}'
+ src/pyhf/utils.py:135 '@software{pyhf,  author = {Lukas Heinrich and Matthew Feickert and Giordon Stark},  title = "{pyhf: v0.7.0rc1}",  version = {0.7.0rc1},  doi = {10.5281/zenodo.1169739},  url = {https://doi.org/10.5281/zenodo.1169739},  note = {https://github.com/scikit-hep/pyhf/releases/tag/v0.7.0rc1}}@article{pyhf_joss,  doi = {10.21105/joss.02823},  url = {https://doi.org/10.21105/joss.02823},  year = {2021},  publisher = {The Open Journal},  volume = {6},  number = {58},  pages = {2823},  author = {Lukas Heinrich and Matthew Feickert and Giordon Stark and Kyle Cranmer},  title = {pyhf: pure-Python implementation of HistFactory statistical models},  journal = {Journal of Open Source Software}}'
- README.rst:292 title = "{pyhf: v0.6.3}",
+ README.rst:292 title = "{pyhf: v0.7.0rc1}",
- README.rst:293 version = {0.6.3},
+ README.rst:293 version = {0.7.0rc1},
- README.rst:296 note = {https://github.com/scikit-hep/pyhf/releases/tag/v0.6.3}
+ README.rst:296 note = {https://github.com/scikit-hep/pyhf/releases/tag/v0.7.0rc1}
- README.rst:343 .. |Docs from latest| image:: https://img.shields.io/badge/docs-v0.6.3-blue.svg
+ README.rst:343 .. |Docs from latest| image:: https://img.shields.io/badge/docs-v0.7.0rc1-blue.svg
- README.rst:356 .. |Docker Hub pyhf| image:: https://img.shields.io/badge/pyhf-v0.6.3-blue?logo=Docker
+ README.rst:356 .. |Docker Hub pyhf| image:: https://img.shields.io/badge/pyhf-v0.7.0rc1-blue?logo=Docker
- src/pyhf/data/citation.bib:3 title = "{pyhf: v0.6.3}",
+ src/pyhf/data/citation.bib:3 title = "{pyhf: v0.7.0rc1}",
- src/pyhf/data/citation.bib:4 version = {0.6.3},
+ src/pyhf/data/citation.bib:4 version = {0.7.0rc1},
- src/pyhf/data/citation.bib:7 note = {https://github.com/scikit-hep/pyhf/releases/tag/v0.6.3}
+ src/pyhf/data/citation.bib:7 note = {https://github.com/scikit-hep/pyhf/releases/tag/v0.7.0rc1}
- .zenodo.json:4 "title": "scikit-hep/pyhf: v0.6.3",
+ .zenodo.json:4 "title": "scikit-hep/pyhf: v0.7.0rc1",
- .zenodo.json:5 "version": "v0.6.3",
+ .zenodo.json:5 "version": "v0.7.0rc1",
- .zenodo.json:39 "identifier": "https://github.com/scikit-hep/pyhf/tree/v0.6.3",
+ .zenodo.json:39 "identifier": "https://github.com/scikit-hep/pyhf/tree/v0.7.0rc1",
- codemeta.json:9 "version": "0.6.3",
+ codemeta.json:9 "version": "0.7.0rc1",
- CITATION.cff:17 title: "pyhf: v0.6.3"
+ CITATION.cff:17 title: "pyhf: v0.7.0rc1"
- CITATION.cff:18 version: 0.6.3
+ CITATION.cff:18 version: 0.7.0rc1
- CITATION.cff:20 repository-code: "https://github.com/scikit-hep/pyhf/releases/tag/v0.6.3"
+ CITATION.cff:20 repository-code: "https://github.com/scikit-hep/pyhf/releases/tag/v0.7.0rc1"
- CITATION.cff:21 url: "https://pyhf.readthedocs.io/en/v0.6.3/"
+ CITATION.cff:21 url: "https://pyhf.readthedocs.io/en/v0.7.0rc1/"
=> Performing git operations
$ git add --update
$ git commit --message Bump version: 0.6.3 →  0.7.0rc1
$ git tag --annotate --message v0.7.0rc1 v0.7.0rc1
[master e96a77c] Bump version: 0.6.3 →  0.7.0rc1
 7 files changed, 19 insertions(+), 19 deletions(-)
(venv) root@5e614876d9ad:/pyhf# tbump --non-interactive --no-push 0.7.0alpha  # This fails which is good
:: Bumping from 0.7.0rc1 to 0.7.0alpha
Error: Could not parse 0.7.0alpha as a valid version string
(venv) root@5e614876d9ad:/pyhf# tbump --non-interactive --no-push 0.7.0  # This passes which is good

Related Issues

This should also be able to address:

Notes on `<major>.<minor>.<patch>rc<candidate>` format

I chose to have the <major>.<minor>.<patch>rc<candidate> format (e.g. 0.7.0rc1) as opposed to using things like <major>.<minor>.<patch>-rc<candidate> or something else as that seems to be what most of the other libraries in the SciPy ecosystem and Scikit-HEP ecosystem do for release candidates:

Notes of caution

tbump is packaged with poetry and so has a bad habit of putting unecessary upper bounds on dependencies. To avoid artificially capping things, don't add tbump to dev dependencies but instruct people to install it with pipx.
At the moment if running locally for testing you'll need to manually enable some command line options like

tbump --non-interactive --no-push <version>

Code of Conduct

I agree to follow the Code of Conduct

The text was updated successfully, but these errors were encountered:

matthewfeickert · 2022-02-02T04:15:53Z

We probably can/should also use a a custom version template on some of the files to only bump things when they aren't a release candidate.

kratsg · 2022-02-04T15:29:16Z

I dislike having poetry or pipx to be required for this one package unfortunately. It seems like a weird ask.

matthewfeickert · 2022-02-04T16:14:43Z

I dislike having poetry or pipx to be required for this one package unfortunately. It seems like a weird ask.

It isn't required as

python -m pip show tbump | grep Requires
Requires: schema, cli-ui, tomlkit, docopt

but if you put it into the dev extra then given that it is packaged with poetry you get all these unnecessary upper bound caps:

$ docker run --rm -ti python:3.9 /bin/bash
root@22de092b52cc:/# pip install tbump
Collecting tbump
  Downloading tbump-6.7.0-py3-none-any.whl (33 kB)
Collecting tomlkit>=0.5.8
  Downloading tomlkit-0.9.0-py3-none-any.whl (32 kB)
Collecting docopt<0.7.0,>=0.6.2
  Downloading docopt-0.6.2.tar.gz (25 kB)
Collecting schema<0.8.0,>=0.7.1
  Downloading schema-0.7.5-py2.py3-none-any.whl (17 kB)
Collecting cli-ui>=0.10.3
  Downloading cli_ui-0.16.0-py3-none-any.whl (12 kB)
Collecting unidecode<2.0.0,>=1.0.23
  Downloading Unidecode-1.3.2-py3-none-any.whl (235 kB)
     |████████████████████████████████| 235 kB 8.3 MB/s 
Collecting tabulate<0.9.0,>=0.8.3
  Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)
Collecting colorama<0.5.0,>=0.4.1
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting contextlib2>=0.5.5
  Downloading contextlib2-21.6.0-py2.py3-none-any.whl (13 kB)
Building wheels for collected packages: docopt
  Building wheel for docopt (setup.py) ... done
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13723 sha256=1856d71f7c528d95c7d5c2170eae45276ed659b929335deb0ab9f4e47c14bcee
  Stored in directory: /root/.cache/pip/wheels/70/4a/46/1309fc853b8d395e60bafaf1b6df7845bdd82c95fd59dd8d2b
Successfully built docopt
Installing collected packages: unidecode, tabulate, contextlib2, colorama, tomlkit, schema, docopt, cli-ui, tbump
Successfully installed cli-ui-0.16.0 colorama-0.4.4 contextlib2-21.6.0 docopt-0.6.2 schema-0.7.5 tabulate-0.8.9 tbump-6.7.0 tomlkit-0.9.0 unidecode-1.3.2

so to avoid these restrictions entering into the CI and restricting things in strange ways when pip does dependency resolution I was just suggesting that we keep it out of there.

matthewfeickert · 2022-02-08T04:38:54Z

tbump is packaged with poetry and so has a bad habit of putting unecessary upper bounds on dependencies. To avoid artificially capping things, don't add tbump to dev dependencies but instruct people to install it with pipx.

@kratsg Actually I think we can safely ignore my previous suggestion about not including in in develop as the dependencies that tbump draws in are totally unrelated to any existing dependencies of pyhf — so there is no way they might accidentally restrict things even though they have capped upper bounds because of poetry.

Here's an example using the following that we both gotten from a clean python -m pip install .[develop] in a Python 3.9 Docker container

$ diff --color -u bump2version_pip_list.txt tbump_pip_list.txt 
--- bump2version_pip_list.txt	2022-02-07 22:24:43.812892541 -0600
+++ tbump_pip_list.txt	2022-02-07 22:29:12.883861655 -0600
@@ -12,18 +12,19 @@
 black                         22.1.0
 bleach                        4.1.0
 build                         0.7.0
-bump2version                  1.0.1
 cachetools                    5.0.0
 certifi                       2021.10.8
 cffi                          1.15.0
 cfgv                          3.3.1
 charset-normalizer            2.0.11
 check-manifest                0.47
+cli-ui                        0.16.0
 click                         8.0.3
 click-completion              0.5.2
 cloudpickle                   2.0.0
 CodeMetaPy                    0.3.5
 colorama                      0.4.4
+contextlib2                   21.6.0
 coverage                      6.3.1
 cryptography                  36.0.1
 cycler                        0.11.0
@@ -32,6 +33,7 @@
 defusedxml                    0.7.1
 distlib                       0.3.4
 dm-tree                       0.1.6
+docopt                        0.6.2
 docutils                      0.17.1
 entrypoints                   0.4
 executing                     0.8.2
@@ -135,7 +137,7 @@
 pygal                         3.0.0
 pygaljs                       1.0.2
 Pygments                      2.11.2
-pyhf                          0.6.4.dev83
+pyhf                          0.6.4.dev85
 pyparsing                     3.0.7
 pyrsistent                    0.18.1
 pytest                        7.0.0
@@ -157,6 +159,7 @@
 requests-toolbelt             0.9.1
 rfc3986                       2.0.0
 rsa                           4.8
+schema                        0.7.5
 scikit-hep-testdata           0.4.11
 scipy                         1.8.0
 SecretStorage                 3.3.1
@@ -180,6 +183,8 @@
 sphinxcontrib-qthelp          1.0.3
 sphinxcontrib-serializinghtml 1.1.5
 stack-data                    0.1.4
+tabulate                      0.8.9
+tbump                         6.7.0
 tenacity                      8.0.1
 tensorboard                   2.8.0
 tensorboard-data-server       0.6.1
@@ -194,12 +199,14 @@
 tf-estimator-nightly          2.8.0.dev2021122109
 toml                          0.10.2
 tomli                         2.0.0
+tomlkit                       0.9.1
 torch                         1.10.2
 tornado                       6.1
 tqdm                          4.62.3
 traitlets                     5.1.1
 twine                         3.8.0
 typing_extensions             4.0.1
+Unidecode                     1.3.2
 uproot                        4.1.9
 urllib3                       1.26.8
 virtualenv                    20.13.1

matthewfeickert · 2022-02-25T01:27:04Z

This should also be able to address:

Issue Guard SCHEMA_VERSION from version bumps #1223 (c.f. https://github.com/dmerejkowsky/tbump#restricting-the-lines-that-are-replaced)

In tests this should be sufficient to address Issue #1223

[[file]]
src = "src/pyhf/utils.py"
# This search is just identifying the line to restrict the
# regex to, but all matches in the line will get bumped
search = "pyhf: v{current_version}"

matthewfeickert · 2022-02-28T05:51:11Z

@lukasheinrich @kratsg On my fork I now have a GitHub Actions workflow dispatch workflow setup that allows for:

bumping versions, creating tags, pushing the tags back to GitHub
release candidates as viable options
automated checks that the type of bump you said you wanted (major, minor, patch) and if it was or wan't a release candidate makes sense given the latest tags available on GitHub and the new tag number you entered
dry runs to test that your options all look good

I've checked it both with intentionally failing and passing workflows, and have used it to create a series of tags and releases with release candidates for patch, minor, and major releases on my fork:

so if this sounds good I'll PR this after we get PR #1790 in.

kratsg · 2022-02-28T16:21:37Z

Probably still need a summary on the tag itself to annotate it. Release is a github functionality, but tags are a git functionality.

matthewfeickert · 2022-02-28T21:23:06Z

Probably still need a summary on the tag itself to annotate it. Release is a github functionality, but tags are a git functionality.

Yeah, I see what you're saying to be able to keep the annotations of the history in the Git tag. We can probably just port most of what you have in

pyhf/.github/workflows/tag.yml

Lines 76 to 87 in 9fd99be

    
                   NEW_TAG=$(git describe --tags --abbrev=0) 
        
                   echo "NEW_TAG=${NEW_TAG}" >> $GITHUB_ENV 
        
                   git tag -n99 -l $NEW_TAG 
        
                   CHANGES=$(git log --pretty=format:'%s' $OLD_TAG..HEAD -i -E --grep='^([a-z]*?):') 
        
                   CHANGES_NEWLINE="$(echo "${CHANGES}" | sed -e 's/^/  - /')" 
        
                   SANITIZED_CHANGES=$(echo "${CHANGES}" | sed -e 's/^/<li>/' -e 's|$|</li>|' -e 's/(#[0-9]\+)//' -e 's/"/'"'"'/g') 
        
                   echo "CHANGES=${SANITIZED_CHANGES//$'\n'/}" >> $GITHUB_ENV 
        
                   NUM_CHANGES=$(echo -n "$CHANGES" | grep -c '^') 
        
                   echo "NUM_CHANGES=${NUM_CHANGES}" >> $GITHUB_ENV 
        
                   git tag $NEW_TAG $NEW_TAG^{} -f -m "$(printf "This is a $BV_PART release from $OLD_TAG → $NEW_TAG.\n\nChanges:\n$CHANGES_NEWLINE")" 
        
                   git tag -n99 -l $NEW_TAG

over to augment the current tbump behavior.

matthewfeickert · 2022-03-01T08:47:06Z

@kratsg Can you take a look at the following tags on my fork:

and let me know how you feel about the annotated tag as well as the reporting of them (especially the v0.7.0rc1 → v0.7.0rc2 tag)?

matthewfeickert added feat/enhancement New feature or request deployment Deploy builds to distribution sites labels Feb 2, 2022

matthewfeickert self-assigned this Feb 2, 2022

matthewfeickert mentioned this issue Feb 25, 2022

feat: Use tbump over bump2version #1790

Merged

4 tasks

matthewfeickert closed this as completed in #1790 Mar 2, 2022

matthewfeickert mentioned this issue Mar 2, 2022

Replace tag workflow with tbump based bump version workflow #1797

Open

matthewfeickert mentioned this issue Apr 9, 2022

ci: Add bump version workflow for release tags #1846

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate from bump2version to tbump #1756

Migrate from bump2version to tbump #1756

matthewfeickert commented Feb 2, 2022 •

edited

Loading

matthewfeickert commented Feb 2, 2022

kratsg commented Feb 4, 2022

matthewfeickert commented Feb 4, 2022

matthewfeickert commented Feb 8, 2022

matthewfeickert commented Feb 25, 2022

matthewfeickert commented Feb 28, 2022

kratsg commented Feb 28, 2022

matthewfeickert commented Feb 28, 2022

matthewfeickert commented Mar 1, 2022

Migrate from bump2version to tbump #1756

Migrate from bump2version to tbump #1756

Comments

matthewfeickert commented Feb 2, 2022 • edited Loading

Summary

Additional Information

Example

Related Issues

Notes on <major>.<minor>.<patch>rc<candidate> format

Notes of caution

Code of Conduct

matthewfeickert commented Feb 2, 2022

kratsg commented Feb 4, 2022

matthewfeickert commented Feb 4, 2022

matthewfeickert commented Feb 8, 2022

matthewfeickert commented Feb 25, 2022

matthewfeickert commented Feb 28, 2022

kratsg commented Feb 28, 2022

matthewfeickert commented Feb 28, 2022

matthewfeickert commented Mar 1, 2022

matthewfeickert commented Feb 2, 2022 •

edited

Loading

Notes on `<major>.<minor>.<patch>rc<candidate>` format