Update Branch Manager handbook instructions for branch cut activities #757

justaugustus · 2019-08-14T07:43:10Z

Updated documentation based on experiences from the 1.16.0-beta.0 release.

Signed-off-by: Stephen Augustus [email protected]

/assign @tpepper @hoegaarden @nikhita @imkin @Katharine @spiffxp
cc: @kubernetes/release-engineering
/milestone v1.16
/priority important-soon
/kind documentation

justaugustus · 2019-08-14T07:43:27Z

/assign @idealhack

hoegaarden · 2019-08-14T09:59:32Z

/lgtm
/hold
... in case someone else want's to review, esp. regarding the test-infra stuff I have near-zero context on

justaugustus · 2019-08-14T11:01:32Z

Added a note about configuring the milestoneapplier plugin.
Example: kubernetes/test-infra#13888

nikhita

As someone new to the process, I feel like it would be easier if the alpha/beta releases sections would be converted to use checklists and/or bullets but that can be done in a follow up. Thanks for the overhaul, @justaugustus! :)

nikhita · 2019-08-14T12:31:50Z

release-engineering/role-handbooks/branch-manager.md

+   - Remove the oldest release variant e.g., `'1.12'`
+   - Add an entry for the newest release variant e.g., `'1.16'`
+   - Ensure the following:
+     - The `K8S_RELEASE` marker for `experimental` matches `master`


Can we also mention that this needs to be stable or is that still under discussion? If the latter, should we create an issue to track this?

And output the issue's outcome into https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md

release-engineering/role-handbooks/branch-manager.md

nikhita · 2019-08-14T12:39:29Z

release-engineering/role-handbooks/branch-manager.md

+   - Remove [Branch Managers](/release-managers.md#branch-managers) from any previous release branches
+   - Add all current [Branch Managers](/release-managers.md#branch-managers)
+2. [Create CI/Presubmit jobs for the new release](#create-cipresubmit-jobs)
+3. [Run `./branchff` approximately a day after the branch has been created]((#branch-fast-forward))


IIUC anyone who'll need to run branchff needs to be in the release-managers GitHub team? Can we also mention that in the Onboarding section?

We mention this in the beginning of the Branch Content Management section.
I'm planning to refactor this doc a bunch and have more guidance on Release Engineering onboarding in the repo, but I'm going to punt on that for this PR.

nikhita · 2019-08-14T12:41:19Z

release-engineering/role-handbooks/branch-manager.md


-`./gcbmgr stage release-1.12 --build-at-head --nomock`
+`prow`’s [`branchprotector`](https://git.k8s.io/test-infra/prow/cmd/branchprotector/README.md) runs once per day and automatically adds [branch protection](https://help.github.com/articles/about-protected-branches/) to any new branch in the `k/k` repo.


nit: would be better to explicitly spell out k/k to kubernetes/kubernetes since newcomers also read the handbook :)

nikhita · 2019-08-14T12:42:29Z

release-engineering/role-handbooks/branch-manager.md

+
+   `grep` for [`manual-release-bump-required`](https://github.com/kubernetes/test-infra/search?q=manual-release-bump-required&unscoped_q=manual-release-bump-required) in [test-infra](https://github.com/kubernetes/test-infra) and duplicate block entries appropriately
+
+7. Update release dashboards in [`testgrid/config.yaml`](https://github.com/kubernetes/test-infra/blob/master/testgrid/config.yaml) ([example commit](https://github.com/kubernetes/test-infra/pull/13882/commits/8aad58b09ae718139b44c70512d2d133d094ed82))


config.yaml link does not exist

nikhita · 2019-08-14T12:44:24Z

release-engineering/role-handbooks/branch-manager.md

@@ -292,77 +310,143 @@ The following are some ways to determine if the release process was successful:

 1. The build tag and release artifacts become [visible on GitHub at https://github.com/kubernetes/kubernetes/releases](https://github.com/kubernetes/kubernetes/releases)

-1. The release is logged automatically by [k8s-release-robot](https://github.com/k8s-release-robot) in [sig-release](https://github.com/k8s-release-robot/sig-release)
+2. The release is logged automatically by [k8s-release-robot](https://github.com/k8s-release-robot) in [sig-release](https://github.com/k8s-release-robot/sig-release)


Bad link for sig-release .. 😬

tpepper

More or less lgtm, though I had to spend a loooot of time finding your meaningful changes versus the mechanical version number increments and markdown styling changes because this is one giant commit.

One question though: this is a lot of changing of branch creation and actions a human needs to run to manage the test-infra changes for a new branch. It was previously asserted this was all automated, and we removed the test-infra role from the release team for that reason. For all of this manual activity, what portions of it can be automated into a push button update for a new branch?

tpepper · 2019-08-14T17:50:52Z

release-engineering/role-handbooks/branch-manager.md

+- [Staging Repositories](#staging-repositories)
+- [Debugging](#debugging)
+- [References](#references)
+- [Background information](#background-information)


It would make review a lot easier if the ToC style update was separate from content modifications. It looks like this is just Cap's to Lower Case? But I need to manually compare across 60 lines looking if a section was added or removed.

This comes up in our reviews without fail and I keep forgetting to address it.
I use Visual Studio Code with two neat extensions:

Markdown All in One

markdownlint

The combination of the two enable linting for Markdown files and table of contents generation.
ToC generation happens on save, so I have limited control there.

I'm willing to give up some control there for the sake of the table of contents always being up to date.

Does VSCode support "git add -i"?

release-engineering/role-handbooks/branch-manager.md

tpepper · 2019-08-14T17:52:59Z

release-engineering/role-handbooks/branch-manager.md


-The first time a specific branch is mentioned for the release process instead of the `master` branch is when beta.0 is being cut, for example:
+Behind the scenes `anago` is doing a branch create and push.


"...doing a git branch create and git push..."

tpepper · 2019-08-14T17:55:22Z

release-engineering/role-handbooks/branch-manager.md


 This being the first build from the newly created release branch, the publication of this build does not send out the typical changelog detail notification, but rather will only send a shorter message with subject line "[k8s release-1.15 branch has been created](https://groups.google.com/forum/#!topic/kubernetes-announce/Gu2JYa4BzAg)".

 To publish (release) the build artifacts from staging beta for example, run:

-`./gcbmgr release release-1.15 --buildversion=v1.15.0-alpha.3.N+commit-hash --nomock`
+`./gcbmgr release release-1.16 --buildversion=v1.16.0-alpha.3.N+commit-hash --nomock`


All of the s/1.15/1.16/ and s/1.16/1.17/ changes would be nice to have as a separate commit. Again that portion is trivially reviewed and merits little discussion, but results in a lot of diffstat which distracts from discussion worthy changes.

I pretty much only touch versions when I've passed over a section... as if to say, "Someone has looked over this recently-ish.". I don't necessarily want to do wholesale search/replaces and have them in a separate commit if I haven't personally reviewed the section.

I'm fine saying solving this is out of scope for this PR, but I agree re: increased review burden. There are also a lot of x.y references in this doc, and it'd be cool to just pick one up top and stick with it. eg: "for the purposes of this doc, the last release is 1.1, the in-development release is 1.2, and the not-yet-started release is 1.3" (eg: https://github.com/kubernetes/community/blob/b7f39f6935264984e24fe8c27c0fd8e4fffac364/contributors/devel/sig-architecture/conformance-tests.md#conformance-test-version-skew-policy)

tpepper · 2019-08-14T17:59:48Z

release-engineering/role-handbooks/branch-manager.md

+## Post-release Activities
+
+- Set the `K8S_RELEASE` marker for the current release variant to `stable-x.y` in the [`variants.yaml` file for `kubekins-e2e`](https://github.com/kubernetes/test-infra/blob/fa43d4a7a6c88c0dedd0db83b250cec485b60736/images/kubekins-e2e/variants.yaml). ([reference PR review comment](https://github.com/kubernetes/test-infra/pull/13870#discussion_r313628808))
+


Should that PR discussion get reflected in https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md plus a description of K8s_RELEASE?

And then this reference to the K8S_RELEASE marker can reference that documentation?

(ie: there seemed to be a fair amount of confusion or uncertainty on what the marker's intent is...that decided should be in official docs for reference as in 3/6/9/12 months we'll be wondering again and again and we'll have fresh issue and PR discussions to reference)

👍 Explicitly documenting post-release steps here.

I would recommend adding something about making sure the "beta" 'channel' goes back to pointing to master / being unused post-release (ref: #696 (comment))

re: K8S_RELEASE, I'm not thrilled when comments in PRs are linked as reference material, but I think it's out of scope to refine here. I would prefer to see reference docs on that variable and others in a README.md for kubekins, so we can understand what is in that image and why. (ref: kubernetes/test-infra#13129 (comment))

tpepper · 2019-08-14T18:03:39Z

release-engineering/role-handbooks/branch-manager.md

+
+3. Update [`images/kubekins-e2e/variants.yaml`](https://github.com/kubernetes/test-infra/blob/master/images/kubekins-e2e/variants.yaml). ([example PR](https://github.com/kubernetes/test-infra/pull/13876))
+
+   (The `variants.yaml` allows us to target various branches or branch combinations during CI tests. The `K8S_VERSION` variable maps to the version marker files viewable in the [`kubernetes-release` GCS bucket](https://gcsweb.k8s.io/gcs/kubernetes-release/release/) e.g., [`latest-1.16.txt`](https://storage.googleapis.com/kubernetes-release/release/latest-1.16.txt))


This is good, but it's now a great summary buried where only a subset of relevant folks will see it. IMO should be in https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md

ok so for example I would dump most of this content around kubekins into a README.md here https://github.com/kubernetes/test-infra/tree/master/images/kubekins-e2e, e.g. the constraints to adhere to, why things are what they are, etc.

the parts that I would leave here are the instructions directly related to "add new variant", "drop old variant"

tpepper · 2019-08-14T18:04:05Z

release-engineering/role-handbooks/branch-manager.md

+   - Remove the oldest release variant e.g., `'1.12'`
+   - Add an entry for the newest release variant e.g., `'1.16'`
+   - Ensure the following:
+     - The `K8S_RELEASE` marker for `experimental` matches `master`


And output the issue's outcome into https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md

release-engineering/role-handbooks/branch-manager.md

justaugustus · 2019-08-14T22:13:52Z

Thanks for the reviews, @hoegaarden, @nikhita, and @tpepper!

As someone new to the process, I feel like it would be easier if the alpha/beta releases sections would be converted to use checklists and/or bullets but that can be done in a follow up.

That's the plan! This is just a little update to capture some of the branch creation/CI tasks that needed updating.

One question though: this is a lot of changing of branch creation and actions a human needs to run to manage the test-infra changes for a new branch. It was previously asserted this was all automated, and we removed the test-infra role from the release team for that reason. For all of this manual activity, what portions of it can be automated into a push button update for a new branch?

The test-infra changes are great improvements, but we are not fully automated yet, just enough so to have removed the role (I think @Katharine has mentioned this on a few calls).
Some things that come to mind are here and are already captured in issues/PRs linked out in that comment.

This is ready for review again.
I'm going to punt on the version marker files for this PR and handle in the near term as it's going to take a bit of investigation.
I've opened an issue to tie the threads together: #759

spiffxp

Some suggestions

spiffxp · 2019-08-14T22:25:23Z

release-engineering/role-handbooks/branch-manager.md

+  - [Configure Merge Automation](#configure-merge-automation)
+  - [Tide](#tide)
+  - [Code Freeze](#code-freeze)
+  - [Code Thaw](#code-thaw)


these three bullets lost their indent under "Configure Merge Automation"

spiffxp · 2019-08-14T22:29:19Z

release-engineering/role-handbooks/branch-manager.md

-1. Update this handbook where appropriate for the next release cycle.
-1. Have willingness to accommodate with different timezone esp. for the release team and your Associates.
-1. Participate in conversations that happen on [#sig-release] and [#release-management]
+2. You must allocate time to setup your system to run the release tools; eventually you'll have a routine going and become more familiar with the process.


general comment on renumbering ordered lists: this obscures what's actually changing, and makes it more toilsome to update (even if you shed that toil to a plugin)

we don't require one way or the other in our markdown style guide (https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md#lists), but it would ease review burden if you left them at 1.

release-engineering/role-handbooks/branch-manager.md

spiffxp · 2019-08-14T23:04:59Z

release-engineering/role-handbooks/branch-manager.md

+
+   `grep` for [`manual-release-bump-required`](https://github.com/kubernetes/test-infra/search?q=manual-release-bump-required&unscoped_q=manual-release-bump-required) in [test-infra](https://github.com/kubernetes/test-infra) and duplicate block entries appropriately
+
+7. Update release dashboards in the [Testgrid config](https://git.k8s.io/test-infra/config/testgrids/config.yaml) ([example commit](https://github.com/kubernetes/test-infra/pull/13882/commits/8aad58b09ae718139b44c70512d2d133d094ed82))


an idea for followup: shard out to config/testgrids/sig-release/config.yaml instead (ref: kubernetes/test-infra#13541)

release-engineering/role-handbooks/branch-manager.md

spiffxp · 2019-08-14T23:17:33Z

release-engineering/role-handbooks/branch-manager.md


 This being the first build from the newly created release branch, the publication of this build does not send out the typical changelog detail notification, but rather will only send a shorter message with subject line "[k8s release-1.15 branch has been created](https://groups.google.com/forum/#!topic/kubernetes-announce/Gu2JYa4BzAg)".

 To publish (release) the build artifacts from staging beta for example, run:

-`./gcbmgr release release-1.15 --buildversion=v1.15.0-alpha.3.N+commit-hash --nomock`
+`./gcbmgr release release-1.16 --buildversion=v1.16.0-alpha.3.N+commit-hash --nomock`


I'm fine saying solving this is out of scope for this PR, but I agree re: increased review burden. There are also a lot of x.y references in this doc, and it'd be cool to just pick one up top and stick with it. eg: "for the purposes of this doc, the last release is 1.1, the in-development release is 1.2, and the not-yet-started release is 1.3" (eg: https://github.com/kubernetes/community/blob/b7f39f6935264984e24fe8c27c0fd8e4fffac364/contributors/devel/sig-architecture/conformance-tests.md#conformance-test-version-skew-policy)

release-engineering/role-handbooks/branch-manager.md

spiffxp · 2019-08-14T23:38:01Z

You are free to do in a followup, but while you are here it would be nice to have a tombstone dropped in release-team to point to this (ref: #761)

Signed-off-by: Stephen Augustus <[email protected]>

- Clarify k/k as kubernetes/kubernetes for newer contributors - Fix link to Testgrid config - Fix link to k/sig-release - Clarify anago is doing git branch and git push during beta release - Update milestoneapplier config to include the `v` in version - Add a note that milestoneapplier should be bumped after code freeze Signed-off-by: Stephen Augustus <[email protected]> Co-Authored-By: Aaron Crickenberger <[email protected]>

idealhack

Mostly LGTM, I'd go for merge and iterate.

/lgtm

k8s-ci-robot · 2019-09-03T16:50:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hoegaarden, idealhack, justaugustus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~release-engineering/OWNERS~~ [hoegaarden,justaugustus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

justaugustus · 2019-09-04T03:29:33Z

@idealhack -- good plan. I've added an action item on #696 to incorporate the remaining feedback.

/hold cancel

k8s-ci-robot assigned hoegaarden and imkin Aug 14, 2019

k8s-ci-robot added this to the v1.16 milestone Aug 14, 2019

k8s-ci-robot assigned Katharine and nikhita Aug 14, 2019

k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Aug 14, 2019

k8s-ci-robot assigned spiffxp and tpepper Aug 14, 2019

k8s-ci-robot assigned idealhack Aug 14, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 14, 2019

k8s-ci-robot requested review from Bubblemelon and listx August 14, 2019 07:43

justaugustus force-pushed the branchmgmt branch from b3d29fe to dd4e31f Compare August 14, 2019 07:50

justaugustus mentioned this pull request Aug 14, 2019

Cut 1.16.0-beta.0 release #756

Closed

5 tasks

justaugustus force-pushed the branchmgmt branch from dd4e31f to da08713 Compare August 14, 2019 08:10

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Aug 14, 2019

justaugustus force-pushed the branchmgmt branch from da08713 to 129e2f1 Compare August 14, 2019 10:58

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 14, 2019

nikhita reviewed Aug 14, 2019

View reviewed changes

justaugustus mentioned this pull request Aug 14, 2019

sig-release: Add justaugustus to Release Managers team kubernetes/org#1101

Merged

tpepper reviewed Aug 14, 2019

View reviewed changes

justaugustus mentioned this pull request Aug 14, 2019

Document current usage of kubernetes version marker files #759

Closed

justaugustus force-pushed the branchmgmt branch from cbbe8cc to 0f84930 Compare August 14, 2019 22:14

spiffxp reviewed Aug 14, 2019

View reviewed changes

justaugustus mentioned this pull request Aug 16, 2019

Remove requirements for /usr/local in anago and branchff kubernetes/release#847

Merged

justaugustus and others added 2 commits August 19, 2019 08:04

Update Branch Manager handbook instructions for branch cut activities

57d6994

Signed-off-by: Stephen Augustus <[email protected]>

justaugustus force-pushed the branchmgmt branch from 4d1d71a to 8daa15a Compare August 19, 2019 12:06

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 3, 2019

idealhack approved these changes Sep 3, 2019

View reviewed changes

Bubblemelon mentioned this pull request Sep 4, 2019

Branch Management Improvements List #696

Closed

18 tasks

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 4, 2019

k8s-ci-robot merged commit 7187fa0 into kubernetes:master Sep 4, 2019

justaugustus mentioned this pull request Dec 6, 2019

branchmgmt: Address previous handbook review comments #891

Merged


		`./gcbmgr stage release-1.12 --build-at-head --nomock`
		`prow`’s [`branchprotector`](https://git.k8s.io/test-infra/prow/cmd/branchprotector/README.md) runs once per day and automatically adds [branch protection](https://help.github.com/articles/about-protected-branches/) to any new branch in the `k/k` repo.


		`grep` for [`manual-release-bump-required`](https://github.com/kubernetes/test-infra/search?q=manual-release-bump-required&unscoped_q=manual-release-bump-required) in [test-infra](https://github.com/kubernetes/test-infra) and duplicate block entries appropriately

		7. Update release dashboards in [`testgrid/config.yaml`](https://github.com/kubernetes/test-infra/blob/master/testgrid/config.yaml) ([example commit](https://github.com/kubernetes/test-infra/pull/13882/commits/8aad58b09ae718139b44c70512d2d133d094ed82))


		The first time a specific branch is mentioned for the release process instead of the `master` branch is when beta.0 is being cut, for example:
		Behind the scenes `anago` is doing a branch create and push.

		## Post-release Activities

		- Set the `K8S_RELEASE` marker for the current release variant to `stable-x.y` in the [`variants.yaml` file for `kubekins-e2e`](https://github.com/kubernetes/test-infra/blob/fa43d4a7a6c88c0dedd0db83b250cec485b60736/images/kubekins-e2e/variants.yaml). ([reference PR review comment](https://github.com/kubernetes/test-infra/pull/13870#discussion_r313628808))


		3. Update [`images/kubekins-e2e/variants.yaml`](https://github.com/kubernetes/test-infra/blob/master/images/kubekins-e2e/variants.yaml). ([example PR](https://github.com/kubernetes/test-infra/pull/13876))

		(The `variants.yaml` allows us to target various branches or branch combinations during CI tests. The `K8S_VERSION` variable maps to the version marker files viewable in the [`kubernetes-release` GCS bucket](https://gcsweb.k8s.io/gcs/kubernetes-release/release/) e.g., [`latest-1.16.txt`](https://storage.googleapis.com/kubernetes-release/release/latest-1.16.txt))


		`grep` for [`manual-release-bump-required`](https://github.com/kubernetes/test-infra/search?q=manual-release-bump-required&unscoped_q=manual-release-bump-required) in [test-infra](https://github.com/kubernetes/test-infra) and duplicate block entries appropriately

		7. Update release dashboards in the [Testgrid config](https://git.k8s.io/test-infra/config/testgrids/config.yaml) ([example commit](https://github.com/kubernetes/test-infra/pull/13882/commits/8aad58b09ae718139b44c70512d2d133d094ed82))

Update Branch Manager handbook instructions for branch cut activities #757

Update Branch Manager handbook instructions for branch cut activities #757

Conversation

justaugustus commented Aug 14, 2019

justaugustus commented Aug 14, 2019

hoegaarden commented Aug 14, 2019

justaugustus commented Aug 14, 2019 • edited Loading

nikhita left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tpepper left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justaugustus commented Aug 14, 2019

spiffxp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiffxp commented Aug 14, 2019

idealhack left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Sep 3, 2019

justaugustus commented Sep 4, 2019

justaugustus commented Aug 14, 2019 •

edited

Loading