Add update-rules command to update publishing-bot rules #253

navidshaikh · 2021-04-05T05:17:07Z

Description:

Add cmd/update-rules to update publishing-bot rules
Required flags: --rules, --branch
Exports the updated rules to file pointed by -o otherwise on prints stdout
Pin go version provided by --go flag else leave the field empty (omitempty)
Refer master branch rule per destination repo to add/update rule for provided branch

Usage:

Usage:  update-rules --branch BRANCH --rules PATH [--go VERSION | -o PATH]

Examples:

Update rules for branch release-1.21 with go version 1.16.4

  update-rules -branch release-1.21 -go 1.16.4 -rules /go/src/k8s.io/kubernetes/staging/publishing/rules.yaml

Update rules using URL to input rules file

update-rules -branch release-1.21 -go 1.16.4 -rules https://raw.githubusercontent.com/kubernetes/kubernetes/master/staging/publishing/rules.yaml

Update rules and export to /tmp/rules.yaml

  update-rules -branch release-1.22 -go 1.17.1 -o /tmp/rules.yaml -rules /go/src/k8s.io/kubernetes/staging/publishing/rules.yaml

fixes #252

TODO:

command
tests
docs

nikhita

@navidshaikh thanks for working on this, left some suggestions :)

cmd/publishing-bot/config/rules.go

cmd/update-rules/main.go

nikhita · 2021-04-05T09:48:14Z

cmd/update-rules/main.go

+		glog.Fatalf("update failed, found invalid rules after update: %v", err)
+	}
+
+	data, err := yaml.Marshal(rules)


This removes comments...maybe consider using go-yaml/yaml to preserve comments (go-yaml/yaml#132)

The comments are removed while the yaml content are deserialized into native ds. So at this point we don't have comments at all. To preserve comments, we'd need to load them into yaml.Node{} and navigate using it (we may also need to update the way rules loading and validations done in cmd/publishing-bot/config/rules.go). It's available in yaml.v3, currently we are using yaml.v2 here.

navidshaikh · 2021-04-06T19:03:31Z

Updated PR to print the updated rules on stdout if no -o flag is specified, this allows providing input rules by URL (using --rules flag).

puerco · 2021-05-13T17:51:05Z

/cc

navidshaikh · 2021-05-31T16:00:54Z

/retitle Add update-rules command to update publishing-bot rules

navidshaikh · 2021-06-01T16:35:51Z

Rebased and squashed the commits.

navidshaikh · 2021-06-15T18:02:11Z

/cc @nikhita @cpanato

this is ready for review, PTAL at PR desc for usage/examples

palnabarun · 2021-07-29T10:29:49Z

I had tried to generate the rules in kubernetes/kubernetes#103844 using the code here.

Some general observations:

The comments in rules.yaml get removed after running this. I think this is a blocker to use this tool.
The order of the keys in dictionaries gets changed. This is not a blocker as the key reordering happens only once. Post the initial change, the diff would be lower as the keys would be sorted in a previous rule update.
We should have a Makefile with a recipe to install the updates-rules binary. This is also good to have as other parties can install the tool consistently.

k8s-triage-robot · 2021-10-27T10:51:58Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

xmudrii · 2021-11-10T10:06:19Z

@navidshaikh @nikhita @palnabarun Is there any update on this PR? It would be nice to try to get this done -- it's definitely a very useful tool.

/remove-lifecycle stale

linux-foundation-easycla · 2021-12-13T17:44:53Z

The committers are authorized under a signed CLA.

✅ Navid Shaikh (38a5fd8, c8155b4, 8775f95, f06bd2b)

navidshaikh · 2021-12-13T18:01:47Z

Is there any update on this PR? It would be nice to try to get this done -- it's definitely a very useful tool.

@xmudrii sorry for late reply! will work through the feedback now.

@palnabarun : thanks for the review!

The comments in rules.yaml get removed after running this. I think this is a blocker to use this tool.

This tool uses the existing utils (using yaml.v2) to load the rules. To retain the comments I think we'll need yaml.v3.

Question: Should we have alternate yaml loading utils (with yaml.v3) and use it with this tool to retain the comments?
or move the existing utils to yaml.v3?

We should have a Makefile with a recipe to install the updates-rules binary. This is also good to have as other parties can install the tool consistently.

👍

`make build` builds the CLI binary `make build-image` includes the CLI binary in the image Signed-off-by: Navid Shaikh <[email protected]>

Signed-off-by: Navid Shaikh <[email protected]>

justaugustus

Echoing from Slack:

@dims @sttts -- Do we feel this is in a place that we could merge/iterate?
I feel like we could have @kubernetes/release-engineering start kicking the tires and cleaning up as needed.

justaugustus · 2021-12-13T19:35:31Z

/assign @dims @sttts

justaugustus · 2021-12-13T19:36:39Z

/assign @nikhita
ref: Slack

palnabarun · 2021-12-14T04:37:12Z

Question: Should we have alternate yaml loading utils (with yaml.v3) and use it with this tool to retain the comments?
or move the existing utils to yaml.v3?

Parsing YAML comments is still non-trivial and I feel specifically for this feature, the benefits do not outweigh the effort needed.

Looking back at the usage of comments in the latest rules.yaml:

# TODO: remove bazel related files after we stop publishing branches with
# bazel files
# See: https://github.com/kubernetes/enhancements/issues/2420

The above blob of comments can be in a GH issue instead of residing in a comment.

With that, we may not need to parse comments at all and comment parsing becomes a non-blocker.

Thanks @navidshaikh for working on this! ❤️

/lgtm

palnabarun · 2021-12-14T04:37:24Z

/hold for other reviews

navidshaikh · 2021-12-14T04:57:31Z

The above blob of comments can be in a GH issue instead of residing in a comment.

👍
There is only one comment in the rules file related to the KEP-2420: Reducing Kubernetes Build Maintenance. We can track updating the rules file part of the KEP tracking issue.

cc: @BenTheElder WDYT?

justaugustus · 2022-01-21T15:48:47Z

@kubernetes/publishing-bot-maintainers -- Where did we land with this one?

dims · 2022-02-11T13:51:21Z

/hold cancel
/approve

@palnabarun @justaugustus what are the follow ups after this merges?

k8s-ci-robot · 2022-02-11T13:51:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, justaugustus, navidshaikh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 5, 2021

k8s-ci-robot requested review from mfojtik and nikhita April 5, 2021 05:17

navidshaikh force-pushed the pr/rules-robot branch from 4ea2dc7 to a4b35dc Compare April 5, 2021 05:20

nikhita reviewed Apr 5, 2021

View reviewed changes

k8s-ci-robot requested a review from puerco May 13, 2021 17:51

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 31, 2021

navidshaikh requested a review from nikhita May 31, 2021 16:00

k8s-ci-robot changed the title ~~WIP: Add update-rules command to update publishing-bot rules~~ Add update-rules command to update publishing-bot rules May 31, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 31, 2021

navidshaikh force-pushed the pr/rules-robot branch from fb5474f to 2f47d35 Compare June 1, 2021 16:34

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 9, 2021

navidshaikh force-pushed the pr/rules-robot branch from 2f47d35 to 074abe7 Compare June 9, 2021 16:40

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 9, 2021

k8s-ci-robot requested a review from cpanato June 15, 2021 18:02

puerco mentioned this pull request Aug 5, 2021

1.22 updates to Branch Manager handbook kubernetes/sig-release#1649

Open

5 tasks

xmudrii mentioned this pull request Sep 26, 2021

Create a tool to update publishing-bot rules for new releases #252

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 27, 2021

xmudrii mentioned this pull request Nov 10, 2021

Add updating publishing-bot rules to the Branch Management handbook and Release issue template kubernetes/sig-release#1504

Merged

navidshaikh added 2 commits December 13, 2021 23:14

Add README

8775f95

Update usage and README

f06bd2b

navidshaikh force-pushed the pr/rules-robot branch from cc27ebf to f06bd2b Compare December 13, 2021 17:44

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 13, 2021

navidshaikh added 3 commits December 14, 2021 00:24

Update Makefile to include update-rules binary

3b7bdde

`make build` builds the CLI binary `make build-image` includes the CLI binary in the image Signed-off-by: Navid Shaikh <[email protected]>

Update README to include build info

9145030

Signed-off-by: Navid Shaikh <[email protected]>

Add help hint in the error message

2c5c00a

Signed-off-by: Navid Shaikh <[email protected]>

justaugustus approved these changes Dec 13, 2021

View reviewed changes

k8s-ci-robot assigned justaugustus Dec 13, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 13, 2021

k8s-ci-robot assigned dims and sttts Dec 13, 2021

k8s-ci-robot assigned nikhita Dec 13, 2021

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 14, 2021

k8s-ci-robot assigned palnabarun Dec 14, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 11, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 11, 2022

k8s-ci-robot merged commit 29e3523 into kubernetes:master Feb 11, 2022

navidshaikh deleted the pr/rules-robot branch February 17, 2022 08:00

palnabarun mentioned this pull request Apr 20, 2022

[release-1.24] Update publishing-bot rules kubernetes/kubernetes#109567

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add update-rules command to update publishing-bot rules #253

Add update-rules command to update publishing-bot rules #253

navidshaikh commented Apr 5, 2021 •

edited

Loading

nikhita left a comment

nikhita Apr 5, 2021

navidshaikh Apr 6, 2021 •

edited

Loading

navidshaikh commented Apr 6, 2021

puerco commented May 13, 2021

navidshaikh commented May 31, 2021

navidshaikh commented Jun 1, 2021

navidshaikh commented Jun 15, 2021

palnabarun commented Jul 29, 2021

k8s-triage-robot commented Oct 27, 2021

xmudrii commented Nov 10, 2021

linux-foundation-easycla bot commented Dec 13, 2021 •

edited

Loading

navidshaikh commented Dec 13, 2021

justaugustus left a comment

justaugustus commented Dec 13, 2021

justaugustus commented Dec 13, 2021

palnabarun commented Dec 14, 2021

palnabarun commented Dec 14, 2021

navidshaikh commented Dec 14, 2021

justaugustus commented Jan 21, 2022

dims commented Feb 11, 2022

k8s-ci-robot commented Feb 11, 2022

Add update-rules command to update publishing-bot rules #253

Add update-rules command to update publishing-bot rules #253

Conversation

navidshaikh commented Apr 5, 2021 • edited Loading

Description:

Usage:

Examples:

nikhita left a comment

Choose a reason for hiding this comment

nikhita Apr 5, 2021

Choose a reason for hiding this comment

navidshaikh Apr 6, 2021 • edited Loading

Choose a reason for hiding this comment

navidshaikh commented Apr 6, 2021

puerco commented May 13, 2021

navidshaikh commented May 31, 2021

navidshaikh commented Jun 1, 2021

navidshaikh commented Jun 15, 2021

palnabarun commented Jul 29, 2021

k8s-triage-robot commented Oct 27, 2021

xmudrii commented Nov 10, 2021

linux-foundation-easycla bot commented Dec 13, 2021 • edited Loading

navidshaikh commented Dec 13, 2021

justaugustus left a comment

Choose a reason for hiding this comment

justaugustus commented Dec 13, 2021

justaugustus commented Dec 13, 2021

palnabarun commented Dec 14, 2021

palnabarun commented Dec 14, 2021

navidshaikh commented Dec 14, 2021

justaugustus commented Jan 21, 2022

dims commented Feb 11, 2022

k8s-ci-robot commented Feb 11, 2022

navidshaikh commented Apr 5, 2021 •

edited

Loading

navidshaikh Apr 6, 2021 •

edited

Loading

linux-foundation-easycla bot commented Dec 13, 2021 •

edited

Loading