Proposal to improve the CHANGELOG process

[djaglowski]

Motivation

The current changelog process causes many unnecessary code conflicts and mistakes. For example:

1. Two developers make unrelated bug fixes. Each adds a line at the end of the "Bug Fixes" section. Now there is a code conflict.
2. Someone opens a PR. Before the PR is merged, a new version of the collector is released. Now the PR is (wrongly) proposing to add a line to previous release.

These kinds of problems degrade developer experience, especially when combined with the somewhat flaky CI process that we have. I believe that we can put an end to unnecessary changelog conflicts, and therefore remove one of the top few causes of unnecessary PR delays.

Proposed Solution

In short, I suggest we generate the changelog from yaml files. There are several examples of this strategy being used elsewhere, such as described in this GitLab blog.

Our implementation could look like this:

1. Define a very simple yaml format that captures the details we want to see in a changelog entry.
2. Instead of adding a changelog entry to CHANGELOG.md, developers will add a yaml file to a ./changelog/ directory. The name of the file does not matter, so long as it does not conflict with any other unreleased change.
3. A simple tool validates and consumes these yaml files, and updates CHANGELOG.md as part of the release process.

I have written a proof of concept for this. There are some TODOs but I think it demonstrates the concept. I'll open a draft PR and link it to this issue. (Edit: see #11540)

Usage

Note that unreleased changes have been removed from CHANGELOG.md and converted to yaml files in ./changelog/.

> make chlog-validate
cd cmd/chloggen && go install .
chloggen validate
PASS: all files in changelog are valid

> make chlog-preview
cd cmd/chloggen && go install .
chloggen preview
Generated changelog updates:

## vTODO

## 🛑 Breaking changes 🛑

- `prometheusexporter`: Automatically rename metrics with units to follow Prometheus naming convention (#8950)

### 💡 Enhancements 💡

- `prometheusreceiver`: Add `target_info` labels to resource attributes (#11034)
- `saphanareceiver`: Fix component memory query, add better error handling (#11507)
- `tanzuobservabilityexporter`: remove calls to deprecated `NewProxySender` methods (#11510)

### 🧰 Bug fixes 🧰

- `aerospikereceiver`: Fix issue where namespaces would not be collected (#11465)

> make chlog-update 
cd cmd/chloggen && go install .
chloggen update
Finished updating CHANGELOG.md

> git status
git status                                                                                                                        ✗ ✭
On branch chloggen
Your branch is up to date with 'djaglowski/chloggen'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   CHANGELOG.md
        deleted:    changelog/aerospike-namespaces-fix.yaml
        deleted:    changelog/prom-rename-metrics.yaml
        deleted:    changelog/prom-target-info.yaml
        deleted:    changelog/saphana-fix-memory-query.yaml
        deleted:    changelog/tanzu-remove-deprecated.yaml

> git diff                                                                                                                          ✗ ✭
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 200001e691..16a86b4428 100644
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 200001e691..16a86b4428 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,22 @@
 
 <!-- next version -->
 
+## vTODO
+
+## 🛑 Breaking changes 🛑
+
+- `prometheusexporter`: Automatically rename metrics with units to follow Prometheus naming convention (#8950)
+
+### 💡 Enhancements 💡
+
+- `prometheusreceiver`: Add `target_info` labels to resource attributes (#11034)
+- `saphanareceiver`: Fix component memory query, add better error handling (#11507)
+- `tanzuobservabilityexporter`: remove calls to deprecated `NewProxySender` methods (#11510)
+
+### 🧰 Bug fixes 🧰
+
+- `aerospikereceiver`: Fix issue where namespaces would not be collected (#11465)
+
 ## v0.54.0

[codeboten]

I'm 100% in favour of finding a way to automate the changelog process as it's currently pretty painful. Curious about your thoughts on using a tool that already exist, maybe something like https://github.com/release-lab/whatchanged? I haven't spent a lot of time looking into tools, but a quick search makes me think this already exists.

[djaglowski]

Typically I would agree, but I think this problem is simple enough that depending on another tool may be a greater pain than rolling our own.

There's not much that another tool would solve for us here. I think we'd still be on the hook for writing and maintaining:

• A template file for the changelog
• CI checks
• Documentation for how we expect it to be used
• (If possible) a set of fields we want specified for each PR

If you feel strongly about not rolling our own, I can't blame you. That said, I believe I've already got a simple and fairly robust implementation of the generator part.

[djaglowski]

I believe #11540 is ready for review, pending agreement on the general approach. It contains all the features I expect we'd need, except for updates to the CI process which I have drafted but would submit in a followup PR.

[dmitryax]

I like the approach. It makes a lot of sense to me.

Couple more improvements that we can achieve with this approach:

• Sort the list of changes within each group.
• Aggregate changes made per specific component to avoid repeating them, e.g.:

- `component1`:
  - Change 1
  - Change 2

[djaglowski]

Sort the list changes within each group.

Yes, already doing this in the proposed implementation actually.

Aggregate changes made per specific component to avoid repeating them.

I like this idea. The challenge currently is that we aren't enumerating the components, so there will be some fragmentation due to minor variations in the way people specify the field. This is something that can be added later though.

[djaglowski]

Since this issue is the actual proposal, I want to call out another part of the process that I am suggesting here, which was included and discussed on the implementation PR (#11540).

In short, I suggested that we document issue #'s in the changelog, rather than PR #'s. This solves another painpoint in the process, where developers were required to open a PR, and push another commit to add the PR number in their changelog entry. By using issue #'s instead, developers can in theory open a PR that requires not additional changes. This also motivates everyone to actually open tracking issues before PRs.

[jpkrohling]

I love this idea and the implementation! Thank you for that, @djaglowski!

This looks complete to me, should this issue be closed?

[djaglowski]

Glad to hear it. I wanted to see it in action for a bit before we called it complete. Today I saw a couple PRs go through smoothly, so I think we can close this now.