go-ipfs biweekly releases

[whyrusleeping]

I would like to move towards releasing a new version of go-ipfs every other week.

Why?

Many reasons:

• Quicker iteration of ideas.
• More emphasis on a hardened product we feel comfortable shipping.
• Less last second 'oh before we ship this lets do...'
• Will help us get better at shipping things.
• Big releases scare me.

Two weeks seems really aggressive, are you sure?

Yes.

How are we going to pull this off?

Two weeks divides itself into roughly ten days. While we're getting into the
swing of this, the first week of the cycle will be for merging in new features
and changes, etc. The second week will be solely for adding tests, fixing
potential regressions, and prepping for release. Thursday of the second week
should be the day that the changelog PR and version bump are finalized so that
on friday we can merge them and go through the full release process (tag,
builds, dists, announce, etc)

Feature code can still be worked on during the 'release week', but not merged
until the next dev cycle. As we get better at this process, we can extend the
dev cycle into the second week to give us a higher margin of time to get things
done.

The hard part here is being able to be more sure than we are now that
everything we merge is good solid code. This is something we need to do
anyways, and moving towards an aggressive release cycle like this will push
us towards actually delivering on this.

Things this will force us to improve:

• faster CI
• better tests
• tests with every PR
• better integration tests
• better continuous deployment on gateways
• better dists infrastructure and workflow
  - nightly builds?

Things that will help us accomplish this:

• Nightly stress tests in CI
• More code coverage numbers
  - go kuba go
• go vet in CI
  - maybe also other things in gometalinter
• automated metrics
  - [ ] dedicated machine for running metrics tests
  - [ ] 'ipfs-whatever' improvements
  - time to run ipfs refs local
  - time to run ipfs repo gc with X things in repo
  - [ ] super basic dashboard to view metrics by version
• automated deploys to certain nodes
  - set up webhooks?
  - i'd like two of the gateway nodes to automatically start running master as soon as its ready
  - i'd also like to be able to easily set up my own nodes to build and run every new update
  - maybe make a docker image that is designed to be self updating?
• more gateway node metrics
  - [ ] correlate cpu time and memory usage with version
  - [ ] bandwidth usage
• bots to help remind us of the cycle
  - [ ] maybe once a day reminds us to give a small update on what we're working on
  - [ ] on week two, remind us that changelog and release stuff needs to be done in X days
  - [ ] on release day, send a reminder to everyone
• milestones for planning tasks for each release (with due dates)
• Add a 'Changelog' comment to each and every PR to make changelog generation easier
• PR status line to make sure there is a 'Changelog' comment on each PR
• Automate initial generation of changelog
  - Try to make this as hands-free as possible
• automate dists building process, dedicated machine/vm for this?
  - on every new tag, do './add-dist go-ipfs $TAG'
• ipfs infrastructure notification server (!!)
  - central place where all infra nodes can report things to
  - ex: 'mercury rebooting', 'new ipfs version building', 'pinbot pinning hash', 'mars crashed again'
  - 'new code pushed to master'
  - 'master being deployed to uranus'

Open questions:

• How can we improve our code review process?
  - code review request API from github will be very nice
  - set aside time every day for review, try to get to all reviews within 24 hours
• How can we make sure we arent wasting time stuck in 'process'?
  - clearly measure time spent on release process vs dev work.
  - good metric might be number of feature PRs merged per release
  - automate anything and everything we can
• What new infrastructure do we need for this? and who will maintain it?
• In general, how do we write and ship better quality code?

When will this start?

Ideally after the release of 0.4.5, But it might take a little bit longer to
get into the swing of this. This is definitely Q1 materiel. I'm gonna start
asking 'what do we need to do to release right now' to help us get into the
mindset of shrinking the release time.

[Kubuxu]

I love the idea.

The fact that we do long releases slows down the release process even more as we want to squeeze features and fixes into given release because we know if we miss the window it won't be out for a long time. Being able to say, "this bug is known, it affects people but it is too late for current window. Expect it to be fixed in 2 weeks." is very healthy and will result in fixes being delivered even faster as currently bugs that were fixed months ago are not "really" fixed as there was no release.

Probably next week the ipfs/infra#100 will be resolved and we can start setting up new pipeline that will make whole process more efficient. More than that we will be sure on what we stand, currently I hastate on improving current CI pipeline as I know it will be replaced.

I have finished Makefiles refactor in go-ipfs and with it comes sharness test coverage: ipfs/kubo#3504

This means that we have even better overview of what is going on, and CR should be easier. We should make sure to try increasing the coverage with every PR and not letting untested code to enter the repo.

• i'd like two of the gateway nodes to automatically start running master as soon as its ready

Are we sure that we want to do this? Failure on them affects gateway end users. Maybe it would be better to mirror the traffic from the gateways to the gateways under test and compare the results (HTTP codes at first, timings, usage stats later).

More comments for sure to come (it is just late for me).

[whyrusleeping]

Yeah, mirroring traffic works too, i've wanted to do A/B testing for quite some time.

[jbenet]

yeah i'd love to see this happen, thank you @whyrusleeping -- totally in support

[Kubuxu]

Bots to improve the cycle:

Milestone bot:

• when new version milestone is created, create issue from template and assign it to the milestone
• when milestone is being closed:
  • create new milestone with patch version incremented in relation with highest version milestone with the same major and minor versions - at first it might seem like there is loop here but milestones can be closed or deleted, on delete it should ignore
  • check checklist for the milestone that was closed and ping it if it is not done and closed
• when PR is merged to master and doesn't have a milestone:
  • assign it to lowest version milestone open

Some others but those are first ideas.