Flux did not notice branch deletion and undeletion

[mflendrich]

Describe the bug

1. My Flux was up and configured to poll a branch named flux-branch in a remote repo
2. I issued git push origin :flux-branch to delete the remote branch
3. Got the following Flux logs (filtered out irrelevant lines):

ts=2019-07-24T06:09:26.378624874Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:09:30.898645292Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:09:48.926886623Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:09:52.846874111Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
<--- remote delete happened here
ts=2019-07-24T06:10:11.394893034Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:10:17.692321457Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:10:33.881518139Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef

Expected the following Flux logs:

ts=2019-07-24T06:09:26.378624874Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:09:30.898645292Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:09:48.926886623Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:09:52.846874111Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
<--- remote delete happened here
ts=2019-07-24T06:10:11.394893034Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:10:17.692321457Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:10:33.881518139Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"

4. I restarted Flux:

$ kubectl delete -n wkp-flux pod/flux-6dfc9bcc94-bfglw
pod "flux-6dfc9bcc94-bfglw" deleted

5. I pushed the deleted branch (to effectively undelete it)
6. Got the following Flux logs:

ts=2019-07-24T06:18:56.665459934Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:06.665618972Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:16.665782681Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:26.665941962Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:36.666085952Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
<-- push happened here
ts=2019-07-24T06:19:46.666235994Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:56.666393099Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:20:06.666652373Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:20:16.666821377Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"

Expected the following Flux logs:

ts=2019-07-24T06:18:56.665459934Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:06.665618972Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:16.665782681Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:26.665941962Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:36.666085952Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
ts=2019-07-24T06:19:46.666235994Z caller=loop.go:85 component=sync-loop err="git repo not ready: configured branch 'flux-branch' does not exist"
<-- push happened here
ts=2019-07-24T06:19:56.666393099Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:20:06.666652373Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef
ts=2019-07-24T06:20:16.666821377Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:mflendrich/configrepo-demo.git branch=flux-branch HEAD=a44875b7890af62ab0cfe27ecb0c2514dd11ecef

To Reproduce
Steps to reproduce the behaviour:
0. Set up an empty k8s cluster

1. Install Flux 1.13.2 and point it to a GitHub repo (with manifest generation using jk enabled, if that's relevant)
2. Observe that Flux can read HEAD from the repo (WAI)
3. Delete the Git branch from the repo Flux polls
4. Observe that Flux didn't notice the deletion and claims that HEAD is unchanged (not WAI)
5. Restart Flux by deleting its pod
6. Observe that Flux says that the branch does not exist (WAI)
7. Recreate the remote branch
8. Observe that Flux says that the branch does not exist (not WAI)

Expected behavior
In reproduction step 4: I expected Flux to notice that the branch did not exist anymore.
In reproduction step 8: I expected Flux to notice the HEAD of the newly created branch.

In a scenario where a disaster happens to the Git repo (e.g. deletion of all the branches), Flux may be unable to recover after the branch is back without restarting.

Logs
Logs above.

Additional context
Add any other context about the problem here, e.g

• Flux version: 1.13.2
• Helm Operator version: n/a
• Kubernetes version: Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
• Git provider: GitHub
• Container registry provider: n/a

[stefanprodan]

I think this has been fixed in #2371 Flux will now detect a new branch without restating it.

[mflendrich]

It seems to me that #2371 may fix step 8, but not step 4 (because the newly added git fetch gets run without -p).

We should probably verify by hand.

[kingdonb]

Flux v1 is formally superseded since the GitOps Toolkit APIs have been declared stable:

https://fluxcd.io/docs/migration/timetable/

The repo will remain in maintenance for some time, but no new features can be accepted. Bugs can be addressed if they are critical and there is a PR to resolve it, but soon only CVEs can be addressed in Flux v1, and new users are all recommended to use Flux v2 for some time now.

Thanks for using Flux!