gotk_reconcile_condition not being marked as Deleted for Alert

[julienduchesne]

Version: 0.23.4 and 0.24.0

We deleted an Alert resource (renamed it) and instead of being marked as "Deleted", it is being marked as not Ready. This has happened across two clusters (and two different Flux versions)

Here's what the metrics look like for that resource:

gotk_reconcile_condition{..., status="Deleted", type="Ready"} | 0
gotk_reconcile_condition{..., status="False", type="Ready"} | 1
gotk_reconcile_condition{..., status="True", type="Ready"} | 0
gotk_reconcile_condition{..., status="Unknown", type="Ready"} | 0

These metrics trigger the alert defined here: https://github.com/fluxcd-community/flux-mixin/blob/main/alerts/flux-alert.libsonnet#L8-L20

I have confirmed that the alert is deleted. Restarting the notification-controller fixes the issue

[Alan01252]

Thank you! We were seeing this too and can confirm restarting the notification controller resolve the alert.

[stefanprodan]

We need to add finalizers to all the notification CRDs to be able to block the deleting, then set the metric, and finally free the object to allow Kubernetes to purge it. Adding finalizers will break flux uninstall so we also need to remove them with the CLI like we're doing for other CRs.