[ATMOSPHERE-103] chore: Add loki rule to delect Nova cell down

[okozachenko1203]

fix #490

[mnaser]

@okozachenko1203 I think the main issue here is that Loki is not integrated with the AlertManager deployed by the operator, so the alerts will be going to /dev/null.

Can you either make a new PR or update this one to send the alerts to AlertManager (as mentioned in https://grafana.com/docs/loki/latest/alert/#scheduling-and-best-practices)

Once we have that setup, then the alerts will actually "make sense" otherwise now they won't be going anywhere.

[mnaser]

Looks like a merge conflict and needs some work to pass CI

[mnaser]

recheck

[okozachenko1203]

ok, i found that ruler sidecar is not deployed in singleBInary deployment mode.
grafana/loki#10967

[okozachenko1203]

grafana/loki#13572

[okozachenko1203]

now i can get the alert rule in grafana GUI

[mnaser]

/cherry-pick stable/2024.1

[mnaser]

/cherry-pick stable/2023.2

[mnaser]

/cherry-pick stable/2023.1

[mnaser]

/cherry-pick stable/zed

[vexxhost-bot]

@mnaser: new pull request created: #1938

In response to this:

/cherry-pick stable/2024.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vexxhost-bot]

@mnaser: new pull request created: #1939

In response to this:

/cherry-pick stable/2023.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vexxhost-bot]

@mnaser: new pull request created: #1940

In response to this:

/cherry-pick stable/2023.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vexxhost-bot]

@mnaser: #495 failed to apply on top of branch "stable/zed":

Applying: Fix lint error
Using index info to reconstruct a base tree...
M	roles/loki/vars/main.yml
Falling back to patching base and 3-way merge...
Auto-merging roles/loki/vars/main.yml
CONFLICT (content): Merge conflict in roles/loki/vars/main.yml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Fix lint error
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick stable/zed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vexxhost-bot]

@mnaser: new issue created for failed cherrypick: #1941

In response to this:

/cherry-pick stable/zed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.