Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bitnami/redis]: hotfix, regenerate sentinel config at each boot-up #7333

Merged
merged 1 commit into from
Aug 30, 2021

Conversation

rsecob
Copy link
Contributor

@rsecob rsecob commented Aug 26, 2021

Description of the change

After upgrading to 15.0.0 we realized there was an issue with sentinel. If the service is restarted within its running container, the sentinel config remained unchanged. The script is supposed to prune the known sentinels and replicas and regenerate them when fetching the available hosts on the network. This part of the script was not working as expected which caused this error:

>>> 'sentinel known-sentinel rt-collector-master 10.8.5.30 26379 ecac44470a643ce5d853d166ad8abc2135e4bdbf'
Duplicate runid for sentinel.

Our proposed solution is to override the existing configuration (if it exists) with the original configuration at each bootup. This ensures all replicas start with the expected configuration file.

Benefits

  • Fixes CrashBackLoop state if the service is restarted in a running container.

Checklist

  • Chart version bumped in Chart.yaml according to semver.
  • Title of the PR starts with chart name (e.g. [bitnami/<name_of_the_chart>])

@yilmi
Copy link
Contributor

yilmi commented Aug 27, 2021

Hi @rsecob,

Thanks for this contribution!!! I've been trying to reproduce your issue using our latest chart 15.0.3 without success.

Would you mind confirming that you also see this issue with the 15.0.3 release? And confirm the version of containers you are using?

Thanks,
Yass

@rsecob rsecob force-pushed the rb/hotfix-sentinel branch from 5bc0d53 to ba07f22 Compare August 27, 2021 15:48
@rsecob
Copy link
Contributor Author

rsecob commented Aug 27, 2021

Thank you @yilmi for your review.

Yes I confirm we have the same issue. In order to see the issue happening, you need to log in into the container and kill the process.

Here is an example:

values.yaml

auth:
  enabled: false

sentinel:
  enabled: true

Deploy and let it dispatch the three workers

helm install -f values.yaml test bitnami/redis

Then log in into one sentinel container:

ktl exec -ti test-redis-node-2 -c sentinel -- bash

And kill the sentinel process with kill 1

@yilmi
Copy link
Contributor

yilmi commented Aug 30, 2021

Thanks for the details, let me check that and get back to you quickly!

@yilmi
Copy link
Contributor

yilmi commented Aug 30, 2021

lgtm, one note, I had to recreate the pods. Thanks again @rsecob for this contribution!

@yilmi yilmi merged commit 18ecfc2 into bitnami:master Aug 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants