Create-federation-secret job should not run if server.updatePartition is not 0

[ishustava]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

If you are trying to upgrade a cluster without federation enabled to a cluster with federation using our recommended upgrade process, create-federation-secret job never succeeds because it tries to read replication token secret that should be created by the server-acl-init job. But the server-acl-init job doesn't run if server partition is not 0 because we want the servers to be fully upgraded to bootstrap ACLs.

Reproduction Steps

Logs

Create-federation-secret job logs

2022-08-30T21:02:19.776Z [INFO]  Retrieving server CA cert data
2022-08-30T21:02:19.776Z [INFO]  Server CA cert retrieved successfully
2022-08-30T21:02:19.776Z [INFO]  Retrieving server CA key data
2022-08-30T21:02:19.776Z [INFO]  Server CA key retrieved successfully
2022-08-30T21:02:19.779Z [INFO]  Retrieving replication token from secret: secret=consul-acl-replication-acl-token ns=consul
2022-08-30T21:02:19.903Z [WARN]  secret not yet created, retrying: secret=consul-acl-replication-acl-token ns=consul
2022-08-30T21:02:20.983Z [WARN]  secret not yet created, retrying: secret=consul-acl-replication-acl-token ns=consul
2022-08-30T21:02:21.991Z [WARN]  secret not yet created, retrying: secret=consul-acl-replication-acl-token ns=consul
2022-08-30T21:02:23.002Z [WARN]  secret not yet created, retrying: secret=consul-acl-replication-acl-token ns=consul
2022-08-30T21:02:24.012Z [WARN]  secret not yet created, retrying: secret=consul-acl-replication-acl-token ns=consul
2022-08-30T21:02:25.020Z [WARN]  secret not yet created, retrying: secret=consul-acl-replication-acl-token ns=consul
2022-08-30T21:02:26.030Z [WARN]  secret not yet created, retrying: secret=consul-acl-replication-acl-token ns=consul
2022-08-30T21:02:27.038Z [WARN]  secret not yet created, retrying: secret=consul-acl-replication-acl-token ns=consul
2022-08-30T21:02:28.046Z [WARN]  secret not yet created, retrying: secret=consul-acl-replication-acl-token ns=consul

--->

Expected behavior

create-federation-secret job should not run when server-acl-job doesn't run (i.e. when server.updatePartition is 0)

[nathancoleman]

@ishustava if no one is working on this, @20sr20 and I would like to

[ishustava]

@nathancoleman and @20sr20 Please go ahead! Thank you!