[question] Change consul SI tokens to be local?

[jorgemarey]

Hi! I was testing nomad+consul connect in some test clusters. I tested to shut down the primary consul datacenter and saw that I wasn't able to run any connect task because the consul SI token can't be created due to it being global. Does it make sense/Can it be local so if the primary DC fails for some reason other datacenters can still work properly?

I will try to change the code here and set a Local: true to test it.

https://github.com/hashicorp/nomad/blob/master/nomad/consul.go#L220

Edit: to clarify, everyting thats running it's running ok, but can't deploy new allocations.

[shoenig]

Good catch @jorgemarey !

Considering each SI token is tied to the allocation in which a task is running, and each SI token is created/destroyed only by the Nomad Servers associated with the cluster of the requesting Nomad Client, yeah I think it makes sense to create them as local tokens.

Did your test work out as expected?

[jorgemarey]

Hey @shoenig. Just tested it and everyting seems to work fine. The tokens are created locally and I was able to deploy new allocations event then the primay DC was down.

I don't know if this has any other implications, but I could make a PR with this change if you think that's ok.

[shoenig]

A PR would be great @jorgemarey , thanks!

[shoenig]

Ahh so unfortunately there is a problem with using Local:true tokens right now - if the Connect service is trying to contact an upstream in a remote datacenter, the local SI token won't be allowed. The issue being in Consul hashicorp/consul#7381, hashicorp/consul#7899.

We'll keep this issue and PR open for now, and merge it when the functionality is in place.

[jorgemarey]

Thanks @shoenig, I'll be looking forward to this fix. We're working on federating our consul DCs but we don't want nomad to fail (fail to run new allocations) if the connection to the primary consul DC fails.

[shoenig]

We should revisit this; according to consul team this may just work now - as long as the remote DC's Consul default agent token is privileged enough for agent:read

[jorgemarey]

Hi @shoenig, any news on this?

We're having some problems, and I think it's due to this. Sometimes the envoy sidecar fails to start (after a few tries it starts correctly), but I think it's related to token creation being done in the primary datacenter, so until the token gets replicated the task fails to start.

[jorgemarey]

Hi, sorry to ping again over here. Any news on this?

[shoenig]

Hey @jorgemarey sorry this has taken 2 years, but we may be in a reasonable place now to make the switch. The fundamental change comes from hashicorp/consul#7414 which shipped in Consul 1.8, which is now beyond EOL. As described in that issue the implication is the Consul agent in the remote DC will now require its anonymous ACL token to contain the permissions

service_prefix "" { policy = "read" }
node_prefix    "" { policy = "read" }

but that's not unusual (acl.tokens.default)

[jorgemarey]

Hi @shoenig. That's great. We have several federated DCs distributed all over the world. This change will allow me to sleep better :D (we never had any problem related to this, but the fear of losing contact with the primary and not being able to deploy on the secondary is there)
The requirements of the anonymous ACL tokens are OK by me. The only thing that I wonder is that this tokens are also generated for the mesh gateways to use. I don't know if changing this will affect the deployment of mesh gateways with nomad. I don't know if consul needs global tokens for the mesh gateways to work.

[MagicRB]

Even with setting the default token to something with sufficient ACLs (i've verified this), consul connect still doesn't work nor does accessing dc-1 as proxied by a server in dc-2 with a local token, my gues is that the token stripping isnt taking place like it should.

Jul 30 09:06:39 toothpick consul[363230]: 2022-07-30T09:06:39.273+0200 [ERROR] agent.http: Request error: method=GET url=/v1/internal/ui/services?dc=dc-1 from=x:42152 error="rpc error making call: ACL not found"
Jul 30 09:06:51 toothpick consul[363230]: 2022-07-30T09:06:51.125+0200 [ERROR] agent.http: Request error: method=GET url=/v1/acl/token/self?dc=dc-1 from=x:42152 error="ACL not found"
Jul 30 09:06:54 toothpick consul[363230]: 2022-07-30T09:06:54.395+0200 [ERROR] agent.http: Request error: method=POST url=/v1/internal/acl/authorize?dc=dc-2 from=x.x.x.x:42152 error="ACL not found"
Jul 30 09:06:54 toothpick consul[363230]: 2022-07-30T09:06:54.452+0200 [ERROR] agent.http: Request error: method=GET url=/v1/internal/ui/services?dc=dc-2 from=x:42152 error="ACL not found"

If I log into Consul through dc-2 with the token generated by Nomad that's set local, I can see no services, so ACL issue. Said token works in dc-1. The anonymous token I generated works everywhere.

[MagicRB]

Indeed reverting this changes fixes all my issues.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.