[roleset] unable to set IAM policy for resource ... service account does not exist

[TJM]

There appears to be an issue with Google's API where it creates a service account, but it doesn't actually exist when Vault tries to assign it to an IAM Policy (#thxg00g).

The error from terraform looks like:

│ Error: Error updating GCP Secrets backend roleset "gcp/roleset/jenkins-finclear-df-x3yi": Error making API request.
│ 
│ URL: PUT https://vault-rops.company.com/v1/gcp/roleset/jenkins-finclear-df-x3yi
│ Code: 400. Errors:
│ 
│ * unable to set IAM policy for resource "//cloudresourcemanager.googleapis.com/projects/rops-finclear-df-rev-p-x3yi": unable to set policy: googleapi: Error 400: Service account vaultjenkins-fincle-1735057926@rops-finclear-df-rev-p-x3yi.iam.gserviceaccount.com does not exist.
│ 
│   with module.project_p.vault_gcp_secret_roleset.jenkins_dataflow,
│   on project/jenkins.tf line 2, in resource "vault_gcp_secret_roleset" "jenkins_dataflow":
│    2: resource "vault_gcp_secret_roleset" "jenkins_dataflow" {

I assume that Vault asked the Google API to create a service account, and got back a good response, then when it attempted to assign it to the project IAM policy, it failed stating that it doesn't exist. When we do this with terraform directly (non-vault), we can hit retry, and the second time through, the service account will exist, but with Vault GCP Secrets rolesets, it seems to cleanup the failed service account, which then means that it tries to create a new service account each time.

Getting Google to stop doing these asynchronous operations is likely a lost cause, so maybe there should be a bit of a retry loop on the code that assigns IAM Policies when it sees the error that the service account does not exist (yet).

~tommy

[TJM]

Opened a ticket with support, and they have opened an internal bug report to track this - VAULT-33220.

[TJM]

I was looking at the code:

vault-plugin-secrets-gcp/plugin/role_set.go

Lines 162 to 172 in 8f84406

	// Created new RoleSet resources
	// Create new service account
	sa, err := b.createServiceAccount(ctx, req, newResources.accountId.Project, newSaName, fmt.Sprintf("role set %s", rs.Name))
	if err != nil {
	return nil, err
	}
	// Create new IAM bindings.
	if err := b.createIamBindings(ctx, req, sa.Email, newResources.bindings); err != nil {
	return nil, err
	}

... and I wonder whether we could add a sortof "verifyServiceAccountExists" method between these two, that might do a "get" with a few retries? I wonder if it would be worth submitting an upstream bug report too?

Maybe using some portion of the logic from the static stuff?

vault-plugin-secrets-gcp/plugin/static_account.go

Lines 110 to 119 in 8f84406

	gcpAcct, err := b.getServiceAccount(iamAdmin, &gcputil.ServiceAccountId{
	Project: gcpServiceAccountInferredProject,
	EmailOrId: input.serviceAccountEmail,
	})
	if err != nil {
	if isGoogleAccountNotFoundErr(err) {
	return fmt.Errorf("unable to create static account, service account %q should exist", input.serviceAccountEmail)
	}
	return errwrap.Wrapf(fmt.Sprintf("unable to create static account, could not confirm service account %q exists: {{err}}", input.serviceAccountEmail), err)
	}

It is also possible that just updating the google golang API might help, the version here appears to be v195 from Aug 28, 2024. There have been a few releases already this year, we have had strange service account related issues in the terraform plugin that were fixed by a fairly recent update to the provider.

Additionally, this is apparently the "old" library, and they are encouraging people to use the "new" one: https://pkg.go.dev/google.golang.org/[email protected]/iam/v1#hdr-Library_status