Receiving a Status: 410 Gone Error for resource: "okta_email_sender" when enabling multibranding in Okta

[mvitt]

Upon Turning on the MultiBranding feature in Okta and running Terraform I get the following error:

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v1.4.5

Okta Terraform version

3.46.0

Affected Resource(s)

• okta_email_sender

Terraform Configuration Files

resource "okta_email_sender" "vanity_email_address" {
  from_name    = "email name"
  from_address = var.vanity_email
  subdomain    = "subdomainName"
}

Expected Behavior

Should be able create a vanity email (It works when the multibranding feature off).

Can this be done in the Admin UI?

Yes

Can this be done in the actual API call?

Yes

Actual Behavior

The API is returning a Error: failed to get custom email sender: the API returned an error: This endpoint has been deprecated., Status: 410 Gone error.

Steps to Reproduce

1. Turn on multibranding
2. Run terraform apply on a okta_email_sender resource

[monde]

Thanks @mvitt we'll have to get this on to the back backlog and take a look. Sounds like a bug to me.
Okta internal reference https://oktainc.atlassian.net/browse/OKTA-606425

[afelder09]

Is there any timeline on when this bug will be fixed? We have an Okta pipeline using this terraform that is now failing with the same error described above. Path forward is to remove the okta_email_sender from the configuration and remove it from state, but a fix to the terraform repo would let us avoid that.

Thanks in advance

[dmsolutionsmn]

This feature appears to be broken due to using an undocumented API /api/v1/org/email/sender. (openapi docs https://github.com/okta/okta-sdk-golang/blob/v2.18.0/.generator/okta-management-APIs-oasv3-noEnums-inheritance.yaml ). This API had a breaking change causing this resource to break as described above.

I'm guessing the current api is the following: https://github.com/okta/okta-sdk-golang/blob/v3.0.2/okta/api_email_domain.go . Or you can look for the /api/v1/email-domains in the opanapi documentation https://github.com/okta/okta-sdk-golang/blob/v3.0.2/okta/api/openapi.yaml

I'm going to likely just delete this resource from terraform state to get my pipelines working. Hopefully there is an update to allow us to manage these in terraform again going forward.

[ClintonianSunBlaster]

Hi @monde,

This really needs to be addressed asap and not put on the backlog.

Okta introduced a breaking change which has now rendered the pipelines of all customers using the okta_email_sender resource useless (including my company). As @dmsolutionsmn suggests, I could remove the okta_email_sender from the state file as a work-around but that defeats the purpose of using a state file in the first place.

Doesn't feel right that a breaking change is being pushed to the back of the bus. Either the change should be backed out or a compensating change should be made to address the issue.

[monde]

@ClintonianSunBlaster / all, @duytiennguyen-okta is finishing up a work around that will hopefully go out within the week. Definitely not on the back burner.

[hpe-noinarisak]

Thanks, @mond! Keeps us updated! We are also waiting to implement the new official endpoints.

[dmbaJoe]

We are one more company waiting on this fix. Thanks for your hard work.

[ClintonianSunBlaster]

@monde,

This continues to be an issue. When will this be fixed?

It's now been 80 days since Okta introduced this breaking change.

We were informed by Okta Support on on June 8th that the okta_email_sender api has been deprecated and replaced by the email_domain api (https://developer.okta.com/docs/api/openapi/okta-management/management/tag/EmailDomain/).

When will the okta_email-sender tf resource be updated to use the new API?

Thanks, Clint

[duytiennguyen-okta]

This is fixed in the release 4.0.2 from #1588

[ClintonianSunBlaster]

@duytiennguyen-okta and @monde,

This issue is "technically" fixed but the fix caused new issues as the new okta_email_domain resource appears to be misconfigured. Specifically, the dns_validation_records attribute contains a sub-attribute called "values" which is is incorrectly documented as "value" at https://registry.terraform.io/providers/okta/okta/latest/docs/resources/email_domain and the "values" attribute contains no data.

This creates an issue as we can no longer setup the associated DNS records with TF and thus the new okta_email_domain_validation resource always fails.

Please investigate and fix as soon as possible. Thanks!

Here is an example. Notice "values" and it's empty values...

okta_email_domain = {
  "brand_id" = "bnd7h60x12vaCa1xxxx7"
  "display_name" = "NoReply"
  "dns_validation_records" = tolist([
    {
      "expiration" = ""
      "fqdn" = "_oktaverification.auth-test.****.com"
      "record_type" = "TXT"
      "values" = tolist([])
    },
    {
      "expiration" = ""
      "fqdn" = "mail.auth-test.****.com"
      "record_type" = "cname"
      "values" = tolist([])
    },
    {
      "expiration" = ""
      "fqdn" = "p06._domainkey.auth-test.****.com"
      "record_type" = "cname"
      "values" = tolist([])
    },
    {
      "expiration" = ""
      "fqdn" = "p36._domainkey.auth-test.****.com"
      "record_type" = "cname"
      "values" = tolist([])
    },
  ])
  "domain" = "auth-test.****.com"
  "id" = "OeD8mxsffffoooU41d7"
  "user_name" = "noreply"
  "validation_status" = "NOT_STARTED"
}

[ClintonianSunBlaster]

@duytiennguyen-okta and @monde,

Sorry to pile on here but...

There appears to be another issue with the new okta_email_domain resource and it's requirement for a brand_id.

okta_email_domain requires a "brand_id" but it appears the only way to retrieve a list of brand_ids is to use the okta_brand data resource which outputs a list of brands based on the okta_brands data resource output (see https://registry.terraform.io/providers/okta/okta/latest/docs/data-sources/brand). This creates a problem because okta_brand doesn't have a reference attribute called "name" so there is no way to retrieve the brand id of a specific custom brand. Instead, you can only reference "tolist(data.okta_brands.test.brands)[x].id" which retrieves a brand id from a list based on a list index. It appears that index 0 always gets the brand id of the default brand which happens to be the first brand on the list. But, if you have multiple custom brands and would like the brand id of a specific one, it's guesswork as to which index to use. And worse yet, if you're managing multiple brands across multiple environments, then it's entirely possible indexes could different between environments.

I think a fix would be to update the okta_brand data resource to include a returned attribute called "name" then I/us/we could use a locals that searches the list of brands returned from okta_brand for a brand which matches a specific name to get its specific brand_id? Once we have the correct brand_id we can then use it in the new okta_email_domain resource.

Thanks again, Clint

[monde]

Following up with @ClintonianSunBlaster

You're correct, the documentation was typo'd, I've submitted a fix: #1603

I'm not following your example okta_email_domain is this pseudo JSON or pseudo HCL? Also, the only arguments you can set on the resource are: brand_id, domain, display_name, user_name
So if your example is pseudo HCL you wouldn't be able to set dns_validation_records, it is a reader that you can refer to in other TF config.

Can you clarify your first question/statement?

In the second comment, as of now determining the brand id is up to the operator it seems to me. The data.okta_brands.test.brands[0].id examples that one sees are trivial quick examples. I would think that an operator would either know and hard code the brand_id into the config OR, and even better, you would import the brand as a resource and refer to its ID.

But, I see your point, if name was surfaced on the brand resource and data source it would allow the operator to implement a selector as you point out. We should add brand as an attribute. I've created a feature request for this #1604 , thanks for pointing that out.

[hpe-noinarisak]

@monde Could we consider the values attribute name to be consistent with email_domain openapi spec. e.g., rename to verificationValue. Which from the doc seems to be a single string (TypeString) and not a list (TypeList), maybe the api reference doc is incorrect.

@ClintonianSunBlaster, My only thought about a temporary workaround for your multiple brands is to create a map and probably use customPrivacyPolicyUrl as a key and with contains using a few other Terraform builtin functions to align okta_email_domain with the correct brand_id. Assuming customPrivacyPolicyUrl is a unique URL of course.

Hopefully, @monde can get into the queue to at least expose the name attribute. 🙏 Since I'm also planning to use these resources but only have single brand_id for now and can use fix brand_id.

[hpe-noinarisak]

Oh, I would also consider re-opening this issue @monde, IMO. 😸

[ClintonianSunBlaster]

@monde,

Thanks for submitting the request to fix the documentation for okta_email_domain and for submitting an enhancement request to have 'name' surfaced for the okta_brand data resource.

The example I provided is the output of our okta_email_domain resource. As you can see, the object is created and everything looks fine except that for some reason the "values" attribute has no actual value. It's simply "tolist([])". The old okta_email_sender resource spit out both the fqdn and value which are both needed to create a DNS record with say the InfoBlox TF Provider. And, if you don't create the DNS records then you can't run the okta_email_domain_verification resource.

So what ends up happening is you create the email domain using Terraform, then you have to login to the Okta Admin Console and manually copy the fqdn and values of each DNS record into Infoblox then once the DNS records are there you need to manually click the "verification" button. This procedure doesn't really fit into CI/CD...

[ClintonianSunBlaster]

@monde,

Here is my best attempt at summary...

• a breaking change was introduced that clobbered the API that okta_email_sender needed
• a new resource called okta_email_domain was created to replace okta_email_sender
• the new okta_email_domain resource requires a brand_id but the old okta_email_sender did not.
• programmatically pulling the specific brand_id of a custom brand out of okta_brand is impossible because "name" is not surfaced leaving only the option of hard-coding the brand_id value. Blech!
• the documentation for the new okta_email_domain incorrect listed "value" instead of "values"
• the "values" attribute doesn't contain any data so there is no way to get the info needed to create DNS records using TF
• since you can't create DNS records with TF, you can't run okta_email_domain_verification which is required in order to actually activate the custom email domain

Layers and layers of poo. ;-)

[ClintonianSunBlaster]

@hpe-noinarisak

Thanks for the work-around idea. Not sure it works for us as the custom_privacy_url is the same for multiple brands...

[monde]

@ClintonianSunBlaster can you run your example with TF_LOG=debug and show me the API output of the GET /api/v1/email-domains/{id} ? See values are documented on dnsValidationRecords but that doesn't mean the API actually returned values. The code for values is just reading the API response, if there isn't anything in the response, then this is an issue with the Management API, not the TF provider.

"Layers and layers of poo. ;-)" are you trying to be offensive?

[ClintonianSunBlaster]

It was a joke. It was not meant to be offensive nor directed toward you personally.

But since we're on the topic, I am not offended by Okta or it's staff although I would say that my company has every right to be offended.

We pay your company a lot of money to provide services and a portion of those services have been rendered useless for over three months by a breaking change introduced by Okta apparently because your API folks and Terraform folks aren't communicating. Then to make matters worse, the fix offered up for the breaking change was designed with a new dependency, clearly untested, and documented poorly. And now you're asking paying customers to troubleshoot whether or not it's Okta's Terraform or Okta's API that isn't working properly.

Okta's mistakes have caused my company to burn through a minimum of 20 hours worth of labor for troubleshooting, finding work-arounds, and manually performing steps that were automatic CI/CD operations three months ago. And it's not just my company that has been feeling this pain for the past three months, it's every company that had the misfortune of using the okta_email_sender resource. I can't imagine the total number of dollars and hours this breaking change has cost Okta's customers but I'm certain it would be eye-popping.

I don't know what person, group, process, procedure, or technology at Okta is to blame but the fact remains that a service which we all pay for is not available because of something that went wrong at Okta. At this point I'm done troubleshooting.

And although my description of the issue is accurate, I apologize for using words you find offensive. Again, it was not directed at you personally or any individual. It was referring to the situation and the interdependent layers of technology that have collectively failed.

Please let me and the rest of Okta's customer's know when the fix will be ready. Thank you for attending to this long outstanding issue. Clint Robinson

[hpe-noinarisak]

@monde or @duytiennguyen-okta, can you bring @ClintonianSunBlaster feedback to the attention of Jeff Taylor and/or Nick Gamb. It is valid feedback since our team also lost time working around this issue, and it is best to have visibility, other than twittering to https://twitter.com/toddmckinnon. 🙃

[jefftaylor-okta]

Hey @ClintonianSunBlaster! First of all, I'm sorry this has been such a toil for you trying to get this functionality to work. Second, thank you for giving us this feedback! 🙂 You have definitely identified some areas we are actively trying to improve.

The deprecation of old process to deal with emails was definitely challenging. What we are doing in response to this pain is taking a holistic look at how we deal with the API lifecycle. We are early on in this journey, but our hope is not repeat our mistakes of the past.

In addition, we are updating the structure of our docs to more directly reflect the state and structure of all of Okta's APIs. Unfortunately, you did discover a bug that exists in the generation of this API. We will definitely be looking into that. We are reaching out to the team to get the issue resolved.

Finally, there is dealing with our defaults, like our default brand. I have spoken with @monde about this issue in particular and I believe we have this capability in the brands resource, but I do understand your need to reference the default brand more explicitly in the emails resource. We will take a look to see how we can improve here as well.

I hope this experience doesn't prevent you from leaving feedback it's important for us to hear to improve and advocate for better experiences.

Thanks again!

[ClintonianSunBlaster]

@jefftaylor-okta ,

I appreciate the message and acknowledgement of some issues Okta is currently working through. I'm also glad to see that Okta is addressing the setup of Brands, Brand, and Default Brands, it's a confusing setup.

I'd also like to point out that @monde and @duytiennguyen-okta have done excellent work and I empathize with the challenging position they're working through.

Thank you, Clint Robinson

[Reflejo]

@monde I'm happy to help debug if useful. But I'm hitting the same issue, this is the one for me:

terraform-provider-okta/sdk/email_sender.go

Line 72 in b4ad5a9

url := fmt.Sprintf("/api/v1/org/email/sender/%s", id)

/api/v1/org/email/sender/<whatever> seems to be deprecated

EDIT: Ok after reading a few more of this thread, I realize now that the other method was (silently?) deprecated and I'm supposed to use okta_domain_sender I'd add a note on the terraform documentation if possible.

[mohankumar-studiographene]

@ClintonianSunBlaster can you run your example with TF_LOG=debug and show me the API output of the GET /api/v1/email-domains/{id} ? See values are documented on dnsValidationRecords but that doesn't mean the API actually returned values. The code for values is just reading the API response, if there isn't anything in the response, then this is an issue with the Management API, not the TF provider.

"Layers and layers of poo. ;-)" are you trying to be offensive?

@monde reminder if this issue is lost in the conversation, here is the Request response

[DEBUG] provider.terraform-provider-okta_v4.0.2: 2023/06/27 12:03:11 [DEBUG] performing request: method=GET url=https://daphne-dev.okta.com/api/v1/email-domains/<  >
[INFO]  provider.terraform-provider-okta_v4.0.2: 2023/06/27 12:03:11 [DEBUG] Okta API Request Details:

---[ REQUEST ]---------------------------------------
GET /api/v1/email-domains/<  > HTTP/1.1
Host: daphne-dev.okta.com
User-Agent: okta-sdk-golang/3.0.0 golang/go1.20.4 linux/amd64 okta-terraform/3.46.0
Accept: application/json
Authorization: SSWS [MASKED]
Accept-Encoding: gzip

[duytiennguyen-okta]

@ClintonianSunBlaster I have add the name for brand and fixed the okta_email_domain on v4.0.3. Can you try again?

[mohankumar-studiographene]

@duytiennguyen-okta I tried with provider v4.1.0. name is available in okta_brands, and okta_email_domain returns the DNS values. Thank you.

Can you make brand ref in data "okta_brand" data block using brand_name instead of brand_id, or better if it is possible to ref brand in okta_email_domain resource by brand_name instead of brand_id?

At present to get brand ID of specific brand, this is the solution, [for m in data.okta_brands.test.brands : m.id if m.name == "thebrandname"][0]

[patrick-m-m]

I have been following this issue as my company's Okta Terraform management has been broken since the email sender endpoint went 410 Gone. I am surprised to find that updating the provider version is not enough to fix it. It seems the actual fix is to replace okta_email_sender resources (which are now flagged as DEPRECATED in the provider docs, which is a bit misleading if the okta_email_sender is actually nonfunctional) with the newer okta_email_domain resources. However, it's not clear how we can do this in a seamless manner. It would seem to me that the simple approach of just changing the resource definition in the .tf files would attempt to destroy the existing resource and/or create a new email_domain, neither of which is desirable.

I looked at the release notes for versions 4.0+ and all the comments on this and related issues and haven't found any upgrade guide or version migration instructions. Did I miss it somewhere? Are we left to figure out how to manipulate the TF State ourselves? I apologize if that's the obvious answer – I've grown to expect that Terraform "just works".