azurerm_pim_eligible_role_assignment fails with nil error on only two resources from mapped variable

[MohnJadden]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.5.3

AzureRM Provider Version

3.64.0

Affected Resource(s)/Data Source(s)

azurerm_pim_eligible_role_assignment

Terraform Configuration Files

variable "teams" {
  type = map(object({
    team_name            = string
    location             = string
    Owner                = string
    TechnicalContact     = string
    SecurityGroup        = string
    DepartmentName       = string
    City                 = string
    ApplicationGroupType = string
    ApplicationType      = string
    LoadBalancerType     = string
    VDIType = string
    MaximumSessions      = number
  }))
}

data "azurerm_subscription" "currentSubscription" {}

resource "azurerm_resource_group" "vdi-rg" {
  for_each = var.teams
  name     = "${each.value.team_name}-VDI"
  location = coalesce(each.value.location, each.key)
  tags = {
    Owner            = coalesce(each.value.Owner, each.key)
    TechnicalContact = coalesce(each.value.TechnicalContact, each.key)
    Location         = coalesce(each.value.City, each.key)
    DepartmentName   = coalesce(each.value.DepartmentName, each.key)
    TeamName         = coalesce(each.value.team_name, each.key)
  }
}
resource "azurerm_pim_eligible_role_assignment" "role-vdi-vmadminpim" {
    for_each             = var.teams
  scope                = "/subscriptions/subscriptionGUID/resourceGroups/${azurerm_resource_group.vdi-rg[each.key].name}"
  role_definition_id = "Virtual Machine Administrator Login"
  principal_id       = coalesce(each.value.SecurityGroup, each.key)
}

Debug Output/Panic Output

Gist of error given in Terraform: https://gist.github.com/MohnJadden/1571bca338b4b91415ce29312acdd469 

Gist of errors given in terraform.log: https://gist.github.com/MohnJadden/c9aded3a187fb1d54edbf71550553cbc

Gist of example entries from mapped variable in our tfvars: https://gist.github.com/MohnJadden/db08ce6976997c1a6c77608aeff94cec

Expected Behaviour

All teams in the mapped variable contained within our tfvars file should have the Virtual Machine Administrator Login created as an Azure AD PIM eligible role for each resource group created in the configuration as specified.

Actual Behaviour

Terraform throws "Error: retrieving Role Management Policy" for only teams 4 and 11 and ends. No changes are made. The full text of the error is given in the gists linked above.

Steps to Reproduce

1. Run terraform plan or terraform apply
2. Allow Terraform to refresh the state
3. Terraform throws the error

Important Factoids

Nothing further, just a standard Azure tenant

References

No response

[bytemech]

Reproducible with same intermittent behaviour, affects groups and only some groups intermittently.

[sorenhuus]

We lost our P2 license, after getting it back, all previous created azurerm_pim_eligible_role_assignment resources are throwing:
Error: retrieving Role Management Policy: (Principal Id "xxxxxxxxx" / Scope "/subscriptions/yyyyyyyyy" / Role Definition Id "/subscriptions/zzzzzzzzz/providers/Microsoft.Authorization/roleDefinitions/qqqqqqqq"):

[MohnJadden]

I am intermittently getting this error - I can create it if the PIM assignment is not already present in most cases, but in others it just still gives a nil error.

Edit: just noticed something interesting. The one team that was giving me problems had the same PIM role already present as active, rather than eligible. Not sure how that came to pass. After removing the active role by going to the resource group in question and removing it from IAM, it still throws the nil error here.

In the end, I had to edit the state file manually to remove the index key item for azurerm_pim_eligible_role_assignment for the index key throwing the nil error. Doing so allowed TF to proceed and create the role successfully. I'm guessing all of this was leftovers from when TF was still having issues from earlier versions of this resource provider.

@sorenhuus If you check in the portal, are the PIM assignments still present for the scopes in question? If so, are you able to destroy them via TF?

[kristeey]

@MohnJadden Removing the resources from state works for me as well, but not permanently. In my experience I get this problem every time I run a terraform plan after the scheduled expiration time is exceeded, which is problematic when I need to extend my terraform configuration to create new eligible role assignments. Does anyone have the same experience using azapi terraform provider instead?

[sorenhuus]

I am seeing the same behavior as @kristeey - Every 3month we are hacking away at the state file to re-apply the pim roles.

[sorenhuus]

@MohnJadden They are there in the portal, on apply I get "A resource with the ID xxxxxxxxxxxx already exists - to be managed via Terraform this resource needs to be imported into the State...", so something must have been removed from the state.

If I then delete the pim assignments form the portal. Terraform fails refreshing on the plan step. So something must still be in the state.
Then I remove the failing resources from the state, and I am able to plan and apply.

[A30001546]

@MohnJadden They are there in the portal, on apply I get "A resource with the ID xxxxxxxxxxxx already exists - to be managed via Terraform this resource needs to be imported into the State...", so something must have been removed from the state.

If I then delete the pim assignments form the portal. Terraform fails refreshing on the plan step. So something must still be in the state. Then I remove the failing resources from the state, and I am able to plan and apply.

yes we are facing the same issue and hence we stopped using the terraform PIM resource and reverted back to bash script

[MohnJadden]

@MohnJadden They are there in the portal, on apply I get "A resource with the ID xxxxxxxxxxxx already exists - to be managed via Terraform this resource needs to be imported into the State...", so something must have been removed from the state.

If I then delete the pim assignments form the portal. Terraform fails refreshing on the plan step. So something must still be in the state. Then I remove the failing resources from the state, and I am able to plan and apply.

Gotcha - the only other thing I can think of is to try terraform state list and see if the PIM resource is already present, then use terraform state rm to remove it. If it's not in terraform state list, maybe it's still in the state file and require manual removal.

Unfortunately none of this fixes the root cause - Terraform somehow misses picking up the existing resource, the import dialog for the PIM resource provider behaves oddly and isn't well scoped or documented for Windows, and it doesn't cleanly add/remove. The other complication is by PIM eligible vs. PIM active, and the fact that we can't do PIM assignment settings - those have to happen in the portal.

Honestly, speaking as someone who doesn't really do much TF and has been trying to undertake more in TF rather than in the Azure portal - this is something we only do once per resource group or subscription, and the fact that we have existing manual steps to fill the gap, this is just not a feature that anyone thought of as anything other than the 1-year block. Hopefully they put in fixes for this issue and the others but it just seems like PIM for Azure resources is just not as sexy a feature for TF to add functionality from end to end.

[sorenhuus]

I made a small PS script that takes the raw Azure DevOps log, with the failed applies from the azurerm_pim_eligible_role_assignment resources, and deletes them. After thats done, plan and apply can be run without errors.

Its not refined in anyway. Hope it helps someone.

$errorText = Get-Content .\erroroutput.txt

# Finds: │ Error: A resource with the ID "/subscriptions/xxxx-xxxx-xxxx-xxxx|/subscriptions/xxxx/providers/Microsoft.Authorization/roleDefinitions/xxxx|xxxx" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_pim_eligible_role_assignment" for more information.
$pimErrors =  $errorText | where { $_ -like '*ID "/subscriptions*azurerm_pim_eligible_role_assignment*' }

$pimErrors.Count

Connect-AzAccount -Tenant "xxxx"

foreach ($item in $pimErrors)
{

    $DirectoryScopeId = $item.Split('"')[1].Split('|')[0]
    $guid = New-Guid

    $roleEligibilityScheduleInstances = (Invoke-AzRestMethod -Path "$($DirectoryScopeId)/providers/Microsoft.Authorization/roleEligibilityScheduleInstances?api-version=2020-10-01&$filter=atScope()" -Method GET).Content | ConvertFrom-Json

    $payload = @{
        Properties = @{
            PrincipalId = ($roleEligibilityScheduleInstances.value.properties.principalId)
            RoleDefinitionId = ($roleEligibilityScheduleInstances.value.properties.roleDefinitionId)
            RequestType = "AdminRemove"
            Justification = $null
            ScheduleInfo  = {
                StartDateTime  = $null
                Expiration = @{
                    Type = "NoExpiration"
                }
            }
            Condition = $null
            ConditionVersion = $null
        }
    }

    $response = (Invoke-AzRestMethod -Path "$($DirectoryScopeId)/providers/Microsoft.Authorization/roleEligibilityScheduleRequests/$($guid)?api-version=2020-10-01" -Method PUT -Payload ($payload | ConvertTo-Json)).Content | ConvertFrom-Json
    $response.properties
}