Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azure_rm_aks_info fails when using new AzureAD authentication experience #225

Closed
adhodgson1 opened this issue Aug 7, 2020 · 19 comments
Closed
Labels
bug Something isn't working work in In trying to solve, or in working with contributors

Comments

@adhodgson1
Copy link

SUMMARY

Azure has just released a new cluster AzureAD integration experience which manages the client and server applications for you. For any cluster that is using this new experience, the azure_rm_aks_info module fails because some of the AAD profile parameters returned are now Null. A workarround for us is to use azure_rm_resource_info instead.

ISSUE TYPE
  • Bug Report
COMPONENT NAME

azure_rm_aks_info

ANSIBLE VERSION

ansible 2.9.11

STEPS TO REPRODUCE
- name: Check if there is a deployed cluster to obtain credentials for
  azure_rm_aks_facts:
    name: "{{ aks_name }}"
    resource_group: "{{ aks_rg_name }}"
  register: output_aks_lookup
  when: not (aks_resource_id | default())
EXPECTED RESULTS

Lookup completes successfully.

ACTUAL RESULTS
TASK [aks_prereqs : Check if there is a deployed cluster to obtain credentials for] ************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: msrest.exceptions.ValidationError: Parameter 'ManagedClusterAADProfile.client_app_id' can not be None.
fatal: [localhost]: FAILED! => {
    "changed": false,
    "rc": 1
}

MSG:

MODULE FAILURE
See stdout/stderr for the exact error


MODULE_STDERR:

Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1596798682.505445-359-252099414179189/AnsiballZ__azure_rm_aks_facts.py", line 102, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1596798682.505445-359-252099414179189/AnsiballZ__azure_rm_aks_facts.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1596798682.505445-359-252099414179189/AnsiballZ__azure_rm_aks_facts.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.cloud.azure._azure_rm_aks_facts', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib/python3.6/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_azure_rm_aks_facts_payload_tef_6iz8/ansible_azure_rm_aks_facts_payload.zip/ansible/modules/cloud/azure/_azure_rm_aks_facts.py", line 191, in <module>
  File "/tmp/ansible_azure_rm_aks_facts_payload_tef_6iz8/ansible_azure_rm_aks_facts_payload.zip/ansible/modules/cloud/azure/_azure_rm_aks_facts.py", line 187, in main
  File "/tmp/ansible_azure_rm_aks_facts_payload_tef_6iz8/ansible_azure_rm_aks_facts_payload.zip/ansible/modules/cloud/azure/_azure_rm_aks_facts.py", line 112, in __init__
  File "/tmp/ansible_azure_rm_aks_facts_payload_tef_6iz8/ansible_azure_rm_aks_facts_payload.zip/ansible/module_utils/azure_rm_common.py", line 348, in __init__
  File "/tmp/ansible_azure_rm_aks_facts_payload_tef_6iz8/ansible_azure_rm_aks_facts_payload.zip/ansible/modules/cloud/azure/_azure_rm_aks_facts.py", line 125, in exec_module
  File "/tmp/ansible_azure_rm_aks_facts_payload_tef_6iz8/ansible_azure_rm_aks_facts_payload.zip/ansible/modules/cloud/azure/_azure_rm_aks_facts.py", line 145, in get_item
  File "/tmp/ansible_azure_rm_aks_facts_payload_tef_6iz8/ansible_azure_rm_aks_facts_payload.zip/ansible/module_utils/azure_rm_common.py", line 509, in serialize_obj
  File "/usr/local/lib/python3.6/dist-packages/msrest/serialization.py", line 581, in body
    raise errors[0]
  File "/usr/local/lib/python3.6/dist-packages/msrest/serialization.py", line 223, in validate
    Serializer.validate(value, debug_name, **self._validation.get(attr_name, {}))
  File "/usr/local/lib/python3.6/dist-packages/msrest/serialization.py", line 664, in validate
    raise ValidationError("required", name, True)
msrest.exceptions.ValidationError: Parameter 'ManagedClusterAADProfile.client_app_id' can not be None.
@Fred-sun
Copy link
Collaborator

@adhodgson1 I tested it locally and didn't have any errors getting aks info. Can you share your complete PlayBook? In addition, please help to provide "pip3 list" information. Thank you very much!

@Fred-sun Fred-sun added the work in In trying to solve, or in working with contributors label Aug 12, 2020
@taasest8
Copy link
Contributor

taasest8 commented Aug 13, 2020

I'm getting the same error for playbooks that target an Azure AKS Cluster that is using Managed Identities instead of Service Principals. To reproduce this issues you might have to create an AKS Cluster with managed identities first.

az aks create -g myResourceGroup -n myManagedCluster --enable-managed-identity

https://docs.microsoft.com/en-us/azure/aks/use-managed-identity

@foofoo-2
Copy link

Please find here the required information:

Playbook example:

- name: Test AKS Info
  hosts: localhost
  collections:
    - azure.azcollection

  tasks:
    - name: Get facts for Azure Kubernetes Service Instance
      azure_rm_aks_info:
        name: "my-cluster-mid"
        resource_group: "my-cluster-mid-rg"
      register: aks_info

pip3 list output:

$ pip3 list
Package                        Version
------------------------------ -------------
adal                           1.2.2
ansible                        2.9.6
apache-libcloud                2.8.0
applicationinsights            0.11.9
argcomplete                    1.8.1
attrs                          19.3.0
Automat                        0.8.0
azure-cli-core                 2.0.35
azure-cli-nspkg                3.0.2
azure-common                   1.1.11
azure-graphrbac                0.61.1
azure-keyvault                 1.0.0a1
azure-mgmt-authorization       0.51.1
azure-mgmt-automation          0.1.1
azure-mgmt-batch               5.0.1
azure-mgmt-cdn                 3.0.0
azure-mgmt-compute             10.0.0
azure-mgmt-containerinstance   1.4.0
azure-mgmt-containerregistry   2.0.0
azure-mgmt-containerservice    9.1.0
azure-mgmt-cosmosdb            0.5.2
azure-mgmt-devtestlabs         3.0.0
azure-mgmt-dns                 2.1.0
azure-mgmt-hdinsight           0.1.0
azure-mgmt-iothub              0.7.0
azure-mgmt-keyvault            1.1.0
azure-mgmt-loganalytics        0.2.0
azure-mgmt-marketplaceordering 0.1.0
azure-mgmt-monitor             0.5.2
azure-mgmt-network             10.2.0
azure-mgmt-nspkg               2.0.0
azure-mgmt-privatedns          0.1.0
azure-mgmt-rdbms               1.4.1
azure-mgmt-redis               5.0.0
azure-mgmt-resource            2.1.0
azure-mgmt-servicebus          0.5.3
azure-mgmt-sql                 0.10.0
azure-mgmt-storage             3.1.0
azure-mgmt-trafficmanager      0.50.0
azure-mgmt-web                 0.41.0
azure-nspkg                    2.0.0
azure-storage                  0.35.1
bcrypt                         3.1.7
blinker                        1.4
certifi                        2019.11.28
cffi                           1.14.0
chardet                        3.0.4
Click                          7.0
cloud-init                     20.1
colorama                       0.4.3
command-not-found              0.3
configobj                      5.0.6
constantly                     15.1.0
cryptography                   2.8
dbus-python                    1.2.16
distro                         1.4.0
distro-info                    0.23ubuntu1
dnspython                      1.16.0
entrypoints                    0.3
httplib2                       0.14.0
humanfriendly                  8.2
hyperlink                      19.0.0
idna                           2.8
importlib-metadata             1.5.0
incremental                    16.10.1
isodate                        0.6.0
Jinja2                         2.10.1
jmespath                       0.9.4
jsonpatch                      1.22
jsonpointer                    2.0
jsonschema                     3.2.0
keyring                        18.0.1
knack                          0.3.3
language-selector              0.1
launchpadlib                   1.10.13
lazr.restfulclient             0.14.2
lazr.uri                       1.0.3
lockfile                       0.12.2
MarkupSafe                     1.1.0
more-itertools                 4.2.0
msrest                         0.6.10
msrestazure                    0.6.2
netaddr                        0.7.19
netifaces                      0.10.4
ntlm-auth                      1.1.0
oauthlib                       3.1.0
packaging                      20.3
paramiko                       2.7.1
pip                            20.0.2
pyasn1                         0.4.2
pyasn1-modules                 0.2.1
pycparser                      2.20
pycrypto                       2.6.1
Pygments                       2.6.1
PyGObject                      3.36.0
PyHamcrest                     1.9.0
PyJWT                          1.7.1
pykerberos                     1.1.14
pymacaroons                    0.13.0
PyNaCl                         1.3.0
pyOpenSSL                      19.0.0
pyparsing                      2.4.6
pyrsistent                     0.15.5
pyserial                       3.4
python-apt                     2.0.0
python-dateutil                2.7.3
python-debian                  0.1.36ubuntu1
pywinrm                        0.3.0
PyYAML                         5.3.1
requests                       2.22.0
requests-kerberos              0.12.0
requests-ntlm                  1.1.0
requests-oauthlib              1.0.0
requests-unixsocket            0.2.0
SecretStorage                  2.3.1
selinux                        3.0
service-identity               18.1.0
setuptools                     45.2.0
simplejson                     3.16.0
six                            1.14.0
ssh-import-id                  5.10
systemd-python                 234
tabulate                       0.8.2
Twisted                        18.9.0
ubuntu-advantage-tools         20.3
ufw                            0.36
unattended-upgrades            0.1
urllib3                        1.25.8
wadllib                        1.3.3
WALinuxAgent                   2.2.46
wheel                          0.30.0
xmltodict                      0.12.0
zipp                           1.0.0
zope.interface                 4.7.1

And the resulting output:

The full traceback is:
Traceback (most recent call last):
  File "/home/myaccount/.ansible/tmp/ansible-tmp-1597330981.2387981-179033125555596/AnsiballZ_azure_rm_aks_info.py", line 102, in <module>
    _ansiballz_main()
  File "/home/myaccount/.ansible/tmp/ansible-tmp-1597330981.2387981-179033125555596/AnsiballZ_azure_rm_aks_info.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/myaccount/.ansible/tmp/ansible-tmp-1597330981.2387981-179033125555596/AnsiballZ_azure_rm_aks_info.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.azure.azcollection.plugins.modules.azure_rm_aks_info', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib/python3.6/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_azure_rm_aks_info_payload_ok6a54i8/ansible_azure_rm_aks_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_aks_info.py", line 192, in <module>
  File "/tmp/ansible_azure_rm_aks_info_payload_ok6a54i8/ansible_azure_rm_aks_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_aks_info.py", line 188, in main
  File "/tmp/ansible_azure_rm_aks_info_payload_ok6a54i8/ansible_azure_rm_aks_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_aks_info.py", line 112, in __init__
  File "/tmp/ansible_azure_rm_aks_info_payload_ok6a54i8/ansible_azure_rm_aks_info_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 428, in __init__
  File "/tmp/ansible_azure_rm_aks_info_payload_ok6a54i8/ansible_azure_rm_aks_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_aks_info.py", line 125, in exec_module
  File "/tmp/ansible_azure_rm_aks_info_payload_ok6a54i8/ansible_azure_rm_aks_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_aks_info.py", line 146, in get_item
  File "/tmp/ansible_azure_rm_aks_info_payload_ok6a54i8/ansible_azure_rm_aks_info_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 589, in serialize_obj
  File "/usr/local/lib/python3.6/dist-packages/msrest/serialization.py", line 578, in body
    raise errors[0]
  File "/usr/local/lib/python3.6/dist-packages/msrest/serialization.py", line 220, in validate
    Serializer.validate(value, debug_name, **self._validation.get(attr_name, {}))
  File "/usr/local/lib/python3.6/dist-packages/msrest/serialization.py", line 661, in validate
    raise ValidationError("required", name, True)
msrest.exceptions.ValidationError: Parameter 'ManagedClusterAADProfile.client_app_id' can not be None.

Thanks in advance for your help!

@Fred-sun
Copy link
Collaborator

I'm getting the same error for playbooks that target an Azure AKS Cluster that is using Managed Identities instead of Service Principals. To reproduce this issues you might have to create an AKS Cluster with managed identities first.

az aks create -g myResourceGroup -n myManagedCluster --enable-managed-identity

https://docs.microsoft.com/en-us/azure/aks/use-managed-identity

I followed the method you said to test locally, or did not encounter the above problems? Thank you!

@Fred-sun
Copy link
Collaborator

Fred-sun commented Aug 14, 2020

@foofoo-2 Can you try upgrading ansible-collection to the latest version? Thank you!

ansible-galaxy collection install azure.azcollection --force

@tgdfool2
Copy link

@Fred-sun: I did a quick test with the latest version of the collection (1.0.0) and same problem occurs.

@Fred-sun
Copy link
Collaborator

@Fred-sun: I did a quick test with the latest version of the collection (1.0.0) and same problem occurs.

Thank you for your feedback, I will keep watching. Thank you!

@Fred-sun Fred-sun added the bug Something isn't working label Aug 14, 2020
@adhodgson1
Copy link
Author

I just reproduced this on a cluster that used managed identities as well. If you create the cluster using the azure_rm_aks module you're good to go but if you create it using the Azure CLI or an ARM template with either the managed identity or managed AAD features then use the azure_rm_aks_info module to get information about the cluster you will run into the same issue.

@tgdfool2
Copy link

Unfortunately we are not using the azure_rm_aks module for the initial cluster deployment, but Terraform. So it would be nice if the azure_rm_aks_info module could be extended to retrieve information about any kind of AKS cluster, independently of its initial deployment mechanism.

@UnwashedMeme
Copy link
Contributor

Recommendation when testing this: use the ansible module of azure.azcollection.azure_rm_aks_info instead of collections: [azure.azcollection] at the playbook level.

If there is something wrong with the collection installation it can be silently swallowed and you will be using the code from upstream ansible. If you use the fully qualified module reference that can't happen.

E.g. in the stacktrace at the top of this ticket:

  File "/tmp/ansible_azure_rm_aks_facts_payload_tef_6iz8/ansible_azure_rm_aks_facts_payload.zip/ansible/modules/cloud/azure/_azure_rm_aks_facts.py", line 145, in get_item

That path pattern "ansible/modules/cloud/azure/_azure_rm_aks_facts.py" is from https://github.com/ansible/ansible/blob/v2.9.13/lib/ansible/modules/cloud/azure/_azure_rm_aks_facts.py, not from https://github.com/ansible-collections/azure/blob/dev/plugins/modules/azure_rm_aks_info.py (note this collection no longer uses the _facts terminology at all, whereas in ansbile 2.9 the symlink still exists).

I don't think this really changes this ticket, @foofoo-2 reproduced the bug and from their stack trace you can see the correct path pattern.

Having been super confused recently about "why doesn't the behavior match the code" I'm trying to help spread the word :-)

@joaocc
Copy link
Contributor

joaocc commented Dec 20, 2020

Any news on this issue?
We have several clusters, created via CLI on different months, but this is failing (both with explicit and implicit collection path) using ansible-2.10.4. It would be very limiting for the module not to be able to work on clusters creating with any of the other MS supported methods. Thx

Thanks

@geekq
Copy link
Contributor

geekq commented Apr 9, 2021

Looks like the whole implementation is outdated, not just azure_rm_aks_info. I have problems creating cluster as well.

While current azure documentation tells

The cluster infrastructure authentication specified is used by Azure Kubernetes Service to manage cloud resources attached to the cluster. This can be either a service principal or a system-assigned managed identity.

the current ansible-collections / azure only supports former, not latter

required: true
declares service_principal as required and allows no system-assigned managed identity.

The plugins/modules/azure_rm_aks.py implementation is created by some microsoft guy in 2018 and has been updated since migration to ansible-collections only once (nodepools feature #440) https://github.com/ansible-collections/azure/commits/dev/plugins/modules/azure_rm_aks.py

Somebody needs to update the implementation. I had a look at the implementation and I doubt I can just fix that particular problem without reworking the whole module. May be we can drive Microsoft's attention to this outdated piece of software so they make using (buying) their services easier for us.

@sdktr
Copy link

sdktr commented May 19, 2021

Same problem here. Clusters created with Managed Identities fail to use the azure_rm_aks_info module because it requires the legacy SPN properties in the response.

aad_profile_spec = dict(

@sdktr
Copy link

sdktr commented May 19, 2021

Workaround using the aks_rm_info module:

# BROKEN with Managed Identities Clusters, see: https://github.com/ansible-collections/azure/issues/225
#
# -   name: "*** Get AKS Cluster deployment state using broken azure_rm_aks_info module ***"
#     tags: k8s_infra, aks, deploy
#     register: aksoutput
#     azure_rm_aks_info:
#         name: "{{ environmenttag }}-{{ k8s.clustername }}"
#         resource_group: "{{ environmenttag }}-K8S-RG"
#
# -   debug: 
#         var: aksoutput['aks'][0]['properties']['provisioningState']
#         verbosity: 2
#     tags: always

-   name: "*** Get AKS Cluster deployment state using RM module ***"
    tags: k8s_infra, aks, deploy
    azure_rm_resource_info:
        resource_name: "{{ environmenttag }}-{{ k8s.clustername }}"
        resource_group: "{{ environmenttag }}-K8S-RG"
        provider: ContainerService
        resource_type: managedClusters
    register: aksoutput

-   name: RM resource output
    tags: always
    debug:
        var: aksoutput['response'][0]['properties']['provisioningState']
        verbosity: 2

@Fred-sun
Copy link
Collaborator

Fred-sun commented Jul 1, 2021

@adhodgson1 I have not had any problems with the latest version of Azure Collection. Can you upgrade to latest to retry. Thank you very much!

To upgrade method:
ansible-galaxy collection install azure.azcollection --force
My playbook
    - name: Create kubernet service by Azure CLI
      shell: "az aks create -g v-xisuRG01 -n FredManagedCluster03 --node-count 7"

    - name: Get aks info
      azure_rm_aks_info:
        name: FredManagedCluster03
        resource_group: v-xisuRG01
      register: output

    - name: Assert that kubernet service name is created
      assert:
        that:
          - output.aks[0].name == "FredManagedCluster03"
Result:
PLAY [Using Azure collection] **********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************
ok: [localhost]

TASK [Create kubernet service by Azure CLI] ********************************************************************************************************************************
changed: [localhost]

TASK [Get aks info] ********************************************************************************************************************************************************
[WARNING]: Azure API profile latest does not define an entry for ContainerServiceClient
ok: [localhost]

TASK [Assert that kubernet service name is created] ************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY RECAP *****************************************************************************************************************************************************************
localhost                  : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

@sdktr
Copy link

sdktr commented Jul 1, 2021

Fred, is your cluster created with Managed Identities? Not specified in your az cli shell, not sure what's the default these days.

Maybe you can post a debug of the 'output' var to verify?

@Fred-sun
Copy link
Collaborator

Fred-sun commented Jul 1, 2021

Like this

TASK [Get aks info] ********************************************************************************************************************************************************
[WARNING]: Azure API profile latest does not define an entry for ContainerServiceClient
ok: [localhost]

TASK [debug] ***************************************************************************************************************************************************************
ok: [localhost] => {
    "output": {
        "aks": [
            {
                "id": "/subscriptions/xxxxxxxxxxxxxxxx/resourcegroups/v-xisuRG01/providers/Microsoft.ContainerService/managedClusters/FredManagedCluster",
                "identity": {
                    "principalId": "xxxxxxxxxx",
                    "tenantId": "72f988bf-86f1-41af-91ab-2d7cd011db47",
                    "type": "SystemAssigned"
                },
                "location": "eastus",
                "name": "FredManagedCluster",
                "properties": {
                    "agentPoolProfiles": [
                        {
                            "count": 7,
                            "maxPods": 110,
                            "name": "nodepool1",
                            "orchestratorVersion": "1.19.11",
                            "osDiskSizeGB": 128,
                            "osType": "Linux",
                            "provisioningState": "Succeeded",
                            "type": "VirtualMachineScaleSets",
                            "vmSize": "Standard_DS2_v2"
                        }
                    ],
                    "dnsPrefix": "FredManage-v-xisuRG01-f64d4e",
                    "enableRBAC": true,
                    "fqdn": "fredmanage-v-xisurg01-f64d4e-8b5ff8ba.hcp.eastus.azmk8s.io",
                    "kubernetesVersion": "1.19.11",
                    "linuxProfile": {
                        "adminUsername": "azureuser",
                        "ssh": {[email protected]\n"
                                }
                            ]
                        }
                    },
                    "maxAgentPools": 100,
                    "networkProfile": {
                        "dnsServiceIP": "10.0.0.10",
                        "dockerBridgeCidr": "172.17.0.1/16",
                        "loadBalancerSku": "Standard",
                        "networkPlugin": "kubenet",
                        "podCidr": "10.244.0.0/16",
                        "serviceCidr": "10.0.0.0/16"
                    },
                    "nodeResourceGroup": "MC_v-xisuRG01_FredManagedCluster_eastus",
                    "provisioningState": "Succeeded",
                    "servicePrincipalProfile": {
                        "clientId": "msi"
                    }
                },
                "type": "Microsoft.ContainerService/ManagedClusters"
            }
        ],
        "available_versions": [],
        "changed": false,
        "failed": false,
        "warnings": [
            "Azure API profile latest does not define an entry for ContainerServiceClient"
        ]
    }
}

@sdktr
Copy link

sdktr commented Jul 1, 2021

Cool, seems fixed indeed:

servicePrincipalProfile": {
"clientId": "msi"

🙏

@Fred-sun
Copy link
Collaborator

Fred-sun commented Jul 1, 2021

Ok, I will closed it!

@Fred-sun Fred-sun closed this as completed Jul 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working work in In trying to solve, or in working with contributors
Projects
None yet
Development

No branches or pull requests

9 participants