Private ACME CA doesn't send "challenges" key

[StevenH1901]

SUMMARY

Our private ACME server doesn't require any challenges, as long as we have a valid ACME account (granted via an external account binding). The acme_certificate role appears to be looking for a "challenges" key in the response from the ACME server, but since ours doesn't provide that it fails, even when passing challenge: "no challenge".

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.crypto.acme_certificate
More specifically it appears to be on line 144 of challenges.py.

ANSIBLE VERSION

ansible [core 2.18.1]
  config file = /Users/shodges/Projects/it-ansible-testing/ansible.cfg
  configured module search path = ['/Users/shodges/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/Cellar/ansible/11.1.0/libexec/lib/python3.13/site-packages/ansible
  ansible collection location = /Users/shodges/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible
  python version = 3.13.1 (main, Dec  3 2024, 17:59:52) [Clang 16.0.0 (clang-1600.0.26.4)] (/opt/homebrew/Cellar/ansible/11.1.0/libexec/bin/python)
  jinja version = 3.1.4
  libyaml = True

COLLECTION VERSION

# /opt/homebrew/Cellar/ansible/11.1.0/libexec/lib/python3.13/site-packages/ansible_collections
Collection       Version
---------------- -------
community.crypto 2.22.3

CONFIGURATION

CONFIG_FILE() = /Users/shodges/Projects/it-ansible-testing/ansible.cfg
DEFAULT_ROLES_PATH(/Users/shodges/Projects/it-ansible-testing/ansible.cfg) = ['/Users/shodges/Projects/it-ansible-testing/src/roles']

GALAXY_SERVERS:

OS / ENVIRONMENT

MacOS Sonoma 14.3.1
Homebrew version 4.4.11

STEPS TO REPRODUCE

Playbook calling custom role

---
- name: Generate Cert against ACME Server
  hosts: localhost
  connection: local
  gather_facts: no
  vars:
    csr_common_name: 'ansible-testing-01.vermeermfg.com'
    subject_alt_names:
      - 'DNS:ansible-testing-01'
      - 'IP:192.254.0.1'
  tasks:
    - name: Use vermeer_cert role to generate cert
      include_role:
        name: vermeer_cert

Custom Role

# - name: Ensure cert_store directory exists
#   ansible.builtin.file:
#     name: "{{ item }}"
#     state: directory
#   with_items:
#     - "{{ cert_store_path }}"
#     - "{{ cert_store_path }}/acme"
#     - "{{ cert_store_path }}/cert"
#     - "{{ cert_store_path }}/csr"
#     - "{{ cert_store_path }}/key"

# - name: Generate or Retrieve ACME Account Key
#   community.crypto.openssl_privatekey:
#     path: "{{ cert_store_path }}/acme/account.key"

# - name: Generate private key
#   community.crypto.openssl_privatekey:
#     path: "{{ cert_store_path }}/key/{{ csr_common_name }}.key"
#     size: 4096

# - name: Generate CSR
#   community.crypto.openssl_csr:
#     path: "{{ cert_store_path }}/csr/{{ csr_common_name }}.csr"
#     privatekey_path: "{{ cert_store_path }}/key/{{ csr_common_name }}.key"
#     common_name: "{{ csr_common_name }}"
#     subject_alt_name: "{{ subject_alt_names | default(omit) }}"
#     country_name: "{{ cert_csr_country }}"
#     organization_name: "{{ cert_csr_organization }}"
#     organizational_unit_name: "{{ cert_csr_ou }}"
#     state_or_province_name: "{{ cert_csr_state }}"
#     locality_name: "{{ cert_csr_locality }}"
#     email_address: "{{ cert_csr_email }}"
#   delegate_to: localhost

- name: Get ACME account info
  community.crypto.acme_account_info:
    account_key_src: "{{ cert_store_path }}/acme/account.key"
    acme_directory: "{{ cert_acme_uri }}"
    acme_version: 2
  register: acme_account_out

- name: Generate Cert against ACME CA
  community.crypto.acme_certificate:
    account_key_src: "{{ cert_store_path }}/acme/account.key"
    account_uri: "{{ acme_account_out.account_uri }}"
    acme_directory: "{{ cert_acme_uri }}"
    acme_version: 2
    csr: "{{ cert_store_path }}/csr/{{ csr_common_name }}.csr"
    challenge: "no challenge"
    dest: "{{ cert_store_path }}/cert/{{ csr_common_name }}.crt"
    fullchain_dest: "{{ cert_store_path }}/cert/{{ csr_common_name }}.fullchain"
  delegate_to: localhost

EXPECTED RESULTS

Certificate to be ordered & returned to Ansible.

ACTUAL RESULTS

ansible_certificate role expects challenges to be a returned key, but our ACME CA doesn't return that since it's private and authenticated via an EAB.

PLAYBOOK: generate_cert.yml *******************************************************************************************************************************************************
1 plays in src/playbooks/generate_cert.yml

PLAY [Generate Cert against ACME CA] *********************************************************************************************************************************************

TASK [Use vermeer_cert role to generate cert] *************************************************************************************************************************************
task path: /Users/shodges/Projects/it-ansible-testing/src/playbooks/generate_cert.yml:12
included: vermeer_cert for localhost

TASK [vermeer_cert : Get ACME account info] ***************************************************************************************************************************************
task path: /Users/shodges/Projects/it-ansible-testing/src/roles/vermeer_cert/tasks/main.yml:36
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: shodges
<127.0.0.1> EXEC /bin/sh -c 'echo ~shodges && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/shodges/.ansible/tmp `"&& mkdir "` echo /Users/shodges/.ansible/tmp/ansible-tmp-1734380065.228309-99389-109021517345991 `" && echo ansible-tmp-1734380065.228309-99389-109021517345991="` echo /Users/shodges/.ansible/tmp/ansible-tmp-1734380065.228309-99389-109021517345991 `" ) && sleep 0'
Using module file /opt/homebrew/Cellar/ansible/11.1.0/libexec/lib/python3.13/site-packages/ansible_collections/community/crypto/plugins/modules/acme_account_info.py
<127.0.0.1> PUT /Users/shodges/.ansible/tmp/ansible-local-99367lfbowm6l/tmp2ftq5hj1 TO /Users/shodges/.ansible/tmp/ansible-tmp-1734380065.228309-99389-109021517345991/AnsiballZ_acme_account_info.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/shodges/.ansible/tmp/ansible-tmp-1734380065.228309-99389-109021517345991/ /Users/shodges/.ansible/tmp/ansible-tmp-1734380065.228309-99389-109021517345991/AnsiballZ_acme_account_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/opt/homebrew/Cellar/ansible/11.1.0/libexec/bin/python /Users/shodges/.ansible/tmp/ansible-tmp-1734380065.228309-99389-109021517345991/AnsiballZ_acme_account_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/shodges/.ansible/tmp/ansible-tmp-1734380065.228309-99389-109021517345991/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "account": {
        "contact": null,
        "orders": "https://[ACME_CA_FQDN]/acme/[ACME_CA_ID]/orders/[ACME_ACCOUNT_ID]}",
        "public_account_key": {
            "e": "AQAB",
            "kty": "RSA",
            "n": "[PUBLIC_KEY]"
        },
        "status": "valid"
    },
    "account_uri": "https://[ACME_CA_FQDN]/acme/[ACME_CA_ID]/account/[ACME_ACCOUNT_ID]}",
    "changed": false,
    "exists": true,
    "invocation": {
        "module_args": {
            "account_key_content": null,
            "account_key_passphrase": null,
            "account_key_src": "/Users/shodges/Projects/it-ansible-testing/cert_store/acme/account.key",
            "account_uri": null,
            "acme_directory": "https://[ACME_CA_FQDN]/acme/[ACME_CA_ID]/directory",
            "acme_version": 2,
            "request_timeout": 10,
            "retrieve_orders": "ignore",
            "select_crypto_backend": "auto",
            "validate_certs": true
        }
    }
}

TASK [vermeer_cert : Generate Cert against Private ACME CA] **********************************************************************************************************************
task path: /Users/shodges/Projects/it-ansible-testing/src/roles/vermeer_cert/tasks/main.yml:45
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: shodges
<localhost> EXEC /bin/sh -c 'echo ~shodges && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/shodges/.ansible/tmp `"&& mkdir "` echo /Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935 `" && echo ansible-tmp-1734380067.549698-99466-186344612070935="` echo /Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935 `" ) && sleep 0'
Using module file /opt/homebrew/Cellar/ansible/11.1.0/libexec/lib/python3.13/site-packages/ansible_collections/community/crypto/plugins/modules/acme_certificate.py
<localhost> PUT /Users/shodges/.ansible/tmp/ansible-local-99367lfbowm6l/tmpxm2p19nr TO /Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/AnsiballZ_acme_certificate.py
<localhost> EXEC /bin/sh -c 'chmod u+x /Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/ /Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/AnsiballZ_acme_certificate.py && sleep 0'
<localhost> EXEC /bin/sh -c '/opt/homebrew/Cellar/ansible/11.1.0/libexec/bin/python /Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/AnsiballZ_acme_certificate.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/AnsiballZ_acme_certificate.py", line 107, in <module>
    _ansiballz_main()
    ~~~~~~~~~~~~~~~^^
  File "/Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/AnsiballZ_acme_certificate.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/AnsiballZ_acme_certificate.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.community.crypto.plugins.modules.acme_certificate', init_globals=dict(_module_fqn='ansible_collections.community.crypto.plugins.modules.acme_certificate', _modlib_path=modlib_path),
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     run_name='__main__', alter_sys=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 226, in run_module
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/modules/acme_certificate.py", line 988, in <module>
  File "/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/modules/acme_certificate.py", line 954, in main
  File "/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/modules/acme_certificate.py", line 741, in start_challenges
  File "/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/module_utils/acme/orders.py", line 96, in load_authorizations
  File "/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/module_utils/acme/challenges.py", line 175, in from_url
  File "/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/module_utils/acme/challenges.py", line 213, in refresh
  File "/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/module_utils/acme/challenges.py", line 144, in _setup
KeyError: 'challenges'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/AnsiballZ_acme_certificate.py\", line 107, in <module>\n    _ansiballz_main()\n    ~~~~~~~~~~~~~~~^^\n  File \"/Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/AnsiballZ_acme_certificate.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/Users/shodges/.ansible/tmp/ansible-tmp-1734380067.549698-99466-186344612070935/AnsiballZ_acme_certificate.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.community.crypto.plugins.modules.acme_certificate', init_globals=dict(_module_fqn='ansible_collections.community.crypto.plugins.modules.acme_certificate', _modlib_path=modlib_path),\n    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n                     run_name='__main__', alter_sys=True)\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"<frozen runpy>\", line 226, in run_module\n  File \"<frozen runpy>\", line 98, in _run_module_code\n  File \"<frozen runpy>\", line 88, in _run_code\n  File \"/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/modules/acme_certificate.py\", line 988, in <module>\n  File \"/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/modules/acme_certificate.py\", line 954, in main\n  File \"/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/modules/acme_certificate.py\", line 741, in start_challenges\n  File \"/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/module_utils/acme/orders.py\", line 96, in load_authorizations\n  File \"/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/module_utils/acme/challenges.py\", line 175, in from_url\n  File \"/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/module_utils/acme/challenges.py\", line 213, in refresh\n  File \"/var/folders/tx/1s53mj_n2wd1cy76ltmm3kv00000gq/T/ansible_community.crypto.acme_certificate_payload_ewtv4tfw/ansible_community.crypto.acme_certificate_payload.zip/ansible_collections/community/crypto/plugins/module_utils/acme/challenges.py\", line 144, in _setup\nKeyError: 'challenges'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE: No start of json char found\nSee stdout/stderr for the exact error",
    "rc": 1
}

[felixfontein]

According to RFC 8555 challenges is a required key in authz objects: https://www.rfc-editor.org/rfc/rfc8555#section-7.1.4

[StevenH1901]

Ah okay. Is there a way to see the response of acme_certificate? I tried acme_inspect but it requires an order URI and I don't have that, as acme_certificate fails with the KeyError.

[felixfontein]

You could add some debug statements that write the replies to a log, for example using https://pypi.org/project/q/.

[StevenH1901]

Appreciate the assistance! As this request is directly against the RFC I'll close this.

[felixfontein]

I implemented some code which accepts the missing challenges key in #832, since this is an incompatibility to the spec that's easy to work around.