Dynamic inventory error in Ansible AWX - Nmap Plugin

[MauriceMorrey]

Summary

I built a custom AWX Execution Environment with ansible-builder that installs Nmap CLI to be able to use the community.general.nmap plugin.

I have a dynamic inventory I was updating with my nmap plugin yaml file - see below. It was working seamlessly until about 2 weeks ago.

The timeline for the error we are getting is in line with when this patch was made.

There seems to be a similar issue that has been posted on this reddit thread.

Issue Type

Bug Report

Component Name

Nmap plugin

Ansible Version

$ ansible --version
AWX Version 23.9.0
AWX Operator v2.12.2

Community.general Version

$ ansible-galaxy collection list community.general
8.5.0

Configuration

$ ansible-config dump --only-changed

OS / Environment

No response

Steps to Reproduce

---

plugin: community.general.nmap
address: 10.x.x.x/24
strict: False
ipv4: yes
ports: no
groups:
  group_one: "'10.x.x.x/24' in hostname"

Expected Results

I expect the log output to show the number of loaded groups and hosts at the end of the file, but it does not get that far. Log output should look like below:

ansible-inventory [core 2.15.9]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
host_list declined parsing /runner/project/collections/requirements.yaml as it did not pass its verify_file() method
script declined parsing /runner/project/collections/requirements.yaml as it did not pass its verify_file() method
[WARNING]: Skipping 'collections' as this is not a valid group definition
Parsed /runner/project/collections/requirements.yaml inventory source with yaml plugin
host_list declined parsing /runner/project/nmap_host_scan.yaml as it did not pass its verify_file() method
script declined parsing /runner/project/nmap_host_scan.yaml as it did not pass its verify_file() method
Using inventory plugin 'ansible_collections.community.general.plugins.inventory.nmap' to process inventory source '/runner/project/nmap_host_scan.yaml'
Parsed /runner/project/nmap_host_scan.yaml inventory source with auto plugin
   11.754 INFO     Processing JSON output...
   11.755 INFO     Loaded 0 groups, 7 hosts
   11.809 INFO     Inventory import completed for testing_nmap_source in 0.1s

Actual Results

  executable location = /usr/local/bin/ansible-inventory
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/project/collections/requirements.yaml as it did not pass its verify_file() method
script declined parsing /runner/project/collections/requirements.yaml as it did not pass its verify_file() method
[WARNING]: Skipping 'collections' as this is not a valid group definition
Parsed /runner/project/collections/requirements.yaml inventory source with yaml plugin
setting up inventory plugins
host_list declined parsing /runner/project/nmap_host_scan.yaml as it did not pass its verify_file() method
script declined parsing /runner/project/nmap_host_scan.yaml as it did not pass its verify_file() method
Loading collection community.general from /runner/requirements_collections/ansible_collections/community/general
Using inventory plugin 'ansible_collections.community.general.plugins.inventory.nmap' to process inventory source '/runner/project/nmap_host_scan.yaml'
Parsed /runner/project/nmap_host_scan.yaml inventory source with auto plugin
   12.843 INFO     Processing JSON output...
   12.844 DEBUG    Loaded group: all

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/inventory/nmap.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[felixfontein]

Hmm, this looks like a bug in AWX, since that extra dictionary in the JSON output of ansible-inventory is what AWX explicitly requested from ansible-core for unsafe data: ansible/ansible#47295 (implemented in ansible/ansible#60602).

I recently also noticed the strange dicts in the JSON output, but was told that was intentional (ansible/ansible#82999), with a reference to that request by AWX.

CC @AlanCoding (in case you're the wrong person to ping, you probably have a better idea who should be pinged for this ;) )

[Nachichuri]

Had the same issue with the Proxmox inventory, rolling back to community.general 8.4.0 has been the fix for now, since the JSON contains a dict where a string is expected by the AWX inventory load. Leaving this for reference as another case just in case.

[MauriceMorrey]

Thanks @felixfontein for providing the extra links for clarity.

Thanks @Nachichuri for that; I found your response on reddit right after I posted this issue :)

[felixfontein]

One thought I had yesterday evening: add a helper function to be used instead of ansible.utils.unsafe_proxy that basically does the same as that function, but for (byte and unicode) strings checks whether they contain { or }, and only in that case calls ansible.utils.unsafe_proxy to make the string unsafe.

That doesn't fix the bug in AWX, but reduces the chance of triggering it.

[MauriceMorrey]

One thought I had yesterday evening: add a helper function to be used instead of ansible.utils.unsafe_proxy that basically does the same as that function, but for (byte and unicode) strings checks whether they contain { or }, and only in that case calls ansible.utils.unsafe_proxy to make the string unsafe.

That doesn't fix the bug in AWX, but reduces the chance of triggering it.

I believe this would be a good band-aid until the AWX bug is fixed.

[BlankBanshee01]

Facing the same issue with Proxmox inventory.

[felixfontein]

I implemented a workaround in #8225 that should avoid most problems until AWX fixes this bug. It would be great if you could test the changes in the PR.

[MauriceMorrey]

I implemented a workaround in #8225 that should avoid most problems until AWX fixes this bug. It would be great if you could test the changes in the PR.

Is it all merged with main?

[felixfontein]

The PR has not yet been merged.

[MauriceMorrey]

I implemented a workaround in #8225 that should avoid most problems until AWX fixes this bug. It would be great if you could test the changes in the PR.

The issue seems to persist even with the changes in the PR. Is anyone else getting a different result?

[felixfontein]

I'm going to merge the PR now since the behavior for @MauriceMorrey also persisted with older community.general versions that do not mark anything as unsafe. It's probably either caching something, or not using the version from the EE. Let's see if the situation gets better at least for some folks with the PR merged and released.

[MauriceMorrey]

Things working as expected with the PR merged. Much appreciated.

[felixfontein]

Great to hear!

[evangraudins]

@MauriceMorrey can you upload the docker image with nmap and community.general please

[MauriceMorrey]

@evangraudins What are you trying to accomplish?

If your aim is to create a custom execution environment, here's a helpful guide.