redis cache plugin does not accept (bracketed) IPv6 addresses #497

branleb · 2020-06-11T12:48:24Z

SUMMARY

The redis cache plugin does not accept a fact_caching_connection with an IPv6 address

ISSUE TYPE

Bug Report

COMPONENT NAME

redis cache plugin

ANSIBLE VERSION

ansible 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0]

CONFIGURATION

ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
INTERPRETER_PYTHON(/etc/ansible/ansible.cfg) = auto_legacy_silent
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False

OS / ENVIRONMENT

I used Debian Buster 10.4 backports packages

STEPS TO REPRODUCE

Change CACHE_PLUGIN_CONNECTION(/etc/ansible/ansible.cfg) to a value with an IPv6 address, e.g. "[::1]:6379:0" (In my actual use case, its a link local unicast address from another machine, but ip6-localhost produces the same result) and use the redis fact cache plugin

- hosts: all
  tasks:
    - setup:

EXPECTED RESULTS

The play would run

ACTUAL RESULTS

ansible won't run at all

ERROR! Unexpected Exception, this is probably a bug: invalid literal for int() with base 10: ''
the full traceback was:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 988, in get_connection
    connection = self._available_connections.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/ansible", line 123, in <module>
    exit_code = cli.run()
  File "/usr/lib/python3/dist-packages/ansible/cli/adhoc.py", line 165, in run
    result = self._tqm.run(play)
  File "/usr/lib/python3/dist-packages/ansible/executor/task_queue_manager.py", line 241, in run
    play_return = strategy.run(iterator, play_context)
  File "/usr/lib/python3/dist-packages/ansible/plugins/strategy/linear.py", line 282, in run
    _hosts=self._hosts_cache, _hosts_all=self._hosts_cache_all)
  File "/usr/lib/python3/dist-packages/ansible/vars/manager.py", line 305, in get_vars
    facts = wrap_var(self._fact_cache.get(host.name, {}))
  File "/usr/lib/python3.7/_collections_abc.py", line 660, in get
    return self[key]
  File "/usr/lib/python3/dist-packages/ansible/vars/fact_cache.py", line 31, in __getitem__
    if not self._plugin.contains(key):
  File "/usr/lib/python3/dist-packages/ansible/plugins/cache/redis.py", line 132, in contains
    self._expire_keys()
  File "/usr/lib/python3/dist-packages/ansible/plugins/cache/redis.py", line 125, in _expire_keys
    self._db.zremrangebyscore(self._keys_set, 0, expiry_age)
  File "/usr/lib/python3/dist-packages/redis/client.py", line 2532, in zremrangebyscore
    return self.execute_command('ZREMRANGEBYSCORE', name, min, max)
  File "/usr/lib/python3/dist-packages/redis/client.py", line 772, in execute_command
    connection = pool.get_connection(command_name, **options)
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 990, in get_connection
    connection = self.make_connection()
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 1025, in make_connection
    return self.connection_class(**self.connection_kwargs)
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 452, in __init__
    self.port = int(port)
ValueError: invalid literal for int() with base 10: ''

The text was updated successfully, but these errors were encountered:

ansibullbot · 2020-06-11T12:49:20Z

Files identified in the description:

plugins/cache/redis.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

mator · 2020-10-23T16:03:42Z

i can't reproduce issue:

[mator@archlinux ansible.dev]$ cat ansible.cfg 
[defaults]
inventory = hosts
# gather_timeout = 20
# debug=true
log_path = ./log
ANSIBLE_NOCOWS = 1
interpreter_python = auto_silent
collections_paths = ./collections
pipelining = ANSIBLE_PIPELINING
DEVEL_WARNING = 0
fact_caching_connection = [::1]

[mator@archlinux ansible.dev]$ ansible-config dump --only-changed               
CACHE_PLUGIN_CONNECTION(/home/mator/ansible.dev/ansible.cfg) = [::1]
COLLECTIONS_PATHS(/home/mator/ansible.dev/ansible.cfg) = ['/home/mator/ansible.dev/collections']
DEFAULT_HOST_LIST(/home/mator/ansible.dev/ansible.cfg) = ['/home/mator/ansible.dev/hosts']
DEFAULT_LOG_PATH(/home/mator/ansible.dev/ansible.cfg) = /home/mator/ansible.dev/log
DEVEL_WARNING(/home/mator/ansible.dev/ansible.cfg) = False
INTERPRETER_PYTHON(/home/mator/ansible.dev/ansible.cfg) = auto_silent

[mator@archlinux ansible.dev]$ ansible --version
ansible 2.11.0.dev0 (devel b6360dc5e0) last updated 2020/10/23 18:50:01 (GMT +300)
  config file = /home/mator/ansible.dev/ansible.cfg
  configured module search path = ['/home/mator/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/mator/ansible.dev/ansible/lib/ansible
  ansible collection location = /home/mator/ansible.dev/collections
  executable location = /home/mator/ansible.dev/ansible/bin/ansible
  python version = 3.8.6 (default, Sep 30 2020, 04:00:38) [GCC 10.2.0]
  libyaml = True

[mator@archlinux ansible.dev]$ ansible -m setup localhost -a 'gather_subset=min'
localhost | SUCCESS => {
    "ansible_facts": {
...

^^ full output of setup module output follows

felixfontein · 2020-10-23T16:34:33Z

Looking at the code, I'm surprised if it really works: https://github.com/ansible-collections/community.general/blob/main/plugins/cache/redis.py#L128

felixfontein · 2020-10-23T16:35:04Z

(Or maybe the redis Python library got some magic to extract the correct IPv6 address, but it was only added later?)

branleb · 2020-10-23T16:52:42Z

i can't reproduce issue:

[mator@archlinux ansible.dev]$ cat ansible.cfg 
[defaults]
inventory = hosts
# gather_timeout = 20
# debug=true
log_path = ./log
ANSIBLE_NOCOWS = 1
interpreter_python = auto_silent
collections_paths = ./collections
pipelining = ANSIBLE_PIPELINING
DEVEL_WARNING = 0
fact_caching_connection = [::1]

Can you reproduce it with your setup if you explicitly specify a port number? Because that was also done in the reported scenario.

mator · 2020-10-23T20:48:58Z

@branleb can you please tell how do i specify it ? i'm actually no idea what is CACHE_PLUGIN_CONNECTION

Akasurde · 2020-12-14T15:36:27Z

@branleb Can you please provide the required details? Thanks.

needs_info

branleb · 2020-12-16T15:39:31Z

@branleb can you please tell how do i specify it ? i'm actually no idea what is CACHE_PLUGIN_CONNECTION

e.g.:
fact_caching_connection = "[::1]:6379:0"
which produces the initially mentioned error - whereas
fact_caching_connection = "localhost:6379:0"
or
fact_caching_connection = "127.0.0.1:6379:0"
work fine.

I just tried the case you specified as cannot reproduce (bracketed IPv6 localhost, no quoting)
fact_caching_connection = [::1]
(and, just to be sure I tried
fact_caching_connection = [::1]:6379:0
too)

Didn't work for me either. I got the same result as initially described:

`PLAY [ansible] ****************************************************************************************************************************************************************************************************************************************************************
ERROR! Unexpected Exception, this is probably a bug: invalid literal for int() with base 10: ''
the full traceback was:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/redis/connection.py", line 988, in get_connection
connection = self._available_connections.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/bin/ansible-playbook", line 123, in
exit_code = cli.run()
File "/usr/lib/python3/dist-packages/ansible/cli/playbook.py", line 127, in run
results = pbex.run()
File "/usr/lib/python3/dist-packages/ansible/executor/playbook_executor.py", line 169, in run
result = self._tqm.run(play=play)
File "/usr/lib/python3/dist-packages/ansible/executor/task_queue_manager.py", line 241, in run
play_return = strategy.run(iterator, play_context)
File "/usr/lib/python3/dist-packages/ansible/plugins/strategy/linear.py", line 282, in run
_hosts=self._hosts_cache, _hosts_all=self._hosts_cache_all)
File "/usr/lib/python3/dist-packages/ansible/vars/manager.py", line 305, in get_vars
facts = wrap_var(self._fact_cache.get(host.name, {}))
File "/usr/lib/python3.7/_collections_abc.py", line 660, in get
return self[key]
File "/usr/lib/python3/dist-packages/ansible/vars/fact_cache.py", line 31, in getitem
if not self._plugin.contains(key):
File "/usr/lib/python3/dist-packages/ansible/plugins/cache/redis.py", line 132, in contains
self._expire_keys()
File "/usr/lib/python3/dist-packages/ansible/plugins/cache/redis.py", line 125, in _expire_keys
self._db.zremrangebyscore(self._keys_set, 0, expiry_age)
File "/usr/lib/python3/dist-packages/redis/client.py", line 2532, in zremrangebyscore
return self.execute_command('ZREMRANGEBYSCORE', name, min, max)
File "/usr/lib/python3/dist-packages/redis/client.py", line 772, in execute_command
connection = pool.get_connection(command_name, **options)
File "/usr/lib/python3/dist-packages/redis/connection.py", line 990, in get_connection
connection = self.make_connection()
File "/usr/lib/python3/dist-packages/redis/connection.py", line 1025, in make_connection
return self.connection_class(**self.connection_kwargs)
File "/usr/lib/python3/dist-packages/redis/connection.py", line 452, in init
self.port = int(port)
ValueError: invalid literal for int() with base 10: ''
`

So apparently, the bug has been fixed somewhere in beetween v2.9.6 and your v2.11.0.dev0?

aminvakil · 2021-05-01T13:13:06Z

@branleb Could you please test if your problem has been solved with latest ansible-core and ansible releases?

needs_info

russoz · 2021-05-22T09:16:43Z

@branleb I took a look at the code, and the fact that it was using .split(':') to sort those fields raised a red-flag in my mind. I have raised this PR above to replace that with a regular expression, that should be more solid.

In the long run, I believe we should deprecate the current syntax and try to use a more standard url, maybe in the like of:

redis://[::1]:6379/0

But that should be a different PR than this one.

ansibullbot added affects_2.10 bug This issue/PR relates to a bug labels Jun 11, 2020

settings bot removed affects_2.10 labels Jun 16, 2020

ansibullbot added the traceback label Jun 16, 2020

ansibullbot added the affects_1 label Oct 23, 2020

ansibullbot added the needs_info This issue requires further information. Please answer any outstanding questions label Dec 14, 2020

ansibullbot removed the needs_info This issue requires further information. Please answer any outstanding questions label Dec 16, 2020

ansibullbot added the needs_info This issue requires further information. Please answer any outstanding questions label May 1, 2021

russoz mentioned this issue May 22, 2021

redis cache - better parsing of connection uri #2579

Merged

ansibullbot added the has_pr label May 22, 2021

felixfontein closed this as completed in #2579 May 26, 2021

This was referenced May 26, 2021

[PR #2579/4764a5de backport][stable-2] redis cache - better parsing of connection uri #2621

Merged

[PR #2579/4764a5de backport][stable-3] redis cache - better parsing of connection uri #2622

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

redis cache plugin does not accept (bracketed) IPv6 addresses #497

redis cache plugin does not accept (bracketed) IPv6 addresses #497

branleb commented Jun 11, 2020

ansibullbot commented Jun 11, 2020

mator commented Oct 23, 2020 •

edited

Loading

felixfontein commented Oct 23, 2020

felixfontein commented Oct 23, 2020

branleb commented Oct 23, 2020

mator commented Oct 23, 2020 •

edited

Loading

Akasurde commented Dec 14, 2020

branleb commented Dec 16, 2020

aminvakil commented May 1, 2021

russoz commented May 22, 2021 •

edited

Loading

redis cache plugin does not accept (bracketed) IPv6 addresses #497

redis cache plugin does not accept (bracketed) IPv6 addresses #497

Comments

branleb commented Jun 11, 2020

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ansibullbot commented Jun 11, 2020

mator commented Oct 23, 2020 • edited Loading

felixfontein commented Oct 23, 2020

felixfontein commented Oct 23, 2020

branleb commented Oct 23, 2020

mator commented Oct 23, 2020 • edited Loading

Akasurde commented Dec 14, 2020

branleb commented Dec 16, 2020

aminvakil commented May 1, 2021

russoz commented May 22, 2021 • edited Loading

mator commented Oct 23, 2020 •

edited

Loading

mator commented Oct 23, 2020 •

edited

Loading

russoz commented May 22, 2021 •

edited

Loading