[BUG] [3006] [3006.0rc2] KeyError: get_load on returner(master_job_cache) from extmods

[onmeac]

Description
master_job_cache returner provided by extmods gives keyerror on salt-syndic

> fails

cleanup

$ sudo rm -rf \
  /opt/saltstack/salt/lib/python3.10/site-packages/salt/returners/__pycache__ \
  /opt/saltstack/salt/lib/python3.10/site-packages/salt/returners/my_test_returner.py \
  /srv/salt/extmods/returners/__pycache__ \
  /srv/salt/extmods/returners/my_test_returner.py

configure extmods directory

$ sudo cat /etc/salt/master.d/extmods.conf 
extension_modules: /srv/salt/extmods

create new returner, this time in configured extmods directory

$ sudo cp \
  /opt/saltstack/salt/lib/python3.10/site-packages/salt/returners/local_cache.py \
  /srv/salt/extmods/returners/my_test_returner.py

start salt-master, salt-minion, salt-syndic

$ sudo salt test_minion test.ping
test_minion:
    True

salt-master output

[DEBUG   ] The functions from module 'my_test_returner' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded my_test_returner.prep_jid
[DEBUG   ] Sending event: tag = 20230322130646730583; data = {'minions': ['test_minion'], '_stamp': '2023-03-22T13:06:46.731832'}
[DEBUG   ] Gathering reactors for tag 20230322130646730583
[DEBUG   ] Sending event: tag = salt/job/20230322130646730583/new; data = {'jid': '20230322130646730583', 'tgt_type': 'glob', 'tgt': 'test_minion', 'user': 'sudo_testuser', 'fun': 'test.ping', 'arg': [], 'minions': ['test_minion'], 'missing': [], '_stamp': '2023-03-22T13:06:46.732868'}
[DEBUG   ] Gathering reactors for tag salt/job/20230322130646730583/new
[DEBUG   ] Adding minions for job 20230322130646730583: ['test_minion']
[INFO    ] User sudo_testuser Published command test.ping with jid 20230322130646730583
[DEBUG   ] Published command details {'fun': 'test.ping', 'arg': [], 'tgt': 'test_minion', 'jid': '20230322130646730583', 'ret': '', 'tgt_type': 'glob', 'user': 'sudo_testuser'}
[DEBUG   ] Sending payload to publish daemon. jid=20230322130646730583 load={'fun': 'test.ping', 'arg': [], 'tgt': '
[DEBUG   ] Connecting to pub server: ipc:///var/run/salt/master/publish_pull.ipc
[DEBUG   ] Sent payload to publish daemon.
[DEBUG   ] Signing data packet
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt.sign_message: Signing message.
[INFO    ] Got return from test_minion for job 20230322130646730583
[DEBUG   ] Sending event: tag = salt/job/20230322130646730583/ret/test_minion; data = {'cmd': '_return', 'id': 'test_minion', 'success': True, 'return': True, 'retcode': 0, 'jid': '20230322130646730583', 'fun': 'test.ping', 'fun_args': [], '_stamp': '2023-03-22T13:06:46.801622'}
[DEBUG   ] Gathering reactors for tag salt/job/20230322130646730583/ret/test_minion

salt-syndic output

[ERROR   ] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f3e0c35f2e0>, b'salt/job/20230322130646730583/ret/test_minion\n\n\x89\xa3cmd\xa7_return\xa2id\xaatest_minion\xa7success\xc3\xa6return\xc3\xa7retcode\x00\xa3jid\xb420230322130646730583\xa3fun\xa9test.ping\xa8fun_args\x90\xa6_stamp\xba2023-03-22T13:06:46.801622')
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 3701, in _process_event
    jdict["__load__"].update(self.mminion.returners[fstr](data["jid"]))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 336, in __getitem__
    super().__getitem__(item)  # try to get the item from the dictionary
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/lazy.py", line 105, in __getitem__
    raise KeyError(key)
KeyError: 'my_test_returner.get_load'

> works

cleanup

$ sudo rm -rf \
  /opt/saltstack/salt/lib/python3.10/site-packages/salt/returners/__pycache__ \
  /opt/saltstack/salt/lib/python3.10/site-packages/salt/returners/my_test_returner.py \
  /srv/salt/extmods/returners/__pycache__ \
  /srv/salt/extmods/returners/my_test_returner.py

create new returner

$ sudo cp \
  /opt/saltstack/salt/lib/python3.10/site-packages/salt/returners/local_cache.py \
  /opt/saltstack/salt/lib/python3.10/site-packages/salt/returners/my_test_returner.py

configure salt to use that returner for master_job_cache

$ sudo cat /etc/salt/master.d/master_job_cache.conf
master_job_cache: my_test_returner

start salt-master, salt-minion, salt-syndic

do something on a minion

$ sudo salt test_minion test.ping
test_minion:
    True

salt-master output

[DEBUG   ] Sending event: tag = 20230322130909031745; data = {'minions': ['test_minion'], '_stamp': '2023-03-22T13:09:09.032009'}
[DEBUG   ] Sending event: tag = salt/job/20230322130909031745/new; data = {'jid': '20230322130909031745', 'tgt_type': 'glob', 'tgt': 'test_minion', 'user': 'sudo_testuser', 'fun': 'test.ping', 'arg': [], 'minions': ['test_minion'], 'missing': [], '_stamp': '2023-03-22T13:09:09.032735'}
[DEBUG   ] Gathering reactors for tag 20230322130909031745
[DEBUG   ] Adding minions for job 20230322130909031745: ['test_minion']
[INFO    ] User sudo_testuser Published command test.ping with jid 20230322130909031745
[DEBUG   ] Published command details {'fun': 'test.ping', 'arg': [], 'tgt': 'test_minion', 'jid': '20230322130909031745', 'ret': '', 'tgt_type': 'glob', 'user': 'sudo_testuser'}
[DEBUG   ] Sending payload to publish daemon. jid=20230322130909031745 load={'fun': 'test.ping', 'arg': [], 'tgt': '
[DEBUG   ] Sent payload to publish daemon.
[DEBUG   ] Signing data packet
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt.sign_message: Signing message.
[DEBUG   ] Gathering reactors for tag salt/job/20230322130909031745/new
[INFO    ] Got return from test_minion for job 20230322130909031745
[DEBUG   ] Sending event: tag = salt/job/20230322130909031745/ret/test_minion; data = {'cmd': '_return', 'id': 'test_minion', 'success': True, 'return': True, 'retcode': 0, 'jid': '20230322130909031745', 'fun': 'test.ping', 'fun_args': [], '_stamp': '2023-03-22T13:09:09.106875'}
[DEBUG   ] Gathering reactors for tag salt/job/20230322130909031745/ret/test_minion

salt-syndic output

nothing, which is good.

---

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify) -- kvm
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.0rc2
 
Python Version:
        Python: 3.10.10 (main, Feb 23 2023, 05:21:34) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.7.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: centos 7 Core
        locale: utf-8
       machine: x86_64
       release: 3.10.0-1160.81.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

[garethgreenaway]

@onmeac Thanks for the report. Are you putting the customer returner under /srv/salt/extmods on the Salt master or on the Salt Syndic master? So far having the customer returner on the Salt master I haven't been able to reproduce this one.

[onmeac]

@garethgreenaway I place them in /srv/salt/extmods/returners

the salt-master salt-minion and salt-syndic processes are all running on the same server.

[garethgreenaway]

Digging into this I hadn't seen the extension_modules configuration option before, but it's purpose is to set the location where custom modules are synced to and cached either on the master or the minion. The documentation indicates that it has the same use as the modules_dir location, but this is incorrect and the extension_modules location is cleared of any files and directories that do not exist in that location when a sync is performed. The attached PR clarifies the documentation for the extension_modules option.

[onmeac]

Ok so to resolve this issue.

Use module_dirs, but configure it for both master AND minion (because salt-syndic uses minion config) so that both salt-master and salt-syndic can find the same configured returner in /srv/salt/extmods/returners

master.d/extmods.conf

module_dirs:
- /srv/salt/extmods

minion.d/extmods.conf

module_dirs:
- /srv/salt/extmods

Thank you for updating the documentation.