salt-minion broken after upgrade to 3000

[githubcdr]

Description of Issue

Salt-minion is not functioning as expected after upgrading to Salt 3000

Setup

apt source:
deb http://repo.saltstack.com/py3/ubuntu/18.04/amd64/latest bionic main

Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic

Steps to Reproduce Issue

run salt-call test.version

Versions Report

/e/a/sources.list.d # ❯❯❯ salt-minion --versions
Salt Version:
Salt: 3000

Dependency Versions:
cffi: 1.12.3
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: 4.0.2
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.9 (default, Nov 7 2019, 10:44:02)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 16.0.2
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5

System Versions:
dist: Ubuntu 18.04 bionic
locale: ANSI_X3.4-1968
machine: x86_64
release: 4.15.0-72-generic
system: Linux
version: Ubuntu 18.04 bionic

/e/a/sources.list.d # ❯❯❯ salt-call test.version
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3000', 'console_scripts', 'salt-call')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 445, in salt_call
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/call.py", line 47, in run
    caller = salt.cli.caller.Caller.factory(self.config)
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 63, in factory
    return ZeroMQCaller(opts, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 312, in __init__
    super(ZeroMQCaller, self).__init__(opts)
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 87, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 868, in __init__
    salt.utils.yaml.safe_dump(cache_top, fp_)
  File "/usr/lib/python3/dist-packages/salt/utils/yamldumper.py", line 124, in safe_dump
    return yaml.dump(data, stream, Dumper=SafeOrderedDumper, **kwargs)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 200, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 188, in dump_all
    dumper.represent(data)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 27, in represent
    self.serialize(node)
  File "ext/_yaml.pyx", line 1350, in _yaml.CEmitter.serialize (ext/_yaml.c:19880)
  File "ext/_yaml.pyx", line 1525, in _yaml.output_handler (ext/_yaml.c:22762)
TypeError: a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3000', 'console_scripts', 'salt-call')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 445, in salt_call
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/call.py", line 47, in run
    caller = salt.cli.caller.Caller.factory(self.config)
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 63, in factory
    return ZeroMQCaller(opts, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 312, in __init__
    super(ZeroMQCaller, self).__init__(opts)
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 87, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 868, in __init__
    salt.utils.yaml.safe_dump(cache_top, fp_)
  File "/usr/lib/python3/dist-packages/salt/utils/yamldumper.py", line 124, in safe_dump
    return yaml.dump(data, stream, Dumper=SafeOrderedDumper, **kwargs)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 200, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 188, in dump_all
    dumper.represent(data)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 27, in represent
    self.serialize(node)
  File "ext/_yaml.pyx", line 1350, in _yaml.CEmitter.serialize (ext/_yaml.c:19880)
  File "ext/_yaml.pyx", line 1525, in _yaml.output_handler (ext/_yaml.c:22762)
TypeError: a bytes-like object is required, not 'str'

All dependencies are provided upstream.

[DmitryKuzmenko]

Reproduced on python 3.7. The issue is actual for all the currently supported salt branches.
Steps:

• Use Python 3.
• Set minion_pillar_cache: True in minion config file and ensure file_client is either unset or set to remote.
• Start salt-master.
• Run salt-call test.version.
  The reason is SMinion.__init__ tries to yaml-dump a dictionary with string data to a cache file opened in a binary mode.

[OrangeDog]

Looks like #51854

[dwoz]

Yes, looks like we need to port this to the master branch #52195

[max-arnold]

A quick & dirty fix:

patches/pillar_cache.sls:

patch:
  pkg.installed

pillar_cache_patch:
  file.patch:
    - name: '{{ grains.saltpath }}'
    - source: salt://patches/files/pillar-cache.diff
    - strip: 2
    - require:
        - pkg: patch

restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: pillar_cache_patch

patches/files/pillar-cache.diff:

diff --git a/salt/minion.py b/salt/minion.py
index 6a77d90185..10d727aebe 100644
--- a/salt/minion.py
+++ b/salt/minion.py
@@ -27,6 +27,7 @@ from salt.ext import six
 from salt._compat import ipaddress
 from salt.utils.network import parse_host_port
 from salt.ext.six.moves import range
+from salt.template import SLS_ENCODING
 from salt.utils.zeromq import zmq, ZMQDefaultLoop, install_zmq, ZMQ_VERSION_INFO
 import salt.transport.client
 import salt.defaults.exitcodes
@@ -865,11 +866,11 @@ class SMinion(MinionBase):
                 penv = 'base'
             cache_top = {penv: {self.opts['id']: ['cache']}}
             with salt.utils.files.fopen(ptop, 'wb') as fp_:
-                salt.utils.yaml.safe_dump(cache_top, fp_)
+                salt.utils.yaml.safe_dump(cache_top, fp_, encoding=SLS_ENCODING)
                 os.chmod(ptop, 0o600)
             cache_sls = os.path.join(pdir, 'cache.sls')
             with salt.utils.files.fopen(cache_sls, 'wb') as fp_:
-                salt.utils.yaml.safe_dump(self.opts['pillar'], fp_)
+                salt.utils.yaml.safe_dump(self.opts['pillar'], fp_, encoding=SLS_ENCODING)
                 os.chmod(cache_sls, 0o600)
 
 

Run sudo salt MINION state.apply patches.pillar_cache to apply the fix.