Fix focal testinfra test failures

[zenmonkeykstop]

Status

Ready for review

Description of Changes

Fixes #5825 .

• loads per-OS apparmor profile details
• uses host.check_output('hostname -s') instead of ansible backend to get hostname, as ansible backend not used in prod verify.

Testing

• set up a prod Focal system on latest 1.8.0 RC (or 1.8.0 if released).

• On the Admin Workstation, switch to this branch.

• In a terminal, run USE_FOCAL=True ./securedrop-admin verify

  • verify that tests pass (with the exception of test_fpf_apt_repo if the apt-test repo was used for the install)

• on a dev workstation, set up a staging environment and run tests with make build-debs-notest && make staging && make testinfra

  • verify that tests pass.

• on a dev workstation, set up a focal staging environment and run tests with make build-debs-focal-notest && make staging-focal && USE_FOCAL=True make testinfra

  • verify that tests pass.

Deployment

test-only.

Checklist

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

[emkll]

Still getting some unexpected failures on my install-and-restore rc1 mac mini (other than sources.list) which is expected. Can anyone reproduce this?

==================================================================================================== FAILURES =====================================================================================================
______________________________________________________________________________ test_apparmor_enforced[paramiko://app-/usr/sbin/ntpd] ______________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>, aa_enforced = '/usr/sbin/ntpd'

    @pytest.mark.parametrize('aa_enforced', sdvars.apparmor_enforce_actual)
    def test_apparmor_enforced(host, aa_enforced):
        awk = ("awk '/[0-9]+ profiles.*enforce./"
               "{flag=1;next}/^[0-9]+.*/{flag=0}flag'")
        with host.sudo():
            c = host.check_output("aa-status | {}".format(awk))
>           assert aa_enforced in c
E           AssertionError: assert '/usr/sbin/ntpd' in '   /usr/bin/man\n   /usr/lib/NetworkManager/nm-dhcp-client.action\n   /usr/lib/NetworkManager/nm-dhcp-helper\n   /usr...bin/dhclient\n   lsb_release\n   man_filter\n   man_groff\n   nvidia_modprobe\n   nvidia_modprobe//kmod\n   system_tor'

app/test_apparmor.py:97: AssertionError
________________________________________________________________________________ test_securedrop_rqrequeue_service[paramiko://app] ________________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>

    def test_securedrop_rqrequeue_service(host):
        """
        Verify configuration of securedrop_rqrequeue systemd service.
        """
        service_file = "/lib/systemd/system/securedrop_rqrequeue.service"
        expected_content = "\n".join([
            "[Unit]",
            "Description=SecureDrop rqrequeue process",
            "After=redis-server.service",
            "Wants=redis-server.service",
            "",
            "[Service]",
            'Environment=PYTHONPATH="{}:{}"'.format(
                securedrop_test_vars.securedrop_code, securedrop_test_vars.securedrop_venv_site_packages
            ),
            "ExecStart={}/python /var/www/securedrop/scripts/rqrequeue --interval 60".format(
                securedrop_test_vars.securedrop_venv_bin
            ),
            "PrivateDevices=yes",
            "PrivateTmp=yes",
            "ProtectSystem=full",
            "ReadOnlyDirectories=/",
            "ReadWriteDirectories={}".format(securedrop_test_vars.securedrop_data),
            "Restart=always",
            "RestartSec=10s",
            "UMask=077",
            "User={}".format(securedrop_test_vars.securedrop_user),
            "WorkingDirectory={}".format(securedrop_test_vars.securedrop_code),
            "",
            "[Install]",
            "WantedBy=multi-user.target\n",
        ])
    
        f = host.file(service_file)
        assert f.is_file
        assert f.mode == 0o644
        assert f.user == "root"
        assert f.group == "root"
>       assert f.content_string == expected_content
E       assert '[Unit]\nDesc...user.target\n' == '[Unit]\nDesc...user.target\n'
E           [Unit]
E           Description=SecureDrop rqrequeue process
E           After=redis-server.service
E           Wants=redis-server.service
E           
E           [Service]
E         - Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"...
E         
E         ...Full output truncated (18 lines hidden), use '-vv' to show

app-code/test_securedrop_rqrequeue.py:45: AssertionError
________________________________________________________________________________ test_securedrop_rqworker_service[paramiko://app] _________________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>

    def test_securedrop_rqworker_service(host):
        """
        Verify configuration of securedrop_rqworker systemd service.
        """
        securedrop_test_vars = sdvars
        service_file = "/lib/systemd/system/securedrop_rqworker.service"
    
        expected_content = "\n".join([
            "[Unit]",
            "Description=SecureDrop rq worker",
            "After=redis-server.service",
            "Wants=redis-server.service",
            "",
            "[Service]",
            'Environment=PYTHONPATH="{}:{}"'.format(
                securedrop_test_vars.securedrop_code, securedrop_test_vars.securedrop_venv_site_packages
            ),
            "ExecStart={}/rqworker".format(securedrop_test_vars.securedrop_venv_bin),
            "PrivateDevices=yes",
            "PrivateTmp=yes",
            "ProtectSystem=full",
            "ReadOnlyDirectories=/",
            "ReadWriteDirectories={}".format(securedrop_test_vars.securedrop_data),
            "Restart=always",
            "RestartSec=10s",
            "UMask=077",
            "User={}".format(securedrop_test_vars.securedrop_user),
            "WorkingDirectory={}".format(securedrop_test_vars.securedrop_code),
            "",
            "[Install]",
            "WantedBy=multi-user.target\n",
        ])
    
        f = host.file(service_file)
        assert f.is_file
        assert f.mode == 0o644
        assert f.user == "root"
        assert f.group == "root"
>       assert f.content_string == expected_content
E       assert '[Unit]\nDesc...user.target\n' == '[Unit]\nDesc...user.target\n'
E           [Unit]
E           Description=SecureDrop rq worker
E           After=redis-server.service
E           Wants=redis-server.service
E           
E           [Service]
E         - Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"...
E         
E         ...Full output truncated (18 lines hidden), use '-vv' to show

app-code/test_securedrop_rqworker.py:45: AssertionError
________________________________________________________________________________ test_securedrop_shredder_service[paramiko://app] _________________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>

    def test_securedrop_shredder_service(host):
        """
        Verify configuration of securedrop_shredder systemd service.
        """
        securedrop_test_vars = sdvars
        service_file = "/lib/systemd/system/securedrop_shredder.service"
        expected_content = "\n".join([
            "[Unit]",
            "Description=SecureDrop shredder",
            "",
            "[Service]",
            'Environment=PYTHONPATH="{}:{}"'.format(
                securedrop_test_vars.securedrop_code, securedrop_test_vars.securedrop_venv_site_packages
            ),
            "ExecStart={}/python /var/www/securedrop/scripts/shredder --interval 60".format(
                securedrop_test_vars.securedrop_venv_bin
            ),
            "PrivateDevices=yes",
            "PrivateTmp=yes",
            "ProtectSystem=full",
            "ReadOnlyDirectories=/",
            "ReadWriteDirectories={}".format(securedrop_test_vars.securedrop_data),
            "Restart=always",
            "RestartSec=10s",
            "UMask=077",
            "User={}".format(securedrop_test_vars.securedrop_user),
            "WorkingDirectory={}".format(securedrop_test_vars.securedrop_code),
            "",
            "[Install]",
            "WantedBy=multi-user.target\n",
        ])
    
        f = host.file(service_file)
        assert f.is_file
        assert f.mode == 0o644
        assert f.user == "root"
        assert f.group == "root"
>       assert f.content_string == expected_content
E       assert '[Unit]\nDesc...user.target\n' == '[Unit]\nDesc...user.target\n'
E           [Unit]
E           Description=SecureDrop shredder
E           
E           [Service]
E         - Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"
E         ?                                                                                        ^
E         + Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.8/site-packages"...
E         
E         ...Full output truncated (16 lines hidden), use '-vv' to show

app-code/test_securedrop_shredder_configuration.py:44: AssertionError
_____________________________________________________________________________ test_securedrop_source_deleter_service[paramiko://app] ______________________________________________________________________________
[gw1] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>

    def test_securedrop_source_deleter_service(host):
        """
        Verify configuration of securedrop_source_deleter systemd service.
        """
        service_file = "/lib/systemd/system/securedrop_source_deleter.service"
        expected_content = "\n".join([
            "[Unit]",
            "Description=SecureDrop Source deleter",
            "",
            "[Service]",
            'Environment=PYTHONPATH="{}:{}"'.format(
                securedrop_test_vars.securedrop_code, securedrop_test_vars.securedrop_venv_site_packages
            ),
            "ExecStart={}/python /var/www/securedrop/scripts/source_deleter --interval 10".format(
                securedrop_test_vars.securedrop_venv_bin
            ),
            "PrivateDevices=yes",
            "PrivateTmp=yes",
            "ProtectSystem=full",
            "ReadOnlyDirectories=/",
            "ReadWriteDirectories={}".format(securedrop_test_vars.securedrop_data),
            "Restart=always",
            "RestartSec=10s",
            "UMask=077",
            "User={}".format(securedrop_test_vars.securedrop_user),
            "WorkingDirectory={}".format(securedrop_test_vars.securedrop_code),
            "",
            "[Install]",
            "WantedBy=multi-user.target\n",
        ])
    
        f = host.file(service_file)
        assert f.is_file
        assert f.mode == 0o644
        assert f.user == "root"
        assert f.group == "root"
>       assert f.content_string == expected_content
E       assert '[Unit]\nDesc...user.target\n' == '[Unit]\nDesc...user.target\n'
E           [Unit]
E           Description=SecureDrop Source deleter
E           
E           [Service]
E         - Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"
E         ?                                                                                        ^
E         + Environment=PYTHONPATH="/var/www/securedrop:/opt/venvs/securedrop-app-code/lib/python3.8/site-packages"...
E         
E         ...Full output truncated (16 lines hidden), use '-vv' to show

app-code/test_securedrop_source_deleter_configuration.py:43: AssertionError
____________________________________________________________________ test_securedrop_application_apt_dependencies[paramiko://app-libpython3.5] ____________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/amnesia/Persistent/securedrop/admin/.venv3/bin/python3

host = <testinfra.host.Host paramiko://app>, package = 'libpython3.5'

    @pytest.mark.parametrize('package', [
        'apache2',
        'apparmor-utils',
        'coreutils',
        'gnupg2',
        'haveged',
        'libapache2-mod-xsendfile',
        'libpython{}'.format(python_version),
        'paxctld',
        'python3',
        'redis-server',
        'securedrop-config',
        'securedrop-keyring',
        'sqlite3',
    ])
    def test_securedrop_application_apt_dependencies(host, package):
        """
        Ensure apt dependencies required to install `securedrop-app-code`
        are present. These should be pulled in automatically via apt,
        due to specification in Depends in package control file.
        """
>       assert host.package(package).is_installed
E       AssertionError: assert False
E        +  where False = <package libpython3.5>.is_installed
E        +    where <package libpython3.5> = <class 'testinfra.modules.base.DebianPackage'>('libpython3.5')
E        +      where <class 'testinfra.modules.base.DebianPackage'> = <testinfra.host.Host paramiko://app>.package

app-code/test_securedrop_app_code.py:41: AssertionError

[rmol]

I did not get those errors on a clean Focal install on NUC7+NUC10. I only saw the expected failures due to the apt-test adjustments. Other than that everything looked good.

Focal staging passed, no errors.

Xenial staging somehow had unattended-upgrades installed, and tcpdump not installed, so missing from the aa-status check.

testinfra, Xenial staging

    =================================== FAILURES ===================================
    _______ test_apparmor_enforced[ansible://app-staging-/usr/sbin/tcpdump] ________
    [gw1] linux -- Python 3.7.3 /home/user/src/fpf/sd/develop/.venv/bin/python3

    host = <testinfra.host.Host ansible://app-staging>
    aa_enforced = '/usr/sbin/tcpdump'

        @pytest.mark.parametrize('aa_enforced', sdvars.apparmor_enforce)
        def test_apparmor_enforced(host, aa_enforced):
            awk = ("awk '/[0-9]+ profiles.*enforce./"
                   "{flag=1;next}/^[0-9]+.*/{flag=0}flag'")
            with host.sudo():
                c = host.check_output("aa-status | {}".format(awk))
    >           assert aa_enforced in c
    E           AssertionError: assert '/usr/sbin/tcpdump' in '   /sbin/dhclient\n   /usr/bin/lxc-start\n   /usr/lib/NetworkManager/nm-dhcp-client.action\n   /usr/lib/NetworkManage...c-container-default-cgns\n   lxc-container-default-with-mounting\n   lxc-container-default-with-nesting\n   system_tor'

    ../testinfra/app/test_apparmor.py:97: AssertionError
    ____________ test_unattended_upgrades_config[ansible://app-staging] ____________
    [gw1] linux -- Python 3.7.3 /home/user/src/fpf/sd/develop/.venv/bin/python3

    host = <testinfra.host.Host ansible://app-staging>

        def test_unattended_upgrades_config(host):
            """
            Ensures the 50unattended-upgrades config is correct only under Ubuntu Focal
            """
            f = host.file('/etc/apt/apt.conf.d/50unattended-upgrades')
            if host.system_info.codename == "xenial":
    >           assert not f.exists
    E           assert not True
    E            +  where True = <file /etc/apt/apt.conf.d/50unattended-upgrades>.exists

    ../testinfra/common/test_automatic_updates.py:196: AssertionError
    ____________ test_unattended_upgrades_config[ansible://mon-staging] ____________
    [gw2] linux -- Python 3.7.3 /home/user/src/fpf/sd/develop/.venv/bin/python3

    host = <testinfra.host.Host ansible://mon-staging>

        def test_unattended_upgrades_config(host):
            """
            Ensures the 50unattended-upgrades config is correct only under Ubuntu Focal
            """
            f = host.file('/etc/apt/apt.conf.d/50unattended-upgrades')
            if host.system_info.codename == "xenial":
    >           assert not f.exists
    E           assert not True
    E            +  where True = <file /etc/apt/apt.conf.d/50unattended-upgrades>.exists

    ../testinfra/common/test_automatic_updates.py:196: AssertionError
    =============================== warnings summary ===============================
    ../../.venv/lib/python3.7/site-packages/jinja2/utils.py:485
    ../../.venv/lib/python3.7/site-packages/jinja2/utils.py:485
    ../../.venv/lib/python3.7/site-packages/jinja2/utils.py:485
    ../../.venv/lib/python3.7/site-packages/jinja2/utils.py:485
      /home/user/src/fpf/sd/develop/.venv/lib/python3.7/site-packages/jinja2/utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
        from collections import MutableMapping

    ../../.venv/lib/python3.7/site-packages/jinja2/runtime.py:318
    ../../.venv/lib/python3.7/site-packages/jinja2/runtime.py:318
    ../../.venv/lib/python3.7/site-packages/jinja2/runtime.py:318
    ../../.venv/lib/python3.7/site-packages/jinja2/runtime.py:318
      /home/user/src/fpf/sd/develop/.venv/lib/python3.7/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
        from collections import Mapping

    -- Docs: https://docs.pytest.org/en/stable/warnings.html
    =========================== short test summary info ============================
    FAILED ../testinfra/app/test_apparmor.py::test_apparmor_enforced[ansible:/app-staging-/usr/sbin/tcpdump]
    FAILED ../testinfra/common/test_automatic_updates.py::test_unattended_upgrades_config[ansible:/app-staging]
    FAILED ../testinfra/common/test_automatic_updates.py::test_unattended_upgrades_config[ansible:/mon-staging]
    = 3 failed, 527 passed, 13 skipped, 3 xfailed, 1 xpassed, 8 warnings in 59.64s =
make: *** [Makefile:201: testinfra] Error 1

[rmol]

Turns out Xenial has always had unattended-upgrades installed, but now we have a test that believes otherwise.

[conorsch]

Judging by the diff, the python version output that @emkll should still be happening. Test against prod VMs via Tails VM now.

[conorsch]

Tested on Focal prod VMs, output looks good.