Update ansible to 2.9.7

[rmol]

Status

Ready for review

Description of Changes

Updates Ansible.

Fixes #5144.

Testing

Ideally, make clean followed by make venv && make build-debs && make staging, then molecule verify -s libvirt-staging-xenial (or whatever your staging scenario is).

Even more ideally, run through a production install on VMs or hardware, using this branch.

Deployment

It will affect admin workstations and development environments, but should not affect the content of deployed instances.

Checklist

If you made changes to the server application code:

• Linting (make lint) and tests (make test) pass in the development container

If you made changes to securedrop-admin:

• Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

• Configuration tests pass

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

If you added or updated a code dependency:

Choose one of the following:

• I have performed a diff review and pasted the contents to the packaging wiki
• I would like someone else to do the diff review

I've done a timeboxed and therefore limited review of the differences between Ansible 2.7.13 and 2.9.7 (hashes of the tarballs I examined are in the packaging wiki) and did not spot any problems in changes to the core functionality. I also ran Bandit checks for high-severity problems on both versions, and found no new problems reported in 2.9.7.

[emkll]

Thanks @rmol, tested as follows:

dev/staging environments

• install dev-requirements
• make build-debs
• make staging
• make upgrade-start
• make upgrade-test-local

Prod environments

• ./securedrop-admin setup
• ./securedrop-admin install
• ❗ ./securedrop-admin install: Subsequent install fails, a |changed was not caught in install_files/ansible-base/tasks/transistion_ssh_local.yml on lines 27 and 35. Modifying to .changed like you do in other tasks resolved (see error output in 1 below)
• ./securedrop-admin logs
• ./securedrop-admin backup
• ./securedrop-admin restore
• ./securedrop-admin tailsconfig

A couple of observations, I think we should address 1 and 2 prior to merge, what do you think?:

1. When running a second install on the server, the install fails with the following message:

TASK [Force a reboot conditionally, when tor_over_ssh status changed] **************************************************************************************************
fatal: [app]: FAILED! => {"msg": "The conditional check 'aths_deletion_results|changed' failed. The error was: template error while templating string: no filter named 'changed'. String: {% if aths_deletion_results|changed %} True {% else %} False {% endif %}\n\nThe error appears to be in '/home/amnesia/Persistent/securedrop/install_files/ansible-base/tasks/transistion_ssh_local.yml': line 23, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Force a reboot conditionally, when tor_over_ssh status changed\n  ^ here\n"}
fatal: [mon]: FAILED! => {"msg": "The conditional check 'aths_deletion_results|changed' failed. The error was: template error while templating string: no filter named 'changed'. String: {% if aths_deletion_results|changed %} True {% else %} False {% endif %}\n\nThe error appears to be in '/home/amnesia/Persistent/securedrop/install_files/ansible-base/tasks/transistion_ssh_local.yml': line 23, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Force a reboot conditionally, when tor_over_ssh status changed\n  ^ here\n"}

2. It appears like 2.9.7 was released as this update was being developed/tested, and there are CVEs associated . It's the win_unzip module which we don't use, but I'm guessing GitHub will soon warn us about the vulnerability [1]. I think it might make sense to update to 2.9.7 to avoid having to do it again in the future.
3. I am observing deprecation warnings during build but also install phase (example is from builder):

[DEPRECATION WARNING]: Distribution Ubuntu 16.04 on host xenial-sd-generic-
ossec-agent2 should use /usr/bin/python3, but is using /usr/bin/python for 
backward compatibility with prior Ansible releases. A future Ansible release 
will default to using the discovered platform python for this host. See https:/
/docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html 
for more information. This feature will be removed in version 2.12. Deprecation
 warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

Based on the link in that message, we don't set the interpreter to Python3. My understanding is that it will use /usr/bin/python which is 2.7.12. it is getting upstream security updates, but might be worth tracking as we eventually target other platforms. Perhaps we can track this is a follow-up?

[1] : https://nvd.nist.gov/vuln/detail/CVE-2020-1737

[rmol]

Agreed on all points. I'm addressing 1 and 2 right now, and think we might as well solve 3 by setting ansible_python_interpreter in ansible.cfg. @conorsch what do you think?

[conorsch]

we might as well solve 3 by setting ansible_python_interpreter in ansible.cfg

Sounds great, the commit you've added doing that looks like just what we need. Given that @emkll has already verified py3 support in prod env, will take a closer look at the upgrade & qubes-staging scenarios, make sure we're covered there.

[emkll]

Given that @emkll has already verified py3 support in prod env, will take a closer look at the upgrade & qubes-staging scenarios, make sure we're covered there.

To be clear, verified that python3 was installed on a fresh Ubuntu 16.04.6 target as follows:

• In libvirt VMs, installed Ubuntu 16.04.06 from the ISO, not from vagrant boxes). Selected standard system utilities and openssh server
• /usr/bin/python3, /usr/bin/python3.5 is installed
• Ran ./securedrop-admin install on this branch, against said VMs, and observed a successful install on those one-off VMs.

[rmol]

OK, I'm still having trouble with Molecule. When I try make upgrade-start I get ERROR! The requested handler 'restart tor' was not found in either the main handlers list nor in the listening handlers list.

I've tried adding listen: restart tor here because "Handlers using import* will not be triggered when notified by their name, as importing overwrites the handler’s named task with the imported task list." (and the classic way of specifying roles in a playbook, as used here, if I'm understanding correctly is equivalent to import). Same error.

I converted the roles: block to a tasks: block containing import_role or include_role. Same error.

I cannot get this playbook to actually fire the handler in Ansible 2.8.11 or 2.9.7.

Any ideas would be incredibly welcome.

[conorsch]

Appended two commits. The first fixes the OSSEC package build by installing python3-requests, rather than python-requests, which was required due to interpreter changes.

The second fixes the "upgrade" scenario.

When I try make upgrade-start I get ERROR! The requested handler 'restart tor' was not found in either the main handlers list nor in the listening handlers list.

The problem was that the "upgrade" scenario uses the old logic from the previous stable release of SecureDrop, cloning it into a separate subdir, which is then used to provision the servers. Since we jumped two minor versions of Ansible, we lost mutual compatibility between the two, so even though the handler task was happy as of 2.9, the combination of 1) a separate subdir with relative lookups for handlers and 2) the old-style "include" running under Ansible 2.9 was leading to the repeated failure to find the handler.

[conorsch]

Great work grinding through a complicated version upgrade, @rmol. I'm satisfied with the changes here, but I haven't run through the Tails/prod portions yet. The commits I appended are minimal, but I'll leave to you & @emkll to merge once you're both satisfied.

[rmol]

@conorsch Works great; the upgrade scenario is now fine. I'm just working through the rest as a final check, but I think you've done it. Thanks!

[emkll]

Thanks @rmol and @conorsch, changes look good to merge when CI passes, from my perspective. Approving this now since my review is blocking, @rmol feel free to merge whenever you've finished your review of the changes.

• diff review and hash match
• changes to ansible version check look good

dev/staging environments

• install dev-requirements
• make build-debs
• make staging
• make upgrade-start
• make upgrade-test-local

Prod environments

• ./securedrop-admin setup
• ./securedrop-admin install
• ./securedrop-admin install: Subsequent install succeeds, also changed ssh_over_tor
• ./securedrop-admin logs
• ./securedrop-admin backup
• ./securedrop-admin restore
• ./securedrop-admin tailsconfig

[rmol]

I've worked through the rest of the tests @emkll listed, and everything looks good. Merging.