[xenial] Perform timeboxed upgrade attempt of SecureDrop from Ubuntu 14.04 to 16.04

[eloquence]

As part of the xenial epic ( #3207), similar to a from-scratch install of SecureDrop with Ubuntu 16.04 (#3207), we should attempt the upgrade of an existing SecureDrop instance from 14.04 to 16.04 in a non-repeatable manner, to gather information that will inform our decision about a potential base OS transition.

For the purposes of the 5/30-6/13 sprint, this task is timeboxed to approximately one working day.

[conorsch]

Performed in-place upgrade of mon-prod VM from Trusty to Xenial. Results below.

Prerequisites

The do-release-upgrade command showed no pending updates:

vagrant@mon-prod:~$ sudo do-release-upgrade
Checking for a new Ubuntu release
No new release found

Based on the Xenial release notes, it was necessary to edit /etc/update-manager/release-upgrades to change Prompt=never to Prompt=lts. Then do-release-upgrade displayed the Xenial option.

Detailed prompts

Note that the cron-apt logic automatically removes unused packages, so sticking with the default of not removing the obsolete packages is still sane.

Kicking off the upgrade

Do you want to start the upgrade?

12 installed packages are no longer supported by Canonical. You can
still get support from the community.

10 packages are going to be removed. 144 new packages are going to be
installed. 419 packages are going to be upgraded.

You have to download a total of 235 M. This download will take about
2 minutes with your connection.

Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.

Continue [yN]  Details [d] d

No longer supported: biosdevname gcc-4.8-base gcc-4.9-base 
  libarchive-extract-perl liblog-message-simple-perl 
  libmodule-pluggable-perl libpod-latex-perl libterm-ui-perl 
  libtext-soundex-perl module-init-tools python-debian w3m 

Remove: libasprintf0c2 libpython3.4-minimal libpython3.4-stdlib 
  perl-modules python3.4 python3.4-minimal systemd-services 

Remove (was auto installed) aptitude libkyotocabinet16 libxapian22 

Install: cgmanager console-setup-linux cpp-5 distro-info-data 
  dmeventd fontconfig-config fonts-dejavu-core gcc-5 gcc-5-base 
  gcc-6-base init initramfs-tools-core libapt-inst2.0 libapt-pkg5.0 
  libasan2 libasprintf0v5 libbind9-140 libboost-iostreams1.58.0 
  libcc1-0 libcilkrts5 libdns-export162 libdns162 libdrm-amdgpu1 
  libdrm-common libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 
  libfdisk1 libfontconfig1 libfontenc1 libgcc-5-dev libgcrypt20 
  libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgnutls30 
  libhogweed4 libice6 libicu55 libisc-export160 libisc160 libisccc140 
  libisccfg140 libisl15 libkyotocabinet16v5 libllvm5.0 
  liblog-message-perl liblsan0 liblvm2app2.2 liblvm2cmd2.02 
  liblwres141 liblz4-1 libmnl0 libmodule-runtime-perl libmpx0 
  libmysqlclient20 libnettle6 libparams-classify-perl libparted2 
  libpciaccess0 libperl5.22 libplymouth4 libprocps4 libpython3.5 
  libpython3.5-minimal libpython3.5-stdlib librtmp1 libsensors4 
  libsm6 libsmartcols1 libsystemd0 libtk8.6 libtxc-dxtn-s2tc0 
  libubsan0 libutempter0 libx11-6 libx11-data libx11-xcb1 
  libxapian-1.3-5 libxapian22v5 libxaw7 libxcb-dri2-0 libxcb-dri3-0 
  libxcb-glx0 libxcb-present0 libxcb-shape0 libxcb-sync1 libxcb1 
  libxcomposite1 libxdamage1 libxext6 libxfixes3 libxft2 libxi6 
  libxinerama1 libxmu6 libxmuu1 libxpm4 libxrandr2 libxrender1 
  libxshmfence1 libxss1 libxt6 libxtables11 libxtst6 libxv1 
  libxxf86dga1 libxxf86vm1 linux-base pastebinit perl-modules-5.22 
  python-attr python-cffi-backend python-cryptography python-enum34 
  python-idna python-ipaddress python-ndg-httpsclient python-pyasn1 
  python-pyasn1-modules python-service-identity python3-chardet 
  python3-debian python3-pkg-resources python3-requests python3-six 
  python3-systemd python3-urllib3 python3-xapian1.3 python3.5 
  python3.5-minimal rename s-nail systemd systemd-sysv tcl-expect 
  tcl8.6 tk8.6 update-motd x11-common x11-utils xbitmaps 
  xdg-user-dirs xterm 

SSH-related config changes

Configuration file '/etc/ssh/moduli'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** moduli (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/etc/ssh/ssh_config'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** ssh_config (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/etc/pam.d/sshd'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** sshd (Y/I/N/O/D/Z) [default=N] ? 

Interactive prompts

Obsolete packages

Remove obsolete packages? 

29 packages are going to be removed. 

 Continue [yN]  Details [d]d
Remove (was auto installed) apt-xapian-index aptitude-common cpp-4.8 
  gcc-4.8 heirloom-mailx libasan0 libboost-iostreams1.54.0 
  libboost-iostreams1.58.0 libck-connector0 libclass-accessor-perl 
  libcloog-isl4 libcwidget3 libept1.4.12 libgcc-4.8-dev libgssglue1 
  libisl10 libmysqlclient18 libparse-debianchangelog-perl 
  libsigc++-2.0-0c2a libsub-name-perl libtimedate-perl 
  libxapian-1.3-5 libxapian22v5 python-ndg-httpsclient 
  python-requests python-urllib3 python-xapian python3-xapian1.3 
  watershed 

Final reboot

System upgrade is complete.

Restart required 

To finish the upgrade, a restart is required. 
If you select 'y' the system will be restarted. 

Continue [yN] y

Then, after a reboot:

vagrant@mon-prod:~$ uname -a
Linux mon-prod 4.4.115-grsec #1 SMP Mon Feb 5 15:01:26 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
vagrant@mon-prod:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

Fixes

Had problems with DNS resolution via apt post-upgrade. Updated the firewall rules to authorize the _apt user, rather than root, to call out over HTTP/S ports. This resolved the problem, and sudo apt-get update worked well again. There may be additional DNS/NTP-related troubleshooting required, given the change to systemd-resolved.

Summary

Throughout the install, default choices were accepted as sufficient, with the following exceptions:

1. The grub partition selection required manual selection of the appropriate partition. The choices displayed above are VM-specific, so we'll need to evaluate the choices presented by the upgrader on hardware.
2. The service bounce disconnected tor, breaking the SSH connection, so we should consider not restarting tor as part of the openssl changes. Reconnecting over SSH in a new terminal window resumed the upgrade operation, after tor bounce, given the tmux attach logic.
3. The final reboot prompt defaults to "no", so overriding and choosing "yes" is required for immediate reboot.

Additionally, firewall tweaks were required. If we move to managing system state in an unattended fashion, e.g. via ansible-pull as described in #3136, we can ensure the firewall rules are correctly set. We should critically evaluate the uid restrictions, given the complications for smooth upgrades.

Will proceed with in-place upgrade testing of App VMs, and report back.

[eloquence]

Since this was a timeboxed research task, if there are no objections, I suggest adding any final notes and closing this task before the beginning of the next sprint.

[conorsch]

Upgrade testing on app-staging is going smoothly. Here's the WIP branch, showing the diff: https://github.com/freedomofpress/securedrop/compare/test-xenial-upgrade-path

The testing procedure is as follows:

1. Checkout develop branch.
2. make build-debs && vagrant provision /stag/
3. Check out test-xenial-upgrade-path branch.
4. make build-debs && ANSIBLE_ARGS="--tags rebuild" vagrant provision /stag/
5. vagrant ssh app-staging
6. sudo do-release-upgrade Follow prompts, accept defaults (see notes above for screenshots).
7. Reboot

Voila!

vagrant@app-staging:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial
vagrant@app-staging:~$ uname -r
4.4.115-grsec
vagrant@app-staging:~$

All in all, an attended upgrade from Trusty -> Xenial seems feasible. Still worth considering the approach for more control over system state described in #3136, but so far the relatively small number of changes can be managed via postinst if necessary.

[conorsch]

Additions from notes that may be relevant down the road:

• haveged apparmor profile may not be necessary, that was generated from aa-logprof. We should try removing it to see if it's really necessary.
• The PAM common-auth customizations include declarations for pam_ecryptfs.so; I had to comment that line out. Did not investigate thoroughly.

That info should be enough to recreate the work above if we decide to move forward with a Trusty to Xenial migration. See also the Ansible logic changes, of course, in #3207.

[kushaldas]

I got this on the app server.

[kushaldas]

Then this one, I chose No

[kushaldas]