Focal channel in apt-test isn't serving all packages

[conorsch]

First noticed by @creviera yesterday. During merge of the 1.7.0~rc3 packages (#84), viewing the packages over HTTPS only didn't reflect the dual-distro support first introduced back in #57. In order to evaluate behavior, I used an apt client in a Xenial & Focal container image, with the apt-test repo configured: https://gist.github.com/conorsch/136688c28da49de9cf5c44e86f96d5e9

The same problem can be viewed in a Focal-based staging VM:

root@sd-staging-app:~# grep ^Package /var/lib/apt/lists/apt-test.freedom.press*_Packages
Package: linux-headers-4.14.188-grsec-securedrop
Package: linux-image-4.14.175-grsec-securedrop
Package: linux-image-4.14.188-grsec-securedrop
Package: securedrop-grsec
Package: tor
Package: tor-geoipdb

Notably missing from that list crucial packages such as securedrop-app-code, for which we have a Focal-specific version string. Why are they missing? Let's figure that out. N.B. the CI tests for Focal still pass because the packages are installed from local sources:

root@sd-staging-app:~# apt-cache policy securedrop-app-code
securedrop-app-code:
  Installed: 1.7.0~rc3+focal
  Candidate: 1.7.0~rc3+focal
  Version table:
 *** 1.7.0~rc3+focal 100
        100 /var/lib/dpkg/status
root@sd-staging-app:~# apt-cache policy securedrop-grsec
securedrop-grsec:
  Installed: 4.14.188+focal
  Candidate: 4.14.188+focal
  Version table:
 *** 4.14.188+focal 100
        100 /var/lib/dpkg/status
     4.14.188 500
        500 https://apt-test.freedom.press focal/main amd64 Packages

[conorsch]

In the context of the 1.7.0 (freedomofpress/securedrop#5689), this isn't a release blocker, since it only affects Focal. Still, given the priority of the overall Focal migration (freedomofpress/securedrop#4768), I'd like to identify a fix soon.

[rmol]

Here are my notes so far from investigating this with @conorsch (please correct if I've gotten something wrong, Conor).

reprepro errors out when it runs into the second occurrence of one of our packages that don't have the distribution in the version, because it already has ossec-agent or whatever and the checksums differ:

ERROR: '/home/user/src/fpf/securedrop-dev-packages-lfs/core/xenial/ossec-agent-3.6.0-amd64.deb' cannot be included as 'pool/main/o/ossec.net/ossec-agent_3.6.0_amd64.deb'.
Already existing files can only be included again, if they are the same, but:
md5 expected: 6a6b30d8df2a8ca0941f344db3eebe0b, got: c05b5bd21fa69c9413b72ecd2f19fa82
sha1 expected: 2664c746c4d7dc07ba6d3acc3c1a4bf602c47345, got: ba507181285220f2b33eb00a547558ce45fb197b
sha256 expected: ece5c3e7b62cc714dcb09ec8471900836250aca9aa21839c11123d29946eccc8, got: fd09e231d228468fe968a949ab2675f768c8fbff73fa388d38afd5889b57e637
size expected: 289072, got: 283854
There have been errors!

The right fix is to include the distribution in all of our Version strings in the Debian control files. That way each distribution can have its own set of files. We can start this with Focal, there's little use in changing the Xenial packages at this point.

We're going to hold off on this until we're done with the 1.7.0 release. Hopefully we'll have the permanent fix in by the middle of next week. For Focal hardware testing before then (@emkll), we can dump the packages in a directory on the servers and point to that in the FPF apt source.

Here's the process I'm using:

• clone the securedrop-dev-packages-lfs repo

• create a Focal-only apt repository from the Focal packages in securedrop-dev-packages-lfs:

  • reprepro --basedir ~/src/fpf/securedrop-dev-packages-lfs/repo --outdir /tmp/reprepro/ --dbdir /tmp/repreprodb includedeb focal ~/src/fpf/securedrop-dev-packages-lfs/core/focal/*.deb

• zip up the result for transfer to the app/mon servers:

  • cd /tmp && zip -r reprepro.zip reprepro

• get reprepro.zip onto your admin workstation -- onionshare, sneakernet, whatever -- the unzip it and rsync it to the servers, e.g.: rsync -av reprepro/ [email protected]:reprepro/. Get on the servers, become root, mv ~/reprepro /reprepro && chown -R root:root /reprepro.

• on your admin workstation, change the apt source to the local directory, assuring apt it's OK that it's unsigned:

  • sed -i "s@https://apt.freedom.press@file:/reprepro@" install_files/ansible-base/roles/install-fpf-repo/defaults/main.yml
  • sed -i "s@\[arch=amd64\]@\[trusted=yes arch=amd64\]@" install_files/ansible-base/roles/install-fpf-repo/tasks/main.yml

• securedrop-admin install

Installation works, and at least the source interface is working. Have not tested more comprehensively than that. The main problem I've seen so far is that booting is extremely slow, on the order of 8-10 minutes, after we touch the machines. Focal's OK immediately after installation, so it's something we're doing. Shutting down is fine, it's coming back up that's the problem.The first installation can time out at reboots because of this. Haven't diagnosed yet, probably grsec or apparmor.

[sssoleileraaa]

I ran through the steps to create a local apt repo (right now i'm testing on xenial since we're also making changes to our xenial packages) and everything worked wonderfully until I got to this step:

• securedrop-admin install

The installation did not work. I see the error fatal: [mon]: FAILED! => {"changed": false, "msg": "apt cache update failed"} and same error for app. When I run sudo apt update directly on the app server, it works fine. So then I tried updating the apt list by hand, chaning apt-qa -> reprepro and adding trusted=yes but then I see the error:

E: Failed to fetch file:/reprepro/dists/xenial/main/binary-amd64/Packages File not found and that's because it is missing, so I'll have to look into this tomorrow - perhaps something went wrong during the reprepro step - the directory is 23.6 MB and I didn't see any errors so I figured it was successful.

Installation works, and at least the source interface is working. Have not tested more comprehensively than that. The main problem I've seen so far is that booting is extremely slow, on the order of 8-10 minutes, after we touch the machines. Focal's OK immediately after installation, so it's something we're doing. Shutting down is fine, it's coming back up that's the problem.The first installation can time out at reboots because of this. Haven't diagnosed yet, probably grsec or apparmor.

At this point, I believe we're going to move forward with merging freedomofpress/securedrop#5765 tomorrow, and this doesn't prevent that from happening, but I am curious how I can get this working, so I'll try again tomorrow.

[emkll]

Also seeing the same issue @creviera is reporting, for a Focal install on hardware

UPDATE: got it to work, the issue was that I did not have all the latest built debs in the local securedrop-dev-packages-lfs/core/focal folder. After building the debs on the branch in freedomofpress/securedrop#5765 and moving them to the local lfs folder, and following the steps described, The packages are installed successfully from the local filesystem on the servers.

[sssoleileraaa]

a-ha! thanks @emkll, that did the trick. :)