Adds dev-focal to run SecureDrop on Focal container #5544

kushaldas · 2020-09-30T16:23:07Z

Status

Ready for review.

Towards #5524

Description of Changes

make dev-focal will start a Focal container with SecureDrop
running.
Also updates the gpg2 --import command to import into the
pubring.gpg keyring file explictly. Related Ansible change
is tracked via

#5499

The tests in Focal will fail to run, and will be fixed via #5585

Testing

make dev-focal to see if it starts a container on Focal
make dev should start SecureDrop on xenial
make test should not have any error.
make test-focal will show test failures (as we are slowly fixing them for Focal)
CI should be green

Deployment

Any special considerations for deployment? Consider both:

Upgrading existing production instances.
New installs.

Checklist

If you made changes to the server application code:

Linting (make lint) and tests (make test) pass in the development container

If you made changes to `securedrop-admin`:

Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

Configuration tests pass

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

If you made changes to documentation:

Doc linting (make docs-lint) passed locally

If you added or updated a code dependency:

Choose one of the following:

I have performed a diff review and pasted the contents to the packaging wiki
I would like someone else to do the diff review

emkll

Thanks for the changes @kushaldas , some preliminary comments inline. There are 33 tests failing here under Focal:

tests/test_alembic.py::test_alembic_head_matches_db_models FAILED        [  0%]
tests/test_alembic.py::test_alembic_migration_downgrade[523fff3f969c] FAILED [  3%]
tests/test_alembic.py::test_alembic_migration_downgrade[60f41bb14d98] FAILED [  4%]
tests/test_alembic.py::test_alembic_migration_downgrade[48a75abc0121] FAILED [  4%]
tests/test_alembic.py::test_alembic_migration_downgrade[35513370ba0d] FAILED [  4%]
tests/test_alembic.py::test_alembic_migration_downgrade[3da3fcab826a] FAILED [  4%]
tests/test_alembic.py::test_alembic_migration_downgrade[b58139cfdc8c] FAILED [  5%]
tests/test_alembic.py::test_alembic_migration_downgrade[f2833ac34bb6] FAILED [  6%]
tests/test_alembic.py::test_alembic_migration_downgrade[a9fe328b053a] FAILED [  6%]
tests/test_alembic.py::test_schema_unchanged_after_up_then_downgrade[fccf57ceef02] FAILED [  6%]
tests/test_alembic.py::test_schema_unchanged_after_up_then_downgrade[3d91d6948753] FAILED [  6%]
tests/test_alembic.py::test_schema_unchanged_after_up_then_downgrade[60f41bb14d98] FAILED [  7%]
tests/test_alembic.py::test_schema_unchanged_after_up_then_downgrade[2d0ce3ee5bdc] FAILED [  7%]
tests/test_alembic.py::test_schema_unchanged_after_up_then_downgrade[e0a525cbab83] FAILED [  7%]
tests/test_alembic.py::test_schema_unchanged_after_up_then_downgrade[b58139cfdc8c] FAILED [  8%]
tests/test_alembic.py::test_schema_unchanged_after_up_then_downgrade[f2833ac34bb6] FAILED [  8%]
tests/test_alembic.py::test_downgrade_with_data[b58139cfdc8c] FAILED     [ 13%]
tests/test_crypto_util.py::test_delete_reply_keypair FAILED              [ 17%]
tests/test_crypto_util.py::test_delete_reply_keypair_pinentry_status_is_handled FAILED [ 17%]
tests/test_integration.py::test_submit_message FAILED                    [ 22%]
tests/test_integration.py::test_submit_file FAILED                       [ 22%]
tests/test_journalist.py::test_delete_source_deletes_source_key FAILED   [ 41%]
tests/test_rm.py::test_secure_delete_capability FAILED                   [ 60%]
tests/test_source.py::test_metadata_route FAILED                         [ 68%]
tests/test_source.py::test_metadata_v2_url FAILED                        [ 68%]
tests/test_source.py::test_metadata_v3_url FAILED                        [ 68%]
tests/functional/test_source_metadata.py::TestInstanceMetadata::test_instance_metadata FAILED [ 79%]
tests/pageslayout/test_journalist.py::TestJournalistLayout::test_col_flagged[en_US] FAILED [ 86%]
tests/pageslayout/test_journalist.py::TestJournalistLayout::test_col_flagged[ar] FAILED [ 86%]
tests/pageslayout/test_journalist.py::TestJournalistLayout::test_flag[en_US] FAILED [ 91%]
tests/pageslayout/test_journalist.py::TestJournalistLayout::test_flag[ar] FAILED [ 91%]
tests/pageslayout/test_source.py::TestSourceLayout::test_source_flagged[en_US] FAILED [ 97%]
tests/pageslayout/test_source.py::TestSourceLayout::test_source_flagged[ar] FAILED [ 97%]

emkll · 2020-09-30T21:14:57Z

securedrop/bin/run-test

@@ -37,6 +37,8 @@ mkdir -p "../test-results"
 export PAGE_LAYOUT_LOCALES
 export TOR_FORCE_NET_CONFIG=0

+python3 -m pip install pytest pytest-xdist pluggy -U


As you've identified in the PR description, it would definitely make sense to move this to develop-requirements and/or test-requirements pinning the proper version and hashes. Since this doesn't affect the app code dependencies, we don't need a diff review, and I'm guessing newer versions should work under Xenial

emkll · 2020-09-30T21:15:31Z

securedrop/dockerfiles/focal/python3/Dockerfile

@@ -0,0 +1,74 @@
+# ubuntu 16.04 image from 2019-03-12


looks like this comment needs to be updated?

emkll · 2020-09-30T21:18:14Z

securedrop/dockerfiles/focal/python3/Dockerfile

@@ -0,0 +1,74 @@
+# ubuntu 16.04 image from 2019-03-12
+FROM ubuntu@sha256:2e70e9c81838224b5311970dbf7ed16802fbfe19e7a70b3cbfa3d7522aa285b4


Because this Dockerfile introduced here is pretty much identical to the Xenial Dockerfile, what do you think about using Docker args to feed the Ubuntu container hash and python version to the Dockerfile so that we only maintain a single dockerfile? This is the approach used in the Workstation dom0 build logic in https://github.com/freedomofpress/securedrop-workstation/pull/612/files

In the past we've used BASE_OS variable and it may indeed be easier for us to continue going that route, especially since we aren't planning supporting both Focal and Xenial for very long.

We also have different packages installed in this container. I would love to keep in this BASE_OS style two different dockerfiles.

kushaldas · 2020-10-08T15:58:15Z

Due to the updating of the pytest module, now i will have to tests. https://app.circleci.com/pipelines/github/freedomofpress/securedrop/1133/workflows/556ab459-6515-4104-a338-7eb3355bd314/jobs/45446 will discuss more during standup.

kushaldas · 2020-10-09T13:56:39Z

Status update is in this commit message: 19f527b

kushaldas · 2020-10-13T16:11:04Z

Still requires update for lint errors.

kushaldas · 2020-10-27T08:43:38Z

pip gave pip._vendor.urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')), I will kick the CI once again.

emkll

make dev-focal is failing due to TBB browser version not available https://github.com/freedomofpress/securedrop/pull/5544/files#diff-96bf85e6e44291b96a0917ee04d234a6986bca1858f16f195446ed697d4bf53eR26

https://github.com/freedomofpress/securedrop/pull/5544/files#r497806221 is a suggestion to minimize maintenance through a single docker file for both focal and xenial

kushaldas · 2020-10-27T13:57:06Z

make dev-focal is failing due to TBB browser version not available https://github.com/freedomofpress/securedrop/pull/5544/files#diff-96bf85e6e44291b96a0917ee04d234a6986bca1858f16f195446ed697d4bf53eR26

I will update this.

https://github.com/freedomofpress/securedrop/pull/5544/files#r497806221 is a suggestion to minimize maintenance through a single docker file for both focal and xenial

I commented above at #5544 (comment) to point out that the Dockerfiles have different dependencies installed. And just maintaining two different versions makes the whole process much simpler.

emkll · 2020-10-27T14:44:41Z

make dev-focal to see if it starts a container on Focal
make dev should start SecureDrop on xenial
make test should not have any error.
❗ make test-focal should have around 33 errors :)

make test-focal is not working for me locally:

var/lib/gems/2.7.0/gems/sass-3.4.23/lib/sass/util.rb:1109: warning: constant ::Fixnum is deprecated
      write static/css/journalist.css
      write static/css/journalist.css.map
      write static/css/source.css
      write static/css/source.css.map
 Generating securedrop/config.py...
Traceback (most recent call last):
  File "/opt/venvs/securedrop-app-code/bin/pytest", line 5, in <module>
    from pytest import main
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pytest.py", line 7, in <module>
    from _pytest.assertion import register_assert_rewrite
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/__init__.py", line 12, in <module>
    from _pytest.assertion import rewrite
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 22, in <module>
    from _pytest.assertion import util
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/util.py", line 11, in <module>
    import _pytest._code
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/_code/__init__.py", line 6, in <module>
    from .code import Code  # noqa
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/_code/code.py", line 15, in <module>
    import pluggy
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pluggy/__init__.py", line 16, in <module>
    from .manager import PluginManager, PluginValidationError
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pluggy/manager.py", line 6, in <module>
    import importlib_metadata
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 471, in <module>
    __version__ = version(__name__)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 438, in version
    return distribution(package).version
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 411, in distribution
    return Distribution.from_name(package)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 179, in from_name
    dists = resolver(name)
  File "<frozen importlib._bootstrap_external>", line 1382, in find_distributions
  File "/usr/lib/python3.8/importlib/metadata.py", line 466, in find_distributions
    found = cls._search_paths(context.name, context.path)
AttributeError: 'str' object has no attribute 'name'
make: *** [Makefile:231: test-focal] Error 1

kushaldas · 2020-10-27T14:48:17Z

make test-focal is not working for me locally:

Sorry, just now updated the test instruction, we can not run make test-focal untill #5585 is merged.

rmol

#5585 is merged, we're now able to run the dev server and tests on Focal, and the test failures under Focal have their own issues (#5602, #5604, #5605). Until we add the test-focal job (#5524) they won't affect CI for other PRs, so I'm OK with merging this.

`make dev-focal` will start a Focal container with SecureDrop running. Also updates the gpg2 --import command to import into the pubring.gpg keyring file explictly. Related Ansible change is tracked via #5499

emkll · 2020-11-04T14:37:39Z

make dev-focal to see if it starts a container on Focal
make dev should start SecureDrop on xenial
make test should not have any error.
❗ make test-focal will show test failures see below
CI should be green

Getting an error where functional tests aren't running in make test-focal, are you seeing that as well @kushaldas ?

[...]
tests/test_template_filters.py::test_journalist_filters PASSED                                                                                                             [ 75%]
tests/test_worker.py::test_no_interrupted_jobs PASSED                                                                                                                      [ 75%]
tests/test_worker.py::test_job_interruption PASSED                                                                                                                         [ 75%]
tests/test_worker.py::test_worker_for_job PASSED                                                                                                                           [ 75%]
tests/functional/test_admin_interface.py::TestAdminInterface::test_admin_interface /home/user/src/securedrop/securedrop/bin/run-test: line 51:    67 Killed                  pytest --force-flaky --max-runs=3 --page-layout --durations 10 --junitxml=../test-results/junit.xml --cov-report term-missing --cov-report html:../test-results/cov_html --cov-report xml:../test-results/cov.xml --cov-report annotate:../test-results/cov_annotate --cov=. "$@"
make: *** [Makefile:231: test-focal] Error 137

emkll

make test-focal is now working for me locally, ran out of memory in my VM (thanks @kushaldas for the assist). Only two tests are failing inside the Focal container, those are captured in #5592.

CI is also green, good to merge from my perspective

emkll reviewed Sep 30, 2020

View reviewed changes

kushaldas force-pushed the dev_focal branch 5 times, most recently from f41cdf8 to 602f593 Compare October 8, 2020 10:13

kushaldas force-pushed the dev_focal branch 2 times, most recently from c4bf136 to 6f87968 Compare October 12, 2020 13:45

kushaldas mentioned this pull request Oct 15, 2020

Support for Ubuntu 20.04 (Focal) #4768

Closed

53 tasks

kushaldas force-pushed the dev_focal branch from fe70863 to b5ea431 Compare October 21, 2020 14:26

kushaldas marked this pull request as ready for review October 21, 2020 16:07

kushaldas requested a review from redshiftzero as a code owner October 21, 2020 16:07

emkll suggested changes Oct 27, 2020

View reviewed changes

kushaldas force-pushed the dev_focal branch from 99f2d46 to fbff754 Compare October 29, 2020 04:38

rmol previously approved these changes Oct 29, 2020

View reviewed changes

kushaldas added 3 commits November 4, 2020 18:46

Adds dev-focal to run SecureDrop on Focal container

62a7d1f

`make dev-focal` will start a Focal container with SecureDrop running. Also updates the gpg2 --import command to import into the pubring.gpg keyring file explictly. Related Ansible change is tracked via #5499

Updates comment for docker image date

f76cc65

Updates Tor Browser version

993e83a

kushaldas dismissed rmol’s stale review via 993e83a November 4, 2020 13:16

kushaldas force-pushed the dev_focal branch from fbff754 to 993e83a Compare November 4, 2020 13:16

emkll approved these changes Nov 4, 2020

View reviewed changes

emkll merged commit dbb020e into develop Nov 4, 2020

emkll deleted the dev_focal branch November 4, 2020 18:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds dev-focal to run SecureDrop on Focal container #5544

Adds dev-focal to run SecureDrop on Focal container #5544

kushaldas commented Sep 30, 2020 •

edited

Loading

emkll left a comment •

edited

Loading

emkll Sep 30, 2020 •

edited

Loading

emkll Sep 30, 2020

emkll Sep 30, 2020

kushaldas Oct 6, 2020

kushaldas commented Oct 8, 2020

kushaldas commented Oct 9, 2020

kushaldas commented Oct 13, 2020

kushaldas commented Oct 27, 2020

emkll left a comment

kushaldas commented Oct 27, 2020

emkll commented Oct 27, 2020

kushaldas commented Oct 27, 2020

rmol left a comment

emkll commented Nov 4, 2020

emkll left a comment

		@@ -0,0 +1,74 @@
		# ubuntu 16.04 image from 2019-03-12
		FROM ubuntu@sha256:2e70e9c81838224b5311970dbf7ed16802fbfe19e7a70b3cbfa3d7522aa285b4

Adds dev-focal to run SecureDrop on Focal container #5544

Adds dev-focal to run SecureDrop on Focal container #5544

Conversation

kushaldas commented Sep 30, 2020 • edited Loading

Status

Description of Changes

Testing

Deployment

Checklist

If you made changes to the server application code:

If you made changes to securedrop-admin:

If you made changes to the system configuration:

If you made non-trivial code changes:

If you made changes to documentation:

If you added or updated a code dependency:

emkll left a comment • edited Loading

Choose a reason for hiding this comment

emkll Sep 30, 2020 • edited Loading

Choose a reason for hiding this comment

emkll Sep 30, 2020

Choose a reason for hiding this comment

emkll Sep 30, 2020

Choose a reason for hiding this comment

kushaldas Oct 6, 2020

Choose a reason for hiding this comment

kushaldas commented Oct 8, 2020

kushaldas commented Oct 9, 2020

kushaldas commented Oct 13, 2020

kushaldas commented Oct 27, 2020

emkll left a comment

Choose a reason for hiding this comment

kushaldas commented Oct 27, 2020

emkll commented Oct 27, 2020

kushaldas commented Oct 27, 2020

rmol left a comment

Choose a reason for hiding this comment

emkll commented Nov 4, 2020

emkll left a comment

Choose a reason for hiding this comment

kushaldas commented Sep 30, 2020 •

edited

Loading

If you made changes to `securedrop-admin`:

emkll left a comment •

edited

Loading

emkll Sep 30, 2020 •

edited

Loading