Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More comprehensive "make clean" action #351

Merged
merged 6 commits into from
Dec 2, 2019
Merged

More comprehensive "make clean" action #351

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f4eac01
Updates config tests for dom0 repo URL
Nov 25, 2019
0f11ac3
Consolidates all Salt top files
Nov 22, 2019
e540ea6
More robust clean action
Nov 22, 2019
63edaed
Includes upstream update.qubes-vm formula
Nov 22, 2019
1fede0e
Leaner provision-all
Nov 24, 2019
1b7a901
Address review comments
Dec 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 19 additions & 35 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,49 +17,35 @@ clone: assert-dom0 ## Pulls the latest repo from work VM to dom0
@./scripts/clone-to-dom0

qubes-rpc: prep-salt ## Places default deny qubes-rpc policies for sd-svs and sd-gpg
sudo qubesctl top.enable sd-dom0-qvm-rpc
sudo qubesctl --show-output --targets sd-dom0-qvm-rpc state.highstate

sd-workstation-template: prep-salt ## Provisions base template for SDW AppVMs
sudo qubesctl top.enable sd-workstation-template
sudo qubesctl top.enable sd-workstation-template-files
sudo qubesctl --show-output --targets sd-workstation-template state.highstate
sudo qubesctl --show-output state.sls sd-workstation-template
sudo qubesctl --show-output --skip-dom0 --targets sd-workstation-template state.highstate

sd-proxy: prep-salt ## Provisions SD Proxy VM
sudo qubesctl top.enable sd-proxy
sudo qubesctl top.enable sd-proxy-files
sudo qubesctl --show-output --targets sd-proxy-template state.highstate
sudo qubesctl --show-output --targets sd-proxy state.highstate
sudo qubesctl --show-output state.sls sd-proxy
sudo qubesctl --show-output --skip-dom0 --targets sd-proxy-template state.highstate

sd-gpg: prep-salt ## Provisions SD GPG keystore VM
sudo qubesctl top.enable sd-gpg
sudo qubesctl top.enable sd-gpg-files
sudo qubesctl --show-output --targets sd-gpg state.highstate
sudo qubesctl --show-output state.sls sd-gpg
sudo qubesctl --show-output --skip-dom0 --targets sd-gpg state.highstate

sd-svs: prep-salt ## Provisions SD SVS VM
sudo qubesctl top.enable sd-svs
sudo qubesctl top.enable sd-svs-files
sudo qubesctl top.enable sd-svs-config
sudo qubesctl --show-output --targets sd-svs-template state.highstate
sudo qubesctl --show-output --targets sd-svs state.highstate
sudo qubesctl --show-output state.sls sd-svs
sudo qubesctl --show-output --skip-dom0 --targets sd-svs-template,sd-svs state.highstate

sd-whonix: prep-salt ## Provisions SD Whonix VM
sudo qubesctl top.enable sd-whonix
sudo qubesctl top.enable sd-whonix-hidserv-key
sudo qubesctl --show-output --targets sd-whonix-template state.highstate
sudo qubesctl --show-output --targets sd-whonix state.highstate
sudo qubesctl --show-output state.sls sd-whonix
sudo qubesctl --show-output --skip-dom0 --targets sd-whonix-template,sd-whonix state.highstate

sd-svs-disp: prep-salt ## Provisions SD Submission Viewing VM
sudo qubesctl top.enable sd-svs-disp
sudo qubesctl top.enable sd-svs-disp-files
sudo qubesctl --show-output --targets sd-svs-disp-template state.highstate
sudo qubesctl --show-output --targets sd-svs-disp state.highstate
sudo qubesctl --show-output state.sls sd-svs-disp
sudo qubesctl --show-output --skip-dom0 --targets sd-svs-disp-template,sd-svs-disp state.highstate

sd-export: prep-salt ## Provisions SD Export VM
sudo qubesctl top.enable sd-export
sudo qubesctl top.enable sd-export-files
sudo qubesctl --show-output --targets sd-export-template state.highstate
sudo qubesctl --show-output --targets sd-export-export-dvm state.highstate
sudo qubesctl --show-output state.sls sd-export
sudo qubesctl --show-output --skip-dom0 --targets sd-export-template,sd-export-usb,sd-export-usb-dvm state.highstate

clean-salt: assert-dom0 ## Purges SD Salt configuration from dom0
@echo "Purging Salt config..."
Expand Down Expand Up @@ -89,10 +75,10 @@ remove-sd-export: assert-dom0 ## Destroys SD EXPORT VMs
@./scripts/destroy-vm sd-export-usb
@./scripts/destroy-vm sd-export-usb-dvm

clean: assert-dom0 destroy-all clean-salt ## Destroys all SD VMs
clean: assert-dom0 prep-salt destroy-all ## Destroys all SD VMs
sudo qubesctl --show-output state.sls sd-clean-all
sudo dnf -y -q remove securedrop-workstation-dom0-config 2>/dev/null || true
sudo rm -f /usr/bin/securedrop-update \
/etc/cron.daily/securedrop-update-cron
$(MAKE) clean-salt

test: assert-dom0 ## Runs all application tests (no integration tests yet)
python3 -m unittest discover -v tests
Expand Down Expand Up @@ -136,11 +122,9 @@ prep-dom0: prep-salt # Copies dom0 config files for VM updates
sudo qubesctl top.enable sd-dom0-files
sudo qubesctl --show-output --targets dom0 state.highstate

list-vms: ## Prints all Qubes VMs managed by Workstation salt config
@./scripts/list-vms

destroy-all: ## Destroys all VMs managed by Workstation salt config
@./scripts/list-vms | xargs ./scripts/destroy-vm
qubes-prefs default_dispvm fedora-30-dvm
./scripts/destroy-vm --all

.PHONY: update-pip-requirements
update-pip-requirements: ## Updates all Python requirements files via pip-compile.
Expand Down
31 changes: 6 additions & 25 deletions dom0/fpf-apt-test-repo.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,8 @@
# -*- coding: utf-8 -*-
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

# Handle misconfigured jessie-backports repo in default debian-9 TemplateVM.
# The Jessie repos aren't maintained anymore, and their inclusion causes
# even apt update to fail.
remove-jessie-backports-repo:
file.line:
- name: /etc/apt/sources.list
# Unclear why "Delete" *must* be capitalized, but that's the case!
- mode: delete
- match: jessie-backports
# quiet param seems to be ignored, so using "onlyif" to test existence
- quiet: True
- onlyif:
- test -f /etc/apt/sources.list
#
include:
- update.qubes-vm
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this salt file located? Is this provided by qubes' build in salt mgmt commands? Might be worth adding a comment here for future maintainers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for more comments everywhere, will add.


# That's right, we need to install a package in order to
# configure a repo to install another package
Expand All @@ -22,22 +11,14 @@ install-python-apt-for-repo-config:
- pkgs:
- python-apt
- require:
- file: remove-jessie-backports-repo
# Require that the Qubes update state has run first
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as https://github.com/freedomofpress/securedrop-workstation/pull/351/files#diff-f739e903ed54a409c69c5835698a9336R5

- sls: update.qubes-vm

configure apt-test apt repo:
configure-apt-test-apt-repo:
pkgrepo.managed:
- name: "deb [arch=amd64] https://apt-test-qubes.freedom.press stretch main"
- file: /etc/apt/sources.list.d/securedrop_workstation.list
- key_url: "salt://sd/sd-workstation/apt-test-pubkey.asc"
- clean_file: True # squash file to ensure there are no duplicates
- require:
- pkg: install-python-apt-for-repo-config

# Ensure all apt updates are applied, since the VMs
# will be cloned, duplicating package version drift.
update-all-apt-packages:
pkg.uptodate:
- cache_valid_time: "3600"
- dist_upgrade: True
- require:
- pkg: install-python-apt-for-repo-config
45 changes: 45 additions & 0 deletions dom0/sd-clean-all.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# -*- coding: utf-8 -*-
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

set-fedora-as-default-dispvm:
cmd.run:
- name: qubes-prefs default_dispvm fedora-30-dvm

remove-dom0-sdw-config-files:
file.absent:
- names:
- /opt/securedrop
- /etc/yum.repos.d/securedrop-workstation-dom0.repo
- /usr/bin/securedrop-update
- /etc/pki/rpm-gpg/RPM-GPG-KEY-securedrop-workstation-test
- /etc/cron.daily/securedrop-update-cron
- /usr/share/securedrop/icons

sd-cleanup-sys-firewall:
cmd.run:
- names:
- qvm-run sys-firewall 'sudo rm -f /rw/config/RPM-GPG-KEY-securedrop-workstation-test'
- qvm-run sys-firewall 'sudo rm -f /rw/config/sd-copy-rpm-repo-pubkey.sh'
- qvm-run sys-firewall 'sudo rm -f /etc/pki/rpm-gpg/RPM-GPG-KEY-securedrop-workstation-test'
- qvm-run sys-firewall 'sudo perl -pi -E "s#^/rw/config/sd-copy-rpm-repo-pubkey.sh##" /rw/config/rc.local'

sd-cleanup-rpc-mgmt-policy:
file.replace:
- names:
- /etc/qubes-rpc/policy/qubes.VMShell
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these policies do not exist on my machine. How/when are these created? qubesctl?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These policies are created automatically during upgrades via a disp-mgmt VM, and also removed automatically. If someone ctrl+c's the make all process, it's possible for them to hang around. Possibly overkill to worry about them here, I'm happy to excise.

- /etc/qubes-rpc/policy/qubes.VMRootShell
- repl: ''
- pattern: '^disp-mgmt-sd-\w+\s+sd-\w+\s+allow,user=root'

{% set sdw_customized_rpc_files = salt['cmd.shell']('grep -rl "BEGIN securedrop-workstation" /etc/qubes-rpc/ | cat').splitlines() %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! There's one edge case that we could. When vim or other editor leaves a swap file (like it did on my system), the file is viewed as a binary and the make clean action will fail. This is obviously an edge case (and deleting that file resolved), see output below. Perhaps grepping for strings that don't start with a dot is sufficient here, but IMO not required for merge.

----------
          ID: sd-cleanup-rpc-policy-grants
    Function: file.replace
        Name: /etc/qubes-rpc/policy/.qubes.Filecopy.swp
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/site-packages/salt/state.py", line 1837, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1794, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/site-packages/salt/states/file.py", line 3934, in replace
                  ignore_if_missing=ignore_if_missing)
                File "/usr/lib/python2.7/site-packages/salt/modules/file.py", line 2035, in replace
                  .format(path)
              SaltInvocationError: Cannot perform string replacements on a binary file: /etc/qubes-rpc/policy/.qubes.Filecopy.swp
     Started: 16:16:15.438709
    Duration: 5.279 ms
     Changes:   

Summary for local
-------------
Succeeded: 28 (changed=26)
Failed:     1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, great catch. We can just pass -I to grep so it doesn't report binary files. That should handle the edge case you describe.

{% if sdw_customized_rpc_files|length > 0 %}
sd-cleanup-rpc-policy-grants:
file.replace:
- names: {{ sdw_customized_rpc_files }}
- pattern: '### BEGIN securedrop-workstation ###.*### END securedrop-workstation ###'
- flags:
- MULTILINE
- DOTALL
- repl: ''
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While running make all/make clean several times, I've observed blank lines in the rpc policy files

I think that adding append_newline to False here could resolve

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, have not seen that, will get a repro and cleanup as you suggest.

- backup: no
{% endif %}
6 changes: 0 additions & 6 deletions dom0/sd-dom0-files.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-dom0-qvm-rpc.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-export-files.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-export.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-gpg-files.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-gpg.top
View file
Open in desktop

This file was deleted.

7 changes: 0 additions & 7 deletions dom0/sd-proxy-files.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-proxy.top
View file
Open in desktop

This file was deleted.

2 changes: 2 additions & 0 deletions dom0/sd-svs-config.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# -*- coding: utf-8 -*-
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :
##
# sd-svs-config
# ========
Expand Down
3 changes: 0 additions & 3 deletions dom0/sd-svs-config.top
View file
Open in desktop

This file was deleted.

7 changes: 0 additions & 7 deletions dom0/sd-svs-disp-files.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-svs-disp.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-svs-files.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-svs.top
View file
Open in desktop

This file was deleted.

3 changes: 3 additions & 0 deletions dom0/sd-sys-firewall-files.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# -*- coding: utf-8 -*-
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :
#
sys-firewall-rpm-test-key:
file.managed:
- name: /rw/config/RPM-GPG-KEY-securedrop-workstation-test
Expand Down
6 changes: 0 additions & 6 deletions dom0/sd-sys-firewall-files.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-sys-vms.top
View file
Open in desktop

This file was deleted.

9 changes: 0 additions & 9 deletions dom0/sd-vm-updates.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-whonix-hidserv-key.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-whonix.top
View file
Open in desktop

This file was deleted.

7 changes: 0 additions & 7 deletions dom0/sd-workstation-template-files.top
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions dom0/sd-workstation-template.top
View file
Open in desktop

This file was deleted.

39 changes: 39 additions & 0 deletions dom0/sd-workstation.top
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# -*- coding: utf-8 -*-
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a massive improvement

# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

base:
dom0:
- sd-sys-vms
- sd-dom0-files
- sd-workstation-template
- sd-dom0-qvm-rpc
- sd-export
- sd-gpg
- sd-proxy
- sd-svs-disp
- sd-svs
- sd-whonix
sd-export-template:
- sd-export-files
sd-gpg:
- sd-gpg-files
sd-proxy-template:
- sd-proxy-template-files
sd-svs:
- sd-svs-config
sd-svs-disp-template:
- sd-svs-disp-files
sd-svs-template:
- sd-svs-files
sys-firewall:
- sd-sys-firewall-files
sd-whonix:
- sd-whonix-hidserv-key
securedrop-workstation:
- sd-workstation-template-files

# "Placeholder" config to trigger TemplateVM boots,
# so upgrades can be applied automatically via cron.
qubes:type:template:
- match: pillar
- topd
Loading