Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Distinguishes between TemplateVM & AppVM in log collection #487

Merged
merged 7 commits into from
Mar 11, 2020
Merged

Distinguishes between TemplateVM & AppVM in log collection #487

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
509fd86
Consolidates log aggregation into single state file
Feb 20, 2020
b6406f1
Expands config tests for logging
Mar 3, 2020
fa2b53b
Removes duplicate sd-log RPC policy
Mar 3, 2020
caa77af
Sets log hostname for Whonix & Viewer
Mar 5, 2020
be27968
Ensures sd-gpg does not log, adds tests
Mar 9, 2020
e73aceb
Removes special case logging setup for proxy
Mar 9, 2020
adc6dd4
Reboots logging VMs during install
Mar 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Sets log hostname for Whonix & Viewer 
The sd-viewer config didn't have the logging setup included, which was
an oversight. The Whonix-related VMs, sd-whonix, sd-proxy, and
corresponding templates, require special treatment that's now accounted
for in the single log-config state file.

Significantly expanded test coverage in order to verify the end state.
  • Loading branch information
Conor Schaefer committed Mar 10, 2020
commit caa77af4bfc9a918a8d0cc142fa575d506932228
16 changes: 10 additions & 6 deletions dom0/sd-logging-setup.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ sd-log-remove-rsyslog-qubes-plugin:
- require:
- file: sd-log-remove-rsyslog-qubes-plugin

{% elif grains['id'] == "sd-whonix" %}
{% elif grains['id'] in ["sd-whonix", "sd-proxy", "sd-proxy-buster-template"] %}
# We can not place the file on the template under /etc/rsyslog.d/ because of whonix
# template. This sdlog.conf file is the same from the securedrop-log package, to
# make sure that rsyslogd use our logging plugin.
Expand All @@ -63,13 +63,17 @@ sd-rc-enable-logging:
- marker_end: "### END securedrop-workstation ###"
- content: |
# Add sd-rsyslog.conf file for syslog
ln -sf /rw/config/sd-rsyslog.conf /etc/sd-rsyslog.conf
if [ ! -f /etc/rsyslog.d/sdlog.conf ]; then
ln -sf /rw/config/sdlog.conf /etc/rsyslog.d/sdlog.conf
fi
ln -sf /rw/config/sdlog.conf /etc/rsyslog.d/sdlog.conf
cat <<EOF > /etc/sd-rsyslog.conf
[sd-rsyslog]
remotevm = sd-log
localvm = {{ grains['id'] }}
EOF
systemctl restart rsyslog
cmd.run:
- name: ln -sf /rw/config/sd-rsyslog.conf /etc/sd-rsyslog.conf && systemctl restart rsyslog
- name: /rw/config/rc.local
- require:
- file: sd-rc-enable-logging-for-sd-whonix

{% else %}
# For all other VMs, configure to send to sd-log
Expand Down
9 changes: 1 addition & 8 deletions dom0/sd-viewer-files.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

include:
- fpf-apt-test-repo
- sd-logging-setup

sd-viewer-install-mimetype-handler-package:
pkg.installed:
Expand All @@ -28,11 +29,3 @@ sd-viewer-install-libreoffice:
attempts: 3
interval: 60
- install_recommends: False

sd-rsyslog-for-sd-viewer:
file.managed:
- name: /etc/sd-rsyslog.conf
- source: "salt://sd-rsyslog.conf.j2"
- template: jinja
- context:
vmname: sd-viewer
3 changes: 3 additions & 0 deletions dom0/sd-workstation.top
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ base:
- sd-gpg-files
sd-proxy-buster-template:
- sd-proxy-template-files
sd-proxy:
- sd-logging-setup
sd-app:
- sd-app-config
sd-viewer-buster-template:
Expand All @@ -38,6 +40,7 @@ base:
- sd-sys-firewall-files
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this is a regression, but getting an issue with any sd-viewer-* vm:


Mar 05 15:46:42 sd-viewer rsyslogd[465]: omprog: program '/usr/sbin/sd-rsyslog' (pid 5737) terminated; will be restarted [v8.1901.0 try https://www.rsyslog.com/e/2119 ]
Mar 05 15:46:42 sd-viewer rsyslogd[465]: action 'action-0-omprog' suspended (module 'omprog'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try https://www.rsysl
Mar 05 15:46:43 sd-viewer rsyslogd[465]: action 'action-0-omprog' resumed (module 'omprog') [v8.1901.0 try https://www.rsyslog.com/e/2359 ]
Mar 05 15:46:43 sd-viewer rsyslogd[465]: child process (pid 5738) exited with status 1 [v8.1901.0]
Mar 05 15:46:43 sd-viewer rsyslogd[465]: omprog: program '/usr/sbin/sd-rsyslog' (pid 5738) terminated; will be restarted [v8.1901.0 try https://www.rsyslog.com/e/2119 ]
Mar 05 15:46:43 sd-viewer rsyslogd[465]: action 'action-0-omprog' suspended (module 'omprog'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try https://www.rsysl
Mar 05 15:46:44 sd-viewer rsyslogd[465]: action 'action-0-omprog' resumed (module 'omprog') [v8.1901.0 try https://www.rsyslog.com/e/2359 ]
Mar 05 15:46:44 sd-viewer rsyslogd[465]: child process (pid 5745) exited with status 1 [v8.1901.0]
Mar 05 15:46:44 sd-viewer rsyslogd[465]: omprog: program '/usr/sbin/sd-rsyslog' (pid 5745) terminated; will be restarted [v8.1901.0 try https://www.rsyslog.com/e/2119 ]
Mar 05 15:46:44 sd-viewer rsyslogd[465]: action 'action-0-omprog' suspended (module 'omprog'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try https://www.rsysl

Copy link
Contributor Author

@conorsch conorsch Mar 6, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I've added a config test for this case, but it's failing, so commented it out. We weren't testing for this case before, so we should compare behavior on master and see if the log messages occur there.

On this branch, I see those messages, even though a given VM is shipping logs successfully to sd-log. So it doesn't appear to be a final indicator of failure.

sd-whonix:
- sd-whonix-hidserv-key
- sd-logging-setup
securedrop-workstation-buster:
- sd-workstation-template-files
- sd-logging-setup
Expand Down
18 changes: 16 additions & 2 deletions tests/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,15 +113,29 @@ def _fileExists(self, remote_path):

return True

def logging_configured(self):
def logging_configured(self, vmname=False):
"""
Make sure rsyslog is configured to send in data to sd-log vm.
Takes an optional 'vmname' argument, in case hostname
returned by system is an insufficient identifier, e.g. Whonix.
"""
self.assertTrue(self._package_is_installed("securedrop-log"))
self.assertTrue(self._fileExists("/usr/sbin/sd-rsyslog"))
self.assertTrue(self._fileExists("/etc/rsyslog.d/sdlog.conf"))
self.assertTrue(self._fileExists("/etc/sd-rsyslog.conf"))
# Then we check the configuration inside of the file.
file_content = self._get_file_contents("/etc/sd-rsyslog.conf")
static_content = """[sd-rsyslog]
remotevm = sd-log
"""
# A hardcoded vmname should only be present if required,
# since securedrop-log will default to value of `hostname`.
if vmname:
static_content += "localvm = {}\n".format(self.vm_name)
self.assertEqual(file_content, static_content)
self.assertTrue(self._package_is_installed("securedrop-log"))
# Check for evidence of misconfigured logging in syslog,
# fail if matching events found
# Several VMs show this error message even though they're shipping logs,
# so let's investigate further.
# cmd_output = self._run("sudo grep -F \"action 'action-0-omprog' suspended (module 'omprog')\" /var/log/syslog | wc -l").strip() # noqa
# self.assertTrue(cmd_output == "0")
19 changes: 19 additions & 0 deletions tests/test_log_vm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,25 @@ def test_redis_service_running(self):
results = self._run("sudo systemctl is-active redis")
assert results == "active"

def test_logs_are_flowing(self):
cmd_output = self._run("ls -1 /home/user/QubesIncomingLogs")
log_dirs = cmd_output.split("\n")
# Confirm AppVMs are sending logs
self.assertTrue("sd-app" in log_dirs)
self.assertTrue("sd-whonix" in log_dirs)
# The following will only have logs if the machine has booted,
# which is not guaranteed given that we randomize test order.
# self.assertTrue("sd-devices" in log_dirs)
# self.assertTrue("sd-proxy" in log_dirs)
# self.assertTrue("sd-viewer" in log_dirs)

def test_log_dirs_properly_named(self):
# Rerunning this command to keep test output readable
cmd_output = self._run("ls -1 /home/user/QubesIncomingLogs")
log_dirs = cmd_output.split("\n")
# Confirm we don't have 'host' entries from Whonix VMs
self.assertFalse("host" in log_dirs)


def load_tests(loader, tests, pattern):
suite = unittest.TestLoader().loadTestsFromTestCase(SD_Log_Tests)
Expand Down
2 changes: 1 addition & 1 deletion tests/test_proxy_vm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def test_whonix_ws_repo_absent(self):
assert not self._fileExists("/etc/apt/sources.list.d/whonix.list")

def test_logging_configured(self):
self.logging_configured()
self.logging_configured(vmname=True)

def test_mime_types(self):
with open("sd-proxy/mimeapps.list", "r") as f:
Expand Down
2 changes: 1 addition & 1 deletion tests/test_sd_whonix.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ def test_sd_whonix_repo_enabled(self):
assert self._fileExists(self.whonix_apt_list)

def test_logging_configured(self):
self.logging_configured()
self.logging_configured(vmname=True)


def load_tests(loader, tests, pattern):
Expand Down