jUnit reports are hard to read

[ryancurrah]

Description

The jUnit exporter is organizing tests under invalid control names. This makes it really hard to read when importing tests results into test reporting tools.

InSpec and Platform Version

InSpec version: 1.9.0
Platform version: RedHat 7.2

Replication Case

1. Import a control from another profile

audit_config_spec.rb:

title 'Audit Configuration Ansible'

require_controls 'STIG_RHEL7' do
  control 'RHEL-07-030090'
  control 'RHEL-07-030310'
  control 'RHEL-07-030380'
  control 'RHEL-07-030381'
  control 'RHEL-07-030382'
  control 'RHEL-07-030383'
  control 'RHEL-07-030390'
  control 'RHEL-07-030391'
  control 'RHEL-07-030392'
  control 'RHEL-07-030400'
  control 'RHEL-07-030401'
  control 'RHEL-07-030402'
  control 'RHEL-07-030403'
  control 'RHEL-07-030404'
  control 'RHEL-07-030405'
  control 'RHEL-07-030420'
  control 'RHEL-07-030421'
  control 'RHEL-07-030422'
  control 'RHEL-07-030423'
  control 'RHEL-07-030424'
  control 'RHEL-07-030425'
  control 'RHEL-07-030441'
  control 'RHEL-07-030442'
  control 'RHEL-07-030443'
  control 'RHEL-07-030444'
  control 'RHEL-07-030490'
  control 'RHEL-07-030491'
  control 'RHEL-07-030492'
  control 'RHEL-07-030510'
  control 'RHEL-07-030511'
  control 'RHEL-07-030512'
  control 'RHEL-07-030513'
  control 'RHEL-07-030514'
  control 'RHEL-07-030521'
  control 'RHEL-07-030522'
  control 'RHEL-07-030523'
  control 'RHEL-07-030524'
  control 'RHEL-07-030525'
  control 'RHEL-07-030526'
  control 'RHEL-07-030530'
  control 'RHEL-07-030531'
  control 'RHEL-07-030540'
  control 'RHEL-07-030541'
  control 'RHEL-07-030550'
  control 'RHEL-07-030560'
  control 'RHEL-07-030561'
  control 'RHEL-07-030630'
  control 'RHEL-07-030670'
  control 'RHEL-07-030671'
  control 'RHEL-07-030672'
  control 'RHEL-07-030673'
  control 'RHEL-07-030674'
  control 'RHEL-07-030710'
  control 'RHEL-07-030750'
  control 'RHEL-07-030751'
  control 'RHEL-07-030752'
  control 'RHEL-07-030753'
  control 'RHEL-07-030754'
end

control 'audit-config-ansible-01' do
  impact 1.0
  title 'Verify audit daemon is configured by Ansible correctly'
  desc 'Audit daemon config should be have a log format of RAW, no have a name set and max log file action should be rotate. Audisp daemon should have q depth set to 150. Audisp remote should not have kerb enabled and not have a remote server set.'

  describe file('/etc/audit/auditd.conf') do
    its('content') { should match /log_format = RAW/ }
    its('content') { should_not match /name =/ }
    its('content') { should match /max_log_file_action = ROTATE/ }
  end

  describe file('/etc/audisp/audispd.conf') do
    its('content') { should match /q_depth = 150/ }
  end

  describe file('/etc/audisp/audisp-remote.conf') do
    its('content') { should match /enable_krb5 = no/ }
    its('content') { should_not match /remote_server = \w+/ }
  end
end

2. Generate a junit report

jUnit report: junit_report.txt
InSpec report: inspec_report.txt

3. Import into Jenkins or another jUnit tool
4. Imported or required tests are not all showing up under the same parent

Main test results page:

tmp..inspec.cache.91e12bdaec14f85c7bb7bf2d7c6177ea821175ef.controls page:

As you can see a bunch of controls are missing. They seem to appear under usr.lib64.ruby.gems.2.3.0.gems.inspec-1.9.0.lib.inspec:

Possible Solutions

Not sure yet I will take a look at the code when I have a chance.

Stacktrace

Not applicable

[aaronlippold]

👍

[chris-rock]

This work will require us to build a junit reporter being based on-top of our InSpec JSON format instead of RSpec

[jkerry]

I put the original junit reporter together and came here to log the same issue so I can cobble together a PR in the next week or two but you did all the hard work for me 👍

I'll see what I can do with this assuming nobody else picks it up first

[jkerry]

Well on the way here. the mapping is like so: Any feedback would be appreciated @ryancurrah @chris-rock @aaronlippold . Items to still accomplish:

• Add in failing test block / reasons for resources that failed

Format:

• test suites are profiles
• test classes are controls, anonymous for resources outside controls
• test cases are individual resources

<?xml version="1.0"?>
<testsuites>
  <testsuite name="webserver" tests="38" failed="1">
    <testcase name="Windows Feature 'Web-Lgcy-Scripting' should be installed" class="Anonymnous"/>
    <testcase name="Windows Feature 'Blag' should be installed" class="Anonymnous"/>
    <testcase name="Service W3SVC should be installed" class="iis-service-0.1.0"/>
    <testcase name="Service W3SVC should be enabled" class="iis-service-0.1.0"/>
    <testcase name="Service W3SVC should be running" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Server' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-WebServer' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Common-Http' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Static-Content' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Default-Doc' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-App-Dev' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Asp-Net' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Net-Ext' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-ASP' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-ISAPI-Ext' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-ISAPI-Filter' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Health' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Http-Logging' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Request-Monitor' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Security' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Basic-Auth' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Windows-Auth' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Filtering' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-IP-Security' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Performance' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Stat-Compression' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Dyn-Compression' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Mgmt-Tools' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Mgmt-Console' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Scripting-Tools' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Mgmt-Service' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Mgmt-Compat' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Metabase' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-WMI' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Lgcy-Scripting' should be installed" class="iis-service-0.1.0"/>
    <testcase name="Windows Feature 'Web-Lgcy-Mgmt-Console' should be installed" class="iis-service-0.1.0"/>
    <testcase name="System Package Microsoft Visual C++ 2012 Redistributable (x64) - 11.0.61030 should be installed" class="installed-dependencies-0.1.0"/>
    <testcase name="System Package Microsoft Visual C++ 2012 Redistributable (x86) - 11.0.61030 should be installed" class="installed-dependencies-0.1.0"/>
  </testsuite>
  <testsuite name="base" tests="3" failed="0">
    <testcase name="Powershell exit_status should eq 2" class="CPU-count-0.1.0"/>
    <testcase name="windows should eq &quot;windows&quot;" class="OS-0.1.0"/>
    <testcase name="Powershell exit_status should eq 4" class="physical-memory-0.1.0"/>
  </testsuite>
</testsuites>

[ryancurrah]

Wow looking good @jkerry. I imported your jUnit into a test Jenkins job it's a lot easier to read. Screenshots below. Only comment in addition to your TODO is add run time to the results as well.

I took a look at the code. Based on @chris-rock feedback it looks like the class InspecRspecJUnit has to look a lot more like InspecRspecJson. Seems @jkerry has made more progress than I have. If you need more help let me know.

[jkerry]

Thanks! That's exactly right on the formatter. I'm actually cheating and using nokogiri to build the xml up from the output hash that the json formatter rolls up. Not the most efficient allocation of resources but it'll do. I'll get a PR submitted tomorrow morning.

[jkerry]

@ryancurrah PR submitted. Would you have time to build the PR and run a test or two to make sure it fits what you're looking for?

[ryancurrah]

Hey @jkerry just built off of your feature branch on my rhel7 vagrant box. It worked like a charm, the results are WAYYY more readable now. This is awesome man. I imported the results into Jenkins just fine too. Here is the gist of the resulting jUnit file https://gist.github.com/ryancurrah/f8ca7f5606a11e83f06fc27f1a8821bb. And screenshots below.

My only comment, which is minor at this point. Is it would be nice if the tests are organized under the control name. Would be pretty cool but not a show stopper. I have some down time, i'll see if can figure it out.

[jkerry]

You can only organize under one thing so it would have to be profile or control ( the control is in the class field for the test cases )

@chris-rock what's better for the product: picking one of these and running with it or adding some kind of runtime config / flag to make it user selectable?