The output of failing tests is not redirected to the junit reporter

[SimonChh]

Describe the bug
Given a test that fails (either via a failing assertion or an uncaught exception), the stdout and stderr output of the test is not redirected to the reporter. For instance, when invoking the test via -r junit, the resulting <system-out> and <system-err> tags are empty.

The output of successful tests is redirected correctly to the reporter, however.

Expected behavior
Personally I expected the output of the failing test to be redirected to the reporter, since having access to the test's output helps when trying to figure out what went wrong (especially when the test is non-deterministic and thus the error is not easy to reproduce). But maybe others have different expectations, so I'm not sure if this is really a bug.

Reproduction steps

#include <Catch/catch.hpp>
#include <iostream>

SCENARIO("A failing test")
{
  std::cout << "Cout!" << std::endl;
  std::cerr << "Cerr!" << std::endl;

  throw std::runtime_error("std::runtime_error generated from test"); // REQUIRE(false) also fails
}

Run the above test using the -r junit command-line option. The <system-out> and <system-err> tags will be empty.

Platform information:

• OS: Windows 10 Pro
• Compiler+version: Visual Studio Professional 2017, version 15.7.4
• Catch version: v2.5.0

[SimonChh]

FYI, the problem can be fixed by slightly changing the RunContext::runCurrentTest() function:

Instead of:

        CATCH_TRY {
            if (m_reporter->getPreferences().shouldRedirectStdOut) {
#if !defined(CATCH_CONFIG_EXPERIMENTAL_REDIRECT)
                RedirectedStdOut redirectedStdOut;
                RedirectedStdErr redirectedStdErr;

                timer.start();
                invokeActiveTestCase();
                redirectedCout += redirectedStdOut.str();
                redirectedCerr += redirectedStdErr.str();
#else

I used this:

        CATCH_TRY {
            if (m_reporter->getPreferences().shouldRedirectStdOut) {
#if !defined(CATCH_CONFIG_EXPERIMENTAL_REDIRECT)
                RedirectedStreams redirectedStreams(redirectedCout, redirectedCerr);

                timer.start();
                invokeActiveTestCase();
#else

Where RedirectedStreams is a RAII-style helper class that simply redirects the test output to redirectedCout and redirectedCerr in its destructor (so that the output gets redirected even if invokeActiveTestCase() throws an exception).

[JoeyGrajciar]

Feel free to create PR :)

[horenmar]

Sorry about the bug, but apparently it escaped notice for well over a year, the commit that introduced it was done on 6.12.2017: aa9d635.

I am going to look at the PR in a moment.

[Butanium]

This is still an issue with segfaults:

TEST_CASE("test") {
    std::cout << "Cout!" << std::endl;
    std::cerr << "Cerr!" << std::endl;
    // Generate a segfault
    int *a = nullptr;
    *a = 1; 
}

/home/clementd/Documents/ENS/INF560/yaalpp/cmake-build-debug/tests -r xml -d yes --order lex test
Testing started at 14:03 ...
Run with rng-seed=1189301905

/home/clementd/Documents/ENS/INF560/yaalpp/tests/test.cpp:217: Failure:
fatal error

Process finished with exit code 139 (interrupted by signal 11:SIGSEGV)
</Catch2TestRun>