TestConsole currently only captures stdOut, it should capture stdErr as well

[sudhz]

Problem

The TestConsole class captures only the standard output (stdout) by default. It uses a StringWriter to capture the console output.

How to fix

To capture both stdout and stderr, you would need to modify the TestConsole class to handle both streams.

Proposed solution

namespace Spectre.Console.Testing;

/// <summary>
/// A testable console.
/// </summary>
public sealed class TestConsole : IAnsiConsole, IDisposable
{
    private readonly IAnsiConsole _console;
    private readonly StringWriter _stdoutWriter;
    private readonly StringWriter _stderrWriter;
    private IAnsiConsoleCursor? _cursor;

    /// <inheritdoc/>
    public Profile Profile => _console.Profile;

    /// <inheritdoc/>
    public IExclusivityMode ExclusivityMode => _console.ExclusivityMode;

    /// <summary>
    /// Gets the console input.
    /// </summary>
    public TestConsoleInput Input { get; }

    /// <inheritdoc/>
    public RenderPipeline Pipeline => _console.Pipeline;

    /// <inheritdoc/>
    public IAnsiConsoleCursor Cursor => _cursor ?? _console.Cursor;

    /// <inheritdoc/>
    IAnsiConsoleInput IAnsiConsole.Input => Input;

    /// <summary>
    /// Gets the console output.
    /// </summary>
    public string Output => _stdoutWriter.ToString();

    /// <summary>
    /// Gets the console error output.
    /// </summary>
    public string ErrorOutput => _stderrWriter.ToString();

    /// <summary>
    /// Gets the console output lines.
    /// </summary>
    public IReadOnlyList<string> Lines => Output.NormalizeLineEndings().TrimEnd('\n').Split(new char[] { '\n' });

    /// <summary>
    /// Gets or sets a value indicating whether or not VT/ANSI sequences
    /// should be emitted to the console.
    /// </summary>
    public bool EmitAnsiSequences { get; set; }

    /// <summary>
    /// Initializes a new instance of the <see cref="TestConsole"/> class.
    /// </summary>
    public TestConsole()
    {
        _stdoutWriter = new StringWriter();
        _stderrWriter = new StringWriter();
        _cursor = new NoopCursor();

        Input = new TestConsoleInput();
        EmitAnsiSequences = false;

        _console = AnsiConsole.Create(new AnsiConsoleSettings
        {
            Ansi = AnsiSupport.Yes,
            ColorSystem = (ColorSystemSupport)ColorSystem.TrueColor,
            Out = new AnsiConsoleOutput(_stdoutWriter),
            Error = new AnsiConsoleOutput(_stderrWriter),
            Interactive = InteractionSupport.No,
            ExclusivityMode = new NoopExclusivityMode(),
            Enrichment = new ProfileEnrichment
            {
                UseDefaultEnrichers = false,
            },
        });

        _console.Profile.Width = 80;
        _console.Profile.Height = 24;
        _console.Profile.Capabilities.Ansi = true;
        _console.Profile.Capabilities.Unicode = true;
    }

    /// <inheritdoc/>
    public void Dispose()
    {
        _stdoutWriter.Dispose();
        _stderrWriter.Dispose();
    }

    /// <inheritdoc/>
    public void Clear(bool home)
    {
        _console.Clear(home);
    }

    /// <inheritdoc/>
    public void Write(IRenderable renderable)
    {
        if (EmitAnsiSequences)
        {
            _console.Write(renderable);
        }
        else
        {
            foreach (var segment in renderable.GetSegments(this))
            {
                if (segment.IsControlCode)
                {
                    continue;
                }

                Profile.Out.Writer.Write(segment.Text);
            }
        }
    }

    internal void SetCursor(IAnsiConsoleCursor? cursor)
    {
        _cursor = cursor;
    }
}

---

Please upvote 👍 this issue if you are interested in it.

[FrankRay78]

Why did you close this @sudhz ?

I'm reading Command Line Interface Guidelines and it says the following:

I came across this issue wondering if/how the TestConsole handles stderr.

It seems that it doesn't, but probably should.

Tell me if I'm mistaken.

[patriksvensson]

@FrankRay78 They closed it because TestConsole supports stderr (see ErrorOutput property).

[FrankRay78]

@patriksvensson I don't see ErrorOutput (or anything similar) on either TestConsole or CommandAppTester - have I missed something? My repo is sync'd with upstream.

[patriksvensson]

I've might be mistaken

[FrankRay78]

@patriksvensson I don't know this as well as you, but where my interested started was trying to unit test the provided exception handler outputs the message to stderr, not stdout. I've searched and I don't find anywhere where stderr is explicitly used for non-expected outputs. eg. Option missing required value etc. There also seems a GNU consensus that --help goes to stdout when explicitly requested, but stderr if say the app is run without a default command and without a command being set see here. Is it fair to say that stdout is used for basically all of spectre.console output? That's what I'm wondering.

[sudhz]

Well, I closed it because I think if you want to check for stderr and stdout separately, then you need to have two instances of TestConsole and two instances of AnsiConsole.

StdErrConsole:
AnsiConsole.Create(new AnsiConsoleSettings { Out = new AnsiConsoleOutput(Console.Error) }))

StdOutConsole:
AnsiConsole.Create(new AnsiConsoleSettings { Out = new AnsiConsoleOutput(Console.Out) }))

Then pass these two instances to your class which needs to print to stdout and stderr. When testing your class, pass in two separate instances of TestConsole respectively for the StdOutConsole and StdErrConsole.

Since I figured out this pattern, I decided to close this.

[Blackbaud-CameronBeacham]

We're using the CommandAppTester to test our commands and I did as @sudhz suggested and injected a factory that provides the stderr and stdout IAnsiConsoles. Then in my test I mocked out the GetStandardErrorConsole() and GetStandardOutConsole() methods from my factory to return distinct instances of TestConsole. I can then test each output stream independently.

However, since CommandAppTester.Run only provides a public method for passing in arguments, there is no way to get your output in the CommandResult. I don't think this is the end of the world since you can already get that via the instances returned by the mock but it would be nice if we could configure the TestOutput like the private overload exposes so that you could do something like this:

var mockAnsiConsoleFactory = new Mock<IAnsiConsoleFactory>(MockBehavior.Strict);
var testStandardErrorConsole = new TestConsole();
var testStandardOutConsole = new TestConsole();

mockAnsiConsoleFactory.Setup(m => m.GetStandardErrorConsole()).Returns(testStandardErrorConsole);
mockAnsiConsoleFactory.Setup(m => m.GetStandardOutConsole()).Returns(testStandardOutConsole);

... 

var commandResult = app.Run(new string[] {"catalog", "list", "--owner", "doesntexist"}, testStandardOutConsole);

// Assert against testStandardErrorConsole.Output
// Assert against commandResult.Output which should be equivalent to testStandardOutConsole.Output minus the extra formatting 

While I can see that this definitely works around the issue it feels like it breaks the CommandAppTester to some extent since nothing will be writing to its TestConsole. That could possibly lead to some confusing test failures.