pytest-codspeed doesn't play nice with stdout/stderr

[kenodegard]

When tests write to stdout/stderr the text ends up writing immediately to the terminal and pytest is unable to intercept it (with the capsys fixture) for inspection. Ideally pytest-codspeed wouldn't intercept stdout/stderr. I have tested this with both Python 3.12 and Python 3.11, both result in the same issue.

To reproduce:

from __future__ import annotations

import sys

import pytest
from pytest import CaptureFixture


@pytest.mark.benchmark
def test_print():
    """Test print statements are captured by pytest (i.e., not printed to terminal in
    the middle of the progress bar) and only displayed after test run (on failures)."""
    print("print to stdout")
    print("print to stderr", file=sys.stderr)


@pytest.mark.benchmark
def test_capsys(capsys: CaptureFixture):
    """Test print statements are captured by capsys (i.e., not printed to terminal in
    the middle of the progress bar) and can be inspected within test."""
    print("print to stdout")
    print("print to stderr", file=sys.stderr)

    stdout, stderr = capsys.readouterr()

    assert stdout == "print to stdout\n"
    assert stderr == "print to stderr\n"

See kenodegard#1 for benchmark run, screenshot below for posterity: