make TestLogger respect maxlog

[ericphanson]

Fixes #41625, borrowing some code from SimpleLogger and ConsoleLogger.

Note that this still doesn't work:

julia> @test_logs (:error, "an error") erronce()

julia> @test_logs erronce()

since @test_logs creates a fresh logger each time (so they don't have a shared count for maxlog).

But at least there is a way to test maxlog logging statements (as shown in the diff, using Test.collect_test_logs). And Test.collect_test_logs is possibly part of the public API since it is referenced from @test_logs? Although it doesn't have a docstring and isn't in the manual.

Edit: actually a better way to test this using only @test_logs is just:

@test_logs (:error, "an error") (erronce(); erronce())

This errors on master but passes on this branch.

[vtjnash]

This could perhaps break someone's test, but it seems unlikely. I think generally we have chosen not to parse the arguments in the TestLogger, so that the test has a true precise record of what was called (for example, we also ignore the minlevel). But it sounds like #41625 has support for changing this particular behavior.

[c42f]

I think generally we have chosen not to parse the arguments in the TestLogger, so that the test has a true precise record of what was called

Yes, I think this change is a bit questionable in that respect — perhaps a neater strategy would be to enhance the pattern matching in @test_logs rather than modify which log events are recorded. This (edit: ie, the current PR) is easier though...

[ericphanson]

perhaps a neater strategy would be to enhance the pattern matching in @test_logs rather than modify which log events are recorded.

What would you suggest?

Alternatively, I wonder how much one can really access the log events recorded via the public API- perhaps exposing a richer interface would let users test things like maxlog without needing special handling here. I’ve found @test_logs a bit difficult to use compared to Test.collect_test_logs but I don’t think that’s part of the public interface.

[ericphanson]

for example, we also ignore the minlevel

@vtjnash that's not true, right?

julia/stdlib/Test/src/logging.jl

Line 34 in 942697f

Logging.min_enabled_level(logger::TestLogger) = logger.min_level

julia> Test.collect_test_logs() do
       @debug "hi"
       end
(Test.LogRecord[], nothing)

[vtjnash]

Ah, apparently I was wrong