Skip to content

Commit

Permalink
Document & expose printing of custom AbstractTestSet (#53215)
Browse files Browse the repository at this point in the history
This was previously alluded to in the docs as being possible ("just
record if we're not the top-level parent"), but trying to do that didn't
actually do anything because the printing in the `DefaultTestSet`
ignored anything that wasn't a `Test.Result` or `Test.DefaultTestSet`.
This is particularly problematic if any of the child-testsets had
failures, because those failures would never be shown to a user.

Co-authored-by: Sukera <[email protected]>
Co-authored-by: Ian Butterworth <[email protected]>
  • Loading branch information
3 people authored Feb 9, 2024
1 parent 65d4ce2 commit 9523361
Show file tree
Hide file tree
Showing 4 changed files with 328 additions and 75 deletions.
45 changes: 44 additions & 1 deletion stdlib/Test/docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -322,8 +322,12 @@ function finish(ts::CustomTestSet)
# just record if we're not the top-level parent
if get_testset_depth() > 0
record(get_testset(), ts)
return ts
end
ts

# so the results are printed if we are at the top level
Test.print_test_results(ts)
return ts
end
```

Expand All @@ -338,6 +342,45 @@ And using that testset looks like:
end
```

In order to use a custom testset and have the recorded results printed as part of any outer default testset,
also define `Test.get_test_counts`. This might look like so:

```julia
using Test: AbstractTestSet, Pass, Fail, Error, Broken, get_test_counts, TestCounts, format_duration

function Test.get_test_counts(ts::CustomTestSet)
passes, fails, errors, broken = 0, 0, 0, 0
# cumulative results
c_passes, c_fails, c_errors, c_broken = 0, 0, 0, 0

for t in ts.results
# count up results
isa(t, Pass) && (passes += 1)
isa(t, Fail) && (fails += 1)
isa(t, Error) && (errors += 1)
isa(t, Broken) && (broken += 1)
# handle children
if isa(t, AbstractTestSet)
tc = get_test_counts(t)::TestCounts
c_passes += tc.passes + tc.cumulative_passes
c_fails += tc.fails + tc.cumulative_fails
c_errors += tc.errors + tc.cumulative_errors
c_broken += tc.broken + tc.cumulative_broken
end
end
# get a duration, if we have one
duration = format_duration(ts)
return TestCounts(true, passes, fails, errors, broken, c_passes, c_fails, c_errors, c_broken, duration)
end
```

```@docs
Test.TestCounts
Test.get_test_counts
Test.format_duration
Test.print_test_results
```

## Test utilities

```@docs
Expand Down
Loading

0 comments on commit 9523361

Please sign in to comment.