Improve TestCaseNameParser #28

becha2 · 2021-12-08T23:16:29Z

Hello! We are running into a lot of "Unable to parse the test name" errors in our test scripts for certain kinds of parametrized tests when the inputs are valid. This adds a ton of noise in our logs, making finding actual errors difficult for our developers. Could the parser be improved? Looking at the remarks on the class, it seems there can be some big wins. I can provide some examples if needed. We run these tests in both NET Framework 4.7.2 and NET 5.0 and the errors occur in both cases.

Siphonophora · 2021-12-08T23:55:51Z

Hi, examples would be good.

becha2 · 2021-12-09T01:23:24Z

Here are some examples. I simplified some info to focus on the test case examples at hand, but please let me know if you'd need more information:

Example Test 1

[TestCaseSource(nameof(ExampleTestCases))]
public void ExampleTest(
     (string a, string b) s
    )
{
    ....
}
private static readonly IReadOnlyList<(string, string)> ExampleTestCases = new List<(string, string)> {
    (null, ""),
    ("test", "test"),
    (@"\", @"\\"),
    (@"""", @" "), // parsing error
    (@"[", @" "),
    (@"]", @" ")
};

Example Test 1 - Results in TestResults.xml

// only one under `UnknownNamespace`/`UnknownType` section
<test-case name="ExampleTest((&quot;,  ))"  ... />

// regular section
<test-case name="ExampleTest((, ))" ... />
<test-case name="ExampleTest((test, test))" ... />
<test-case name="ExampleTest((\, \\))"  ... />
<test-case name="ExampleTest(([,  ))" ... />
<test-case name="ExampleTest((],  ))" ... />

Example Test 2

[TestCaseSource(nameof(ExampleTest2Cases))]
public void ExampleTest2(
    bool a,
    decimal? b,
    decimal? c,
    (decimal?, bool) d
    )
{
    ...
}
private static IEnumerable ExampleTest2Cases()
{
    yield return new TestCaseData(false, null, 4.5m, ((decimal?)null, false));
    yield return new TestCaseData(false, 4.5m, 4.6m, ((decimal?)4.5m, false)); // parsing error
    yield return new TestCaseData(false, 4.5m, null, ((decimal?)4.5m, false));
    yield return new TestCaseData(true, 4.3m, null, ((decimal?)4.3m, false));
    yield return new TestCaseData(true, 4.1m, 5.0m, ((decimal?)5.0m, true)); // parsing error
    yield return new TestCaseData(true, 4.8m, 4.5m, ((decimal?)4.8m, false)); // parsing error
    yield return new TestCaseData(true, 4.5m, 4.5m, ((decimal?)4.5m, false)); // parsing error
}

Example Test 2 - Results in TestResult.xml

// `UnknownNamespace`/`UnknownType` section
<test-case name="ExampleTest2(False,4.5m,4.6m,(4.5, False))" fullname="UnknownNamespace.UnknownType.MyNamespace.ExampleTest.ExampleTest2(False,4.5m,4.6m,(4.5, False))" methodname="MyNamespace.ExampleTest.ExampleTest2(False,4.5m,4.6m,(4.5, False))" classname="UnknownType" ... />
<test-case name="ExampleTest2(True,4.1m,5.0m,(5.0, True))" fullname="UnknownNamespace.UnknownType.MyNamespace.ExampleTest.ExampleTest2(True,4.1m,5.0m,(5.0, True))" methodname="MyNamespace.ExampleTest.ExampleTest2(True,4.1m,5.0m,(5.0, True))" classname="UnknownType" ... />
<test-case name="ExampleTest2(True,4.8m,4.5m,(4.8, False))" fullname="UnknownNamespace.UnknownType.MyNamespace.ExampleTest.ExampleTest2(True,4.8m,4.5m,(4.8, False))" methodname="MyNamespace.ExampleTest.ExampleTest2(True,4.8m,4.5m,(4.8, False))" classname="UnknownType" ... />
<test-case name="ExampleTest2(True,4.5m,4.5m,(4.5, False))" fullname="UnknownNamespace.UnknownType.MyNamespace.ExampleTest.ExampleTest2(True,4.5m,4.5m,(4.5, False))" methodname="MyNamespace.ExampleTest.ExampleTest2(True,4.5m,4.5m,(4.5, False))" classname="UnknownType" ... />

// regular section
<test-suite type="TestSuite" name="ExampleTest" fullname="MyNamespace.ExampleTest" total="3" passed="3" failed="0" inconclusive="0" skipped="0" ... >
    <test-suite type="TestFixture" name="ExampleTest2(False,4" fullname="MyNamespace.ExampleTest.ExampleTest2(False,4" total="1" passed="1" failed="0" inconclusive="0" skipped="0" ...>
        <test-case name="ExampleTest2(False,4.5m,null,(4.5, False))" fullname="MyNamespace.ExampleTest.ExampleTest2(False,4.5m,null,(4.5, False))" methodname="5m,null,(4.5, False))" classname="ExampleTest2(False,4" ... />
    </test-suite>
    <test-suite type="TestFixture" name="ExampleTest2(False,null,4" fullname="MyNamespace.ExampleTest.ExampleTest2(False,null,4" total="1" passed="1" failed="0" inconclusive="0" skipped="0" ...>
        <test-case name="ExampleTest2(False,null,4.5m,(, False))" fullname="MyNamespace.ExampleTest.ExampleTest2(False,null,4.5m,(, False))" methodname="5m,(, False))" classname="ExampleTest2(False,null,4" .../>
    </test-suite>
    <test-suite type="TestFixture" name="ExampleTest2(True,4" fullname="MyNamespace.ExampleTest.ExampleTest2(True,4" total="1" passed="1" failed="0" inconclusive="0" skipped="0" ...>
        <test-case name="ExampleTest2(True,4.3m,null,(4.3, False))" fullname="MyNamespace.ExampleTest.ExampleTest2(True,4.3m,null,(4.3, False))" methodname="3m,null,(4.3, False))" classname="ExampleTest2(True,4" ... />
    </test-suite>
</test-suite>

Example Test 3

[TestCase('0')]
[TestCase('a')]
[TestCase('A')]
[TestCase('!')]
[TestCase('-')]
[TestCase('_')]
[TestCase('.')]
[TestCase('*')]
[TestCase('\'')]
[TestCase('(')]
[TestCase(')')]
[TestCase('/')]
public void ExampleTest3(
    char c
    )
{
    ...
}

Example Test 3 - Results in TestResults.xml

// `UnknownNamespace`/`UnknownType` section
<test-case name="ExampleTest3('(')" fullname="UnknownNamespace.UnknownType.MyNamespace.ExampleTest3('(')" methodname="MyNamespace.ExampleTest3('(')" classname="UnknownType" ... />
<test-case name="ExampleTest3(')')" fullname="UnknownNamespace.UnknownType.MyNamespace.ExampleTest3(')')" methodname="MyNamespace.ExampleTest3(')')" classname="UnknownType" ... />

// regular section
<test-case name="ExampleTest3('0')" fullname="MyNamespace.ExampleTest3('0')" methodname="ExampleTest3('0')" ... />
<test-case name="ExampleTest3('a')" fullname="MyNamespace.ExampleTest3('a')" methodname="ExampleTest3('a')" ... />
<test-case name="ExampleTest3('A')" fullname="MyNamespace.ExampleTest3('A')" methodname="ExampleTest3('A')" ... />
<test-case name="ExampleTest3('!')" fullname="MyNamespace.ExampleTest3('!')" methodname="ExampleTest3('!')" ... />
<test-case name="ExampleTest3('-')" fullname="MyNamespace.ExampleTest3('-')" methodname="ExampleTest3('-')" ... />
<test-case name="ExampleTest3('_')" fullname="MyNamespace.ExampleTest3('_')" methodname="ExampleTest3('_')" ... />
<test-case name="ExampleTest3('.')" fullname="MyNamespace.ExampleTest3('.')" methodname="ExampleTest3('.')" ... />
<test-case name="ExampleTest3('*')" fullname="MyNamespace.ExampleTest3('*')" methodname="ExampleTest3('*')" ... />
<test-case name="ExampleTest3('\'')" fullname="MyNamespace.ExampleTest3('\'')" methodname="ExampleTest3('\'')" ... />
<test-case name="ExampleTest3('/')" fullname="MyNamespace.ExampleTest3('/')" methodname="ExampleTest3('/')"... />

Example Test 4

[TestCaseSource(nameof(ExceptionTestCases))]
public void ExampleTest4(
    Exception e
    )
{
    ...
}

private static readonly Exception[] ExeptionTestCases = {
    new MyException1(),
    new MyException2(),
    new AggregateException(new MyException1()),
    new AggregateException(new MyException2())
};

Example Test 4 - Results in TestResults.xml

// `UnknownNamespace`/`UnknownType` section
<test-case name="ExampleTest4(System.AggregateException: One or more errors occurred. (Syntax error, command unrecognized.)&#xD;&#xA; ---&gt; MyNamespace.MyException1: Syntax error, command unrecognized.&#xD;&#xA;   --- End of inner exception stack trace ---)" fullname="UnknownNamespace.UnknownType.MyNamespace.ExampleTest4(System.AggregateException: One or more errors occurred. (Syntax error, command unrecognized.)&#xD;&#xA; ---&gt; MyNamespace.MyException1: Syntax error, command unrecognized.&#xD;&#xA;   --- End of inner exception stack trace ---)" methodname="MyNamespace.ExampleTest4(System.AggregateException: One or more errors occurred. (Syntax error, command unrecognized.)&#xD;&#xA; ---&gt; MyNamespace.MyException1: Syntax error, command unrecognized.&#xD;&#xA;   --- End of inner exception stack trace ---)" classname="UnknownType" ... />
<test-case name="ExampleTest4(System.AggregateException: One or more errors occurred. (Syntax error, command unrecognized.)&#xD;&#xA; ---&gt; MyNamespace.MyException2: Syntax error, command unrecognized.&#xD;&#xA;   --- End of inner exception stack trace ---)" fullname="UnknownNamespace.UnknownType.MyNamespace.ExampleTest4(System.AggregateException: One or more errors occurred. (Syntax error, command unrecognized.)&#xD;&#xA; ---&gt; MyNamespace.MyException2: Syntax error, command unrecognized.&#xD;&#xA;   --- End of inner exception stack trace ---)" methodname="MyNamespace.ExampleTest4(System.AggregateException: One or more errors occurred. (Syntax error, command unrecognized.)&#xD;&#xA; ---&gt; MyNamespace.MyException2: Syntax error, command unrecognized.&#xD;&#xA;   --- End of inner exception stack trace ---)" classname="UnknownType" ... />
        
        
// regular section
<test-case name="ExampleTest4(MyNamespace.MyException1: Syntax error, command unrecognized.)" fullname="MyNamespace.ExampleTest4(MyNamesapce.MyException1: Syntax error, command unrecognized.)" methodname="ExampleTest4(MyNamespace.MyException1: Syntax error, command unrecognized.)" ... />
<test-case name="ExampleTest4(MyNamesapce.MyException2: Syntax error, command unrecognized.)" fullname="MyNamespace.ExampleTest4(MyNamespace.MyException2: Syntax error, command unrecognized.)" methodname="ExampleTest4(MyNamespace.MyException2: Syntax error, command unrecognized.)" ... />

Example 5

Then there are these ones that work if I replace .SetName with .SetDescription. This isn't in the NUnit docs though, so I'm expecting this to work.. https://docs.nunit.org/articles/nunit/running-tests/Template-Based-Test-Naming.html

[TestCaseSource(nameof(MyTestCases))]
public void MyTest(string b)
{
    ...
}

private static IEnumerable<TestCaseData> MyTestCases()
{
    yield return new TestCaseData("a").SetName("Testing. Input.");
    yield return new TestCaseData("a").SetName("Testing. Input");
    yield return new TestCaseData("a").SetName("Testing.");
    yield return new TestCaseData("a").SetName("Testing");
}

Example 5 - Results in TestResults.xml

// `UnknownNamespace`/`UnknownType` section
<test-case name="Testing. Input." fullname="UnknownNamespace.UnknownType.MyNamespace.MyTestClass.Testing. Input." methodname="MyNamespace.MyTestClass.Testing. Input." classname="UnknownType" ... />
<test-case name="Testing." fullname="UnknownNamespace.UnknownType.MyNamespace.MyTestClass.Testing." methodname="MyNamespace.MyTestClass.Testing." classname="UnknownType" ... />

// Regular section
<test-case name="Testing" fullname="MyNamespace.MyTestClass.Testing" methodname="Testing" classname="MyTestClass" ... />
            
// In its own section
<test-suite type="TestSuite" name="Helpers" fullname="MyNamespace" total="1" passed="1" failed="0" inconclusive="0" skipped="0" ...>
    <test-suite type="TestSuite" name="MyTestClass" fullname="MyNamespace.MyTestClass" total="1" passed="1" failed="0" inconclusive="0" skipped="0" ...">
      <test-suite type="TestFixture" name="Testing" fullname="MyNamespace.MyTestClass.Testing" total="1" passed="1" failed="0" inconclusive="0" skipped="0" ...>
        <test-case name="Testing. Input" fullname="MyNamespace.MyTestClass.Testing. Input" methodname=" Input" classname="Testing" ... />
      </test-suite>
    </test-suite>
</test-suite>

Siphonophora · 2021-12-12T17:02:57Z

Hi,

So example 3 was just fixed.

Example 2 looks like a fixable issue with the parser.

Everything else seems to be facing the same problem. The parser is expecting the string it is given to parse would look like valid c#. Note that we get a single string representation of the test name from TestCase.FullyQualifiedName, which is why we are doing the parsing at all. So we expect strings to be quoted, which is why something like a.b.c((", )) vs a.b.c(("\""," ")) is a problem. It looks like the TestCaseSource data is just calling ToString() for the tuples and exceptions in 1 & 4. The cases that fail have things like parenthesis that should be in a string, somewhere the parser cant handle them. It isn't immediately clear to me how to handle this unless we have the parser fall back on guessing how to split up the string. So I will need to think about it.

For Example 5, its the same kind of issue, but I'm not sure this is something we can directly support. If the default format is overridden and it isn't something we can parse, then there isn't much we can do.

Depending on what else we come up with, we could always offer an option to do something different on a parsing failure. Presumably not adding the UnknownNamespace and UnknownType text.

For my notes:
Doesn't look like there are existing options to have the ToString thats returned escaped andquoted.
https://github.com/nunit/nunit/blob/40dcb26f8a1323a9f32fa0ebad939134222bb34a/src/NUnitFramework/framework/Internal/TestNameGenerator.cs#L524
Default representation is coming from here and just being returned. : https://github.com/nunit/nunit/blob/b34eba3ac1aa6957157857bddd116256c634afab/src/NUnitFramework/framework/Internal/TestNameGenerator.cs#L202-L204

Co-Authored-By: Rebekah Cha <[email protected]>

Updated testlogger dependency. See spekt/testlogger#28

Siphonophora · 2022-04-23T17:25:32Z

@codito Hi. I think all three loggers are ready to be released with these updates. Just wanted to see if we can do that or if you were waiting on me to do something.

codito · 2022-05-03T11:54:37Z

@Siphonophora sorry for the delay, work is a bit hectic at this time. I hope to cut a release in a week or so.

codito · 2022-05-18T01:32:44Z

@Siphonophora we have pushed junit testlogger to nuget: https://www.nuget.org/packages/JunitXml.TestLogger/3.0.114

becha2 · 2022-06-22T22:00:15Z

@Siphonophora we have pushed junit testlogger to nuget: https://www.nuget.org/packages/JunitXml.TestLogger/3.0.114

just wondering if there's any progress for pushing out an update for the nunit test logger?

Siphonophora · 2022-06-22T22:42:26Z

@becha2 I didn't realize that spekt/nunit.testlogger#93 was waiting on me. As soon as it gets merged there will be a pre-release version in myget.

codito · 2022-09-10T02:31:34Z

https://www.nuget.org/packages/NunitXml.TestLogger/3.0.127 is also released.

becha2 · 2022-10-20T03:04:06Z

awesome! thank you

Siphonophora added a commit to Siphonophora/testlogger that referenced this issue Feb 27, 2022

Pull in examples from spekt#28

df7b402

Co-Authored-By: Rebekah Cha <[email protected]>

Siphonophora mentioned this issue Feb 27, 2022

Overhaul parser & Test all supported test frameworks #30

Merged

This was referenced Mar 27, 2022

Use regex parser spekt/nunit.testlogger#93

Merged

Use regex parser spekt/xunit.testlogger#38

Merged

Bump logger to use regex parser spekt/junit.testlogger#52

Merged

codito pushed a commit to spekt/xunit.testlogger that referenced this issue Apr 3, 2022

feat: use regex parser (#38)

7d6b7d5

Updated testlogger dependency. See spekt/testlogger#28

becha2 closed this as completed Oct 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve TestCaseNameParser #28

Improve TestCaseNameParser #28

becha2 commented Dec 8, 2021 •

edited

Loading

Siphonophora commented Dec 8, 2021

becha2 commented Dec 9, 2021

Siphonophora commented Dec 12, 2021

Siphonophora commented Apr 23, 2022

codito commented May 3, 2022

codito commented May 18, 2022

becha2 commented Jun 22, 2022

Siphonophora commented Jun 22, 2022

codito commented Sep 10, 2022

becha2 commented Oct 20, 2022

Improve TestCaseNameParser #28

Improve TestCaseNameParser #28

Comments

becha2 commented Dec 8, 2021 • edited Loading

Siphonophora commented Dec 8, 2021

becha2 commented Dec 9, 2021

Example Test 1

Example Test 1 - Results in TestResults.xml

Example Test 2

Example Test 2 - Results in TestResult.xml

Example Test 3

Example Test 3 - Results in TestResults.xml

Example Test 4

Example Test 4 - Results in TestResults.xml

Example 5

Example 5 - Results in TestResults.xml

Siphonophora commented Dec 12, 2021

Siphonophora commented Apr 23, 2022

codito commented May 3, 2022

codito commented May 18, 2022

becha2 commented Jun 22, 2022

Siphonophora commented Jun 22, 2022

codito commented Sep 10, 2022

becha2 commented Oct 20, 2022

becha2 commented Dec 8, 2021 •

edited

Loading