Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Adapter: parametrized tests are discovered/logged to data collector as 1 test (no information about specific parameters) #340

Closed
pkrukp opened this issue Dec 27, 2017 · 6 comments

Comments

@pkrukp
Copy link

pkrukp commented Dec 27, 2017

Description

Hi,

MsTest tests may be parametrizable, for example a method with DataTestMethod and DataRow attributes. For example:

        [DataTestMethod]
        [DataRow("baz", false)]
        [DataRow("foo", true)]
        public void TestMethod(string name, bool expectedResult)

Current behavior

For such test, MsTest Test Adapter:

  1. during test discovery, only 1 VSTest TestCase is discovered (which represents whole TestMethod)
  2. during test execution, the method is executed for data row, but VSTest's DataCollectors (both In-Proc and out of process) receive only 1 TestCaseStart/TestCaseStop event (for TestMethod)

Desired behavior

Would it be possible to have per data row discovery/logging resolution? That is, ideally:

  1. discovery would discover 2 test cases: TestMethod("baz", false) and TestMethod("foo", true)
  2. DataCollector would receive TestCaseStart/TestCaseStop for both above test cases

In fact, the 2nd point is more important for me - I would like to change environment and monitor each execution of TestMethod (for each data row separately).

If it is not possible, could you please share the reason?

Thanks

Environment

mstest.testadapter.1.2.0
microsoft.testplatform.15.6.0-preview-20171211-02

@pkrukp pkrukp changed the title Parametrized tests are discovered/logged to data collector as 1 test (no information about specific parameters) Test Adapter: parametrized tests are discovered/logged to data collector as 1 test (no information about specific parameters) Dec 27, 2017
@cltshivash
Copy link
Contributor

@pkruk2 Having separate tests for each datarow is not in plan for the near future. Would it be possible to check the parameter value in your test code (or) does this need to be done outside the test code ?

@pkrukp
Copy link
Author

pkrukp commented Dec 28, 2017

@cltshivash It must be outside of the test code.

In case I need to add this myself to the test adapter, do you have any suggestions? Ideally it'd be accepted upstream.

I understand it'd also change VS behavior, because currently when you discover/run such parametrized test, VS shows 1 test in Test Explorer, and all data rows results are visible in test details section.

@jayaranigarg
Copy link
Member

Tagging @pvlakshm , @cltshivash to see if and how can we bring this in?

@AbhitejJohn
Copy link
Contributor

@jayaranigarg : Another reason why it would be really nice to have separate test cases is in #171. To be able to run/debug an individual test case would greatly improve dev efficiency especially if these are long running tests.

@ShreyasRmsft
Copy link
Member

@pkruk2 this is by design and it is a non-goal to have discovery by data row.

But your ask for DataCollectors to be aware of iterations is a valid ask and can be solved (Given the constraint of not discovering each data row as a separate test method) is for us to add new events for test iteration start and test iteration end for data collectors to hook in to. But this will require a breaking change in the test Platform object model.

A simple solution is for us to have custom events in the Test Platform for datacollectors. Anyone who wants to send custom events from their adapters (in our case MsTestv2) can use this Extensible List of events to register and listen to custom events like test iteration start and test iteration end. But this won't be a simple thing to implement.

@jayaranigarg
Copy link
Member

Closing this as duplicate. Having seperate test method for each data row will fix this As mentioned in #1 of this issue : #464

singhsarab pushed a commit to singhsarab/testfx that referenced this issue Apr 8, 2019
* Fix assembly loading in symbol loader. (microsoft#332)

Instead of attempting to load the test assembly, try to query for it. It should
have been already loaded by the test adapter.

In netcoreapp1.1, attempting to load the same executable within itself throws.

* Add error tracing for symbol parse failure.

* Fix platform tests for objectmodel.

* Fix debug type. Copy manifest file and comcomponents to output directory.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants