Introduce first-class support for scenario tests

[sbrannen]

Proposal

The current proposal is to introduce the following:

• @ScenarioTest: class-level annotation used to denote that a test class contains steps that make up a single scenario test.
• @Step: method-level annotation used to denote that a test method is a single step within the scenario test.

The @Step annotation will need to provide an attribute that can be used to declare the next step within the scenario test. Steps will then be ordered according to the resulting dependency graph and executed in exactly that order.

For example, a scenario test could look similar to the following:

@ScenarioTest
class WebSecurityScenarioTest {

    @Step(next = "login")
    void visitPageRequiringAuthorizationWhileNotLoggedIn() {
        // attempt to visit page which requires that a user is logged in
        // assert user is redirected to login page
    }

    @Step(next = "visitSecondPageRequiringAuthorizationWhileLoggedIn")
    void login() {
        // submit login form with valid credentials
        // assert user is redirected back to previous page requiring authorization
    }

    @Step(next = "logout")
    void visitSecondPageRequiringAuthorizationWhileLoggedIn() {
        // visit another page which requires that a user is logged in
        // assert user can access page
    }

    @Step(next = END)
    void logout() {
        // visit logout URL
        // assert user has been logged out
    }

}

Related Issues

• Provide mechanism to order the sequence of tests #13
• Allow @BeforeAll and @AfterAll methods to be non-static #419
• Introduce mechanism for inter-test dependencies #607
• Running tests in certain order #884

[mmerdes]

@sbrannen: do you prefer referencing the next step by name? i assume these references would be refactoring-safe, so no real problem here. but what about typos? (i guess just using numbered steps will be awkward if you introduce steps in the middle...)

[jlink]

Is it in scope for Alpha1 (end of next week)?

[bechte]

Thanks for writing down the concept and providing a detailed example. I think it is great and I also like the next attribute pointing to the methodName. Two reasons:

• IDE refactoring of the method name should also replace the next attribute string
• IDE vendors will have a nice and easy way to support auto-completion for steps

I still wonder if we want to point to the next or rather to the previous step? From a reader's perspective, I would like to have an idea what are the preconditions for this step? Especially as the annotation stands above the method declaration, I would suggest using a "previous" or "ancester" attribute. But it's just a feeling.

Concerning the Alpha-M1: I don't think it will be part of it. We should probably move it to a different milestone. What do you think, @sbrannen ?

[jlink]

Why not use preconditions as the concept to determine the order? A step can
have none, one or many. A runner could thereby parallelize test execution
as much as possible. And one could determine the minimum set of steps that
must be executed for a certain step to make sense at all.

2015-12-04 11:24 GMT+01:00 Stefan Bechtold [email protected]:

Thanks for writing down the concept and providing a detailed example. I
think it is great and I also like the next attribute pointing to the
methodName. Two reasons:

• IDE refactoring of the method name should also replace the next
  attribute string
• IDE vendors will have a nice and easy way to support auto-completion
  for steps

I still wonder if we want to point to the next or rather to the previous
step? From a reader's perspective, I would like to have an idea what are
the preconditions for this step? Especially as the annotation stands above
the method declaration, I would suggest using a "previous" or "ancester"
attribute. But it's just a feeling.

—
Reply to this email directly or view it on GitHub
#48 (comment)
.

[bechte]

Yes, this will be a consequence of turning from "next" to "ancestor". Preconditions are great, but we need to be aware of the added complexity. Maybe we discuss this topic in person? Anyways, I like this feature. 👍

[dinesh707]

Referring a method by string sounds like way to do more mistakes and as @bechte commented it will not give IDE support when coding. I would suggest something like following

final String LOGIN_ID = "login_id";
final String LOGOUT_ID = "logout_id";
final String VISIT_PAGE_ID = "visit_page_id";

@step(id = LOGOUT_ID, waitFor = VISIT_PAGE)

so the waitFor will wait till VISIT_PAGE is completed. That way executor should be able to plan concurrent executions as well.

But again I really see the benefit of the original proposal

[sormuras]

Utilizing a simple state machine (using enums instead of strings) is an overkill, right?
Like https://github.com/oxo42/stateless4j or others.

[bechte]

I was trying to say that I like the method name references. I would like to avoid introducing another ID for the tests. I think IDE vendors are great for picking up method names (outline view, etc.) and they will easily support refactorings and support test writers with auto-completion. This can all happen on static code analysis and, therefore, is something they can support right-away.

At least as a test writer, I dont want to maintain the list of IDs. It's boilerplate anyways. We might think about support the test name:

@Test(name = "Step1") 

and allow referencing the name as well. But for a first alpha, I would stick with the easy method name approach provided by @sbrannen .

[sbrannen]

@jlink and @bechte, scheduling this issue for Alpha M1 was intentionally optimistic.

Whether or not it's possible for Alpha M1 really depends on how far we get with the rewrite of the JUnit 5 engine -- for example, whether we have extension points for test instance lifecycle and ordering of tests.

If we don't get far enough in Alpha M1, we will naturally push the implementation of this issue to a subsequent milestone. However, we need to make sure that everyone has this on his radar; otherwise, it will likely be more challenging to integrate this feature as an afterthought.

[dsaff]

If this wasn't first-class supported, how hard would it be to add via a
current extension point?

David Saff

[sbrannen]

@mmerdes, @bechte, @jlink, @dinesh707,

Thanks for the feedback on ordering and dependencies!

The reason I have thus far only proposed an @Step annotation with a next attribute is to keep it as simple as possible for the first round of implementation.

To address the issues you have raised...

Depending on how we decide to support ordering in general (i.e., for standard @Test methods), we may find a common mechanism for ordering steps in a scenario test as well. For example, if we introduce something like @Order for @Test methods, we could optionally reuse that for @Step methods.

The difference is that an annotation like @Order is typically based on numerical ordering. If you initially, intentionally define gaps between the numbers you declare (e.g., 10, 20, 30 instead of 1, 2, 3), it is then possible to easily insert another test or step in one of those gaps. In that sense numerical ordering is rather flexible though at the same time not as precise. For steps in a scenario test, however, I imagine people will wish to be very precise about the dependencies between steps.

As for whether a step declares its dependency by referencing the previous step vs. the next step, I think it will become more clear what the better solution is once we begin to experiment more with this feature.

With regard to supporting a precondition mechanism, that would certainly increase the flexibility of step execution within a scenario test, but I think we should first focus on a simpler approach before adding the level of complexity that would be required to support that correctly.

Along the same line of thinking, for the initial draft of this feature I would recommend that we not concern ourselves with parallel execution of steps within a scenario test.

[sbrannen]

@dsaff,

With the current set of extension points, it would be impossible to implement a feature like this as an extension.

However, that may soon change, depending on the outcome of the current effort to redesign the test discovery and test execution models.

In general, our goal is to introduce as many extension points that we deem feasible. Thus, if we create the necessary extension points in the engine to make it possible to implement scenario tests as an extension, we would naturally opt to implement it as an extension even if the feature is available out-of-the-box. That's analogous to how we implemented @Disabled and @TestName.

[sbrannen]

@sormuras,

Yes, I think that using or implementing a true state machine would be overkill for this feature.

[sbrannen]

@bechte,

I think you meant:

@Name("Step1")
@Step

instead of:

@Test(name = "Step1")

Right?

[bechte]

@sbrannen True 👍 - Thanks ;-)

[luontola]

Here is an idea for an alternative way of defining test order, using the normal rules of code execution order: http://specsy.org/documentation.html#non-isolated-execution-model

That way requires that tests can be nested and that tests are declared as method calls which take a lambda as parameter, so it can be hard to fit it together with the JUnit style of declaring tests as method declarations.

[sbrannen]

@orfjackal, thanks for sharing the link to Specsy's Non-Isolated Execution Model.

shareSideEffects() does in fact support the same underlying runtime model that we aim to achieve; however, you are also correct that it does not align with the standard Java class and method based approach used in JUnit 4 and JUnit 5.

Since the JDK does not guarantee that methods are returned in the order in which they are declared when using reflection, we have to introduce an explicit mechanism to control the order. So unless we introduce an alternative programming model based solely on lambdas (which has in fact been discussed), we unfortunately won't be able to support a programming model as simple as Specsy in this regard.

[thomasdarimont]

Interesting discussion - I just wanted to throw in two alternative ideas that came to my mind:

One alternative for referring to test methods via Strings or explicit lambdas could be method-references.
It's a "bit" more involved but would look less verbose than lambdas and could also be checked by the compiler.
Here is an example for this: https://gist.github.com/thomasdarimont/eb1bc1d24ccf3decbdc6
(Though I admit that this is more of a hack...)

Another idea is to create a proxied mock object of the current test class that would allow you to specify the
order of test executions via the invocation order of the mock methods. This is similar to what Mocking frameworks
like mockito support but here one would describe the actual "execution plan" for the test:

    @ScenarioSetup
    public void simpleScenario(WebSecurityScenarioTest mock){

        mock.visitPageRequiringAuthorizationWhileNotLoggedIn();
        mock.login();
        mock.visitSecondPageRequiringAuthorizationWhileLoggedIn();
        mock.logout();
    }

[jillesvangurp]

Why not rename @ScenarioTest to @Test? I don't see the need for introducing a completely new annotation. Currently junit doesn't have any class level @Test, like testng does have. I've always thought this was a strange omission and junit 5 looks like it could be a nice moment to fix that. The meaning of this is simple: every public method in this class is a test (unless annotated with BeforeXXX/AfterXXX). This would eliminate a great deal of verbosity since it allows you to leave out redundant @Test declarations on each method.

I would also change the @Step annotation to @Next("testName"). The testName should match to either the methodName or the @Name if configured. For the last step, the next would not be needed.

Any test methods with @Next would be guaranteed to be executed in the chained order. Any other tests are executed before or after in whatever default order configured for junit. Together with the class level @Test this gives you the least verbose way to do scenario testing. Combined with nested tests as proposed for junit5 this could actually provide something that closely resembles rspec.

Also having chained test methods finally gives us something like @BeforeEach and @AfterEach because each chain implictly has a first and last method without requiring the use of static methods (which I've always found a very unfortunate decision in junit). But maybe, we can make this more explicit by introducing @BeforeScenario and @AfterScenario.

[jlink]

Sounsd like a good use case for a test engine of your own?

I wouldn't like to overload the meaning of @test within the JUnit 5 engine.

Am 31.12.2015 um 11:16 schrieb Jilles van Gurp [email protected]:

Why not rename @scenariotest to @test? I don't see the need for introducing a completely new annotation. Currently junit doesn't have any class level @test, like testng does have. I've always thought this was a strange omission and junit 5 looks like it could be a nice moment to fix that. The meaning of this is simple: every public method in this class is a test (unless annotated with Before/After). This would eliminate a great deal of verbosity since it allows you to leave out redundant @test declarations on each method.

I would also change the @step annotation to @Next("testName"). The testName should match to either the methodName or the @name if configured. For the last step, the next would not be needed.

Any test methods with @Next would be guaranteed to be executed in the chained order. Any other tests are executed before or after in whatever default order configured for junit. Together with the class level @test this gives you the least verbose way to do scenario testing. Combined with nested tests as proposed for junit5 this could actually provide something that closely resembles rspec.

Also having chained test methods finally gives us something like @beforeeach and @AfterEach without requiring the use of static methods like testng provides (which I've always found a very unfortunate decision in junit). But maybe, we can make this more explicit by introducing @BeforeScenario and @AfterScenario.

—
Reply to this email directly or view it on GitHub.

[jillesvangurp]

In my view it wouldn't overload @Test at all but merely give it exactly the same meaning except at the class level; just like testng has provided for some time. In junit3, you could actually extend TestCase to achieve the same goal.

The second part of what I propose is a somewhat less verbose alternative to what @sbrannen proposed and would simply allow you to chain together test methods using @Next; which is pretty much the minimum needed to support scenarios. One benefit here is that nothing else changes and all the other planned changes (e.g. dependency injection via parameters) and annotations should work here for scenarios as well.

[bechte]

Three things I would like to add to the discussion:

1. Of course we could simply use @Test on the class level to minimize the number of annotations that JUnit serves. But I don't see a good reason for doing so. We are not aiming on to minimize the number of annotations, but we are maximizing the flexibility and the expressiveness of JUnit itself. In this manner, I think it is a good thing to introduce a separate annotation that clarifies that we are testing scenarios, i.e. @ScenarioTest. I really think, this is a good thing. Anyways, we still could change the name to something else (even though I like it the way it is).

2. We could introduce another engine that supports scenarios but this also means that this engine is required to provide its own semantics (probably using its own annotations).

3. The @Step annotation with a next attribute is an early draft. A @Next annotation would also solve this issue of finding out what the next step is in the scenario. Still, we should also think about some form of preconditions. I prefer specifying previous steps, i.e. what are the preconditions for this actual step to run? In this way, we don't get a list of steps, but a tree of steps, which I think is more flexible. But we might find it to hard to solve the problem internally and stick by the list approach. Just to mention...

I really think we should take this issue and have a group discussion within the dev team in January.

[binkley]

Hi, I'm encountering an error which directly references this issue by link. Gradle says:

* What went wrong:
Execution failed for task ':basilisk-contracts:generateContractTests'.
> Not implemented yet in JUnit5 - https://github.com/junit-team/junit5/issues/48

I'm using Spring Cloud Contract (SCC), which places all the individual contract tests into a single class, relying on an inherited test base class for setup.

I have no annotations on the test methods (SCC marks the methods with @Test from JUnit 5), and the base class of the generated test class has @ExtendsWith(SpringExtension.class), inherited as a meta annotation.

I'm surprised to see this junit failure message.

If this is the wrong place to ask about this, I'm happy to ping Spring Contract. Since the error message references here, I thought I'd ask here first.

[binkley]

Some more digging. This is definitely SCC, not JUnit, complaining:

https://github.com/spring-cloud/spring-cloud-contract/blob/8ce80e6e7fc0267a9cba1679bdf20f42ce875741/spring-cloud-contract-verifier/src/main/groovy/org/springframework/cloud/contract/verifier/config/framework/JUnit5Definition.groovy#L63

SCC believes this @TestMethodOrder(Alphanumeric.class) is unsupported. To be fair, the annotation is marked since = "5.4".

I'll talk to them.

[panchenko]

This issue looks almost completed: @TestInstance(Lifecycle.PER_CLASS) and @TestMethodOrder(MethodOrderer.OrderAnnotation.class) would work for arranging the order of the steps.

The only additional thing which is missing IMHO: skip the remaining tests after a failure.
#431 looks related, but such functionality should be generic, not DynamicTest specific.

[mkobit]

@panchenko I think you are right.

Here is my first attempt I have been trying out (haven't tried with nested tests or anything complex):

import org.junit.jupiter.api.MethodOrderer
import org.junit.jupiter.api.TestInstance
import org.junit.jupiter.api.TestMethodOrder
import org.junit.jupiter.api.extension.ConditionEvaluationResult
import org.junit.jupiter.api.extension.ExecutionCondition
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.TestExecutionExceptionHandler

// https://github.com/junit-team/junit5/issues/48 - Introduce first-class support for scenario tests
// https://github.com/junit-team/junit5/issues/431 - Introduce mechanism for terminating Dynamic Tests early

private class StepwiseExtension : ExecutionCondition, TestExecutionExceptionHandler {
  override fun handleTestExecutionException(context: ExtensionContext, throwable: Throwable) {
    val namespace = namespaceFor(context)
    val store = storeFor(context, namespace)
    store.put(StepwiseExtension::class, context.displayName)
    throw throwable
  }

  override fun evaluateExecutionCondition(context: ExtensionContext): ConditionEvaluationResult {
    val namespace = namespaceFor(context)
    val store = storeFor(context, namespace)
    val value: String? = store.get(StepwiseExtension::class, String::class.java)
    return if (value == null) {
      ConditionEvaluationResult.enabled("No test failures in stepwise tests")
    } else {
      ConditionEvaluationResult.disabled("Stepwise test disabled due to previous failure in '$value'")
    }
  }

  private fun namespaceFor(context: ExtensionContext): ExtensionContext.Namespace =
    ExtensionContext.Namespace.create(StepwiseExtension::class, context.parent)

  private fun storeFor(context: ExtensionContext, namespace: ExtensionContext.Namespace): ExtensionContext.Store =
    context.parent.get().getStore(namespace)
}


@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
@ExtendWith(StepwiseExtension::class)
@Target(AnnotationTarget.CLASS)
@Retention(AnnotationRetention.RUNTIME)
annotation class Stepwise

import org.junit.jupiter.api.Order
import org.junit.jupiter.api.Test

@Stepwise
internal class ScenarioExample {
  @Test
  @Order(1)
  internal fun first() {
    println("1")
  }

  @Test
  @Order(6)
  internal fun sixth() {
    println("6")
    throw IllegalStateException("Should be skipped")
  }

  @Test
  @Order(5)
  internal fun fifth() {
    println("5")
    throw IllegalStateException("Should be skipped")
  }

  @Test
  @Order(3)
  internal fun third() {
    println("3")
  }

  @Test
  @Order(2)
  internal fun second() {
    println("2")
  }

  @Test
  @Order(4)
  internal fun fourth() {
    println("4")
    throw IllegalArgumentException("FAILURE")
  }
}

Which fails on the fourth() test and disables fifth() and sixth().

[marcphilipp]

@mkobit Thanks for sharing, that looks promising! It should use TestWatcher instead of TestExecutionExceptionHandler to make sure it catches all test failures.

[rliesenfeld]

My two cents, since I just implemented a feature for scenario-oriented tests in my own testing tool (which I currently use with JUnit 4, but should migrate to JUnit 5 eventually).

As a user, I would prefer to simply put @ScenarioTest or @Stepwise on my test classes, and use regular @Test methods for the scenario steps. Having to explicitly specify a numerical order, or worse, by test name, is just too cumbersome.

Test (step) ordering should respect the textual order I write the tests in the test class. This can be implemented without much difficulty using ASM. (I just did it, hacking JUnit 4's TestClass, but probably won't release this in my tool.) The Java classfile normally preserves textual order, but even if it didn't, textual ordering could still be guaranteed by reading the "LineNumberTable" classfile attribute with an ASM MethodVisitor.

[sbrannen]

Test (step) ordering should respect the textual order I write the tests in the test class. This can be implemented without much difficulty using ASM. (I just did it, hacking JUnit 4's TestClass, but probably won't release this in my tool.) The Java classfile normally preserves textual order, but even if it didn't, textual ordering could still be guaranteed by reading the "LineNumberTable" classfile attribute with an ASM MethodVisitor.

That's related to the discussion here: #1919 (comment)

[vab2048]

Is it a goal of this feature to support mapping from a Gherkin feature file to the relevant scenario classes?
Or any sort of first class integration with cucumber?

[marcphilipp]

Is it a goal of this feature to support mapping from a Gherkin feature file to the relevant scenario classes?

No, this is about providing sth. similar to Spock's @Stepwise in JUnit Jupiter.

Or any sort of first class integration with cucumber?

AFAIK Cucumber already provides a custom TestEngine so integration should no longer be an issue or am I missing sth.?

[marcin-chwedczuk]

Extension provided by @mkobit works fine for me. Here is Java version:

package pl.marcinchwedczuk.javafx.validation.demo.utils;

import org.junit.jupiter.api.extension.ConditionEvaluationResult;
import org.junit.jupiter.api.extension.ExecutionCondition;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.api.extension.TestExecutionExceptionHandler;

// https://github.com/junit-team/junit5/issues/48 - Introduce first-class support for scenario tests

class StopOnFirstFailureExtension implements ExecutionCondition, TestExecutionExceptionHandler {
    @Override
    public void handleTestExecutionException(ExtensionContext context, Throwable throwable) throws Throwable {
        try {
            ExtensionContext.Namespace namespace = namespaceFor(context);
            ExtensionContext.Store store = storeFor(context, namespace);
            store.put(StopOnFirstFailureExtension.class, context.getDisplayName());
        } catch (Exception e) {
            e.printStackTrace();
        }

        throw throwable;
    }

    @Override
    public ConditionEvaluationResult evaluateExecutionCondition(ExtensionContext context) {
        var namespace = namespaceFor(context);
        var store = storeFor(context, namespace);

        var value = store.get(StopOnFirstFailureExtension.class, String.class);
        if (value == null) {
            return ConditionEvaluationResult.enabled("No test failures in stepwise tests");
        } else {
            return ConditionEvaluationResult.disabled(
                    "Stepwise test disabled due to previous failure in '" + value + "'");
        }
    }

    private ExtensionContext.Namespace namespaceFor(ExtensionContext context) {
        return ExtensionContext.Namespace.create(StopOnFirstFailureExtension.class, context.getParent());
    }

    private ExtensionContext.Store storeFor(ExtensionContext context, ExtensionContext.Namespace namespace) {
        return context.getParent().get().getStore(namespace);
    }
}

@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@ExtendWith(StopOnFirstFailureExtension.class)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface StopOnFirstFailure { }

[marcphilipp]

@nipafx Could this be a feature for JUnit Pioneer?

[jbduncan]

For anyone stumbling upon this, I've found the Test Robot Pattern a great way of expressing "steps" or "scenarios" inside a test in a readable manner. My current team uses it in an Android project to great effect, and the pattern also applies to things like Selenium tests.

[nipafx]

@nipafx Could this be a feature for JUnit Pioneer?

Yes. 😁 I assume that means that Jupiter will not implement this (for now)?

[marcphilipp]

It seems there's a relatively simple way to achieve this using already existing extensible mechanisms so I would tend not to. 🙂

[Saljack]

I do not understand why don't you want to add it directly to JUnit? Or do you want to test it in JUnit Pioneer and then move it to JUnit? Or are there another reasons? Because I think this is one of the most wanted feature.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

[igilbert]

Note that the workaround offered by StopOnFirstFailureExtension is quite different with inter-test dependency. The latter should work like this: when running a step3 test, its dependence tests would run as well.
this happens when debugging step3 test, while the workaround would not run step1-2 tests at all.

[marcphilipp]

For anyone landing on this page, looking for a complete example of a StopOnFirstFailureExtension, please see #3099 (comment).