Introduced CucumberJUnitRunner - a bridge between Cucumber Features and Android Test Orchestrator #14

lifedemons · 2019-02-15T00:19:15Z

This PR proposes the way of integration between Android Test Orchestrator(e.g. AndroidJUnitRunner) and Cucumber for Android.
Essentially:

Cucumber operates with PickleEvents - representation of tests
Android Test Orchestrator operates with Junit Tests/Runners

So, we map List of PickleEvents to List of JUnit Description via implementing of custom CucumberJUnitRunner

As result, this allows:

to shard tests on Cucumber
to use Spoon dynamic sharding with Cucumber - when Spoon detects all connected devices and distributes tests between them

…nd JUnit tests

lifedemons · 2019-02-15T00:33:58Z

cucumber-android/src/main/java/cucumber/api/android/CucumberAndroidJUnitRunner.java

+    @Override
+    public void onCreate(final Bundle bundle) {
+        bundle.putString("plugin", getPluginConfigurationString()); // we programmatically create the plugin configuration
+        bundle.putString(ARGUMENT_ORCHESTRATOR_RUNNER_BUILDER, CucumberJUnitRunnerBuilder.class.getName());


we need to pass CucumberJUnitRunnerBuilder to AndroidJUnitRunner so then it will automatically be asked to create CucumberJUnitRunner

lifedemons · 2019-02-15T00:36:54Z

cucumber-android/src/main/java/cucumber/runtime/android/CucumberExecutor.java

    }

    /**
     * Runs the cucumber scenarios with the specified arguments.
     */
    public void execute() {
+        instrumentationReporter.setNumberOfTests(getNumberOfConcreteScenarios());


we need to defer a moment of reporting of total tests to right before the execution, because then filtering of tests can be applied:
fcd70d3#diff-9d77d6e39a9caffbbd7c7854efbb2fe0R67

lifedemons · 2019-02-15T00:38:34Z

cucumber-android/src/main/java/cucumber/runtime/android/CucumberJUnitRunner.java

+    public CucumberJUnitRunner(Class testClass) {
+        instrumentationRunner = (CucumberAndroidJUnitRunner) InstrumentationRegistry
+                .getInstrumentation();
+        cucumberExecutor = new CucumberExecutor(new Arguments(instrumentationRunner.getArguments()),


Bundle arguments are stored in CucumberAndroidJUnitRunner for backward compatibility with Cucumbers tags/configuring

lifedemons · 2019-02-15T00:41:04Z

cucumber-android/src/main/java/cucumber/runtime/android/CucumberJUnitRunnerBuilder.java

+public class CucumberJUnitRunnerBuilder extends RunnerBuilder {
+    @Override
+    public Runner runnerForClass(final Class<?> testClass) {
+        if (testClass.equals(this.getClass())) {


it is needed for this Builder to work only once - so we run it one Class from tests Classpath - for itself

lifedemons · 2019-02-15T00:41:27Z

cukeulator/build.gradle

@@ -96,6 +96,7 @@ spoon {
    debug = true
    singleInstrumentationCall = true
    grantAll = true
+    shard = true


dynamic Spoon sharding

lifedemons · 2019-02-15T10:16:00Z

it is strange, but the project doesn't compile for me on cucumber java 4.2.3-SNAPSHOT or 4.2.3 but compiles fine on 4.2.2

mpkorstanje · 2019-02-17T21:02:23Z

There have been some internal API changes in 2.4.3. Glue and feature paths are now passed as URI's internally. This avoids having to deal with arbitrary user input everywhere.

lsuski · 2019-02-17T21:52:16Z

I'll try to check this in next week

…_orchestrator_integration

lsuski · 2019-02-19T17:02:34Z

I'm currently making small refactoring in this PR: removing duplicates and migrating to androidx and this runner is not working, likely due to changes in androidx.test classes. I'll figure this out and push my changes in few days

lsuski · 2019-02-20T09:55:45Z

This branch should compile now but it does not execute any tests (at least running from Android Studio). @lifedemons did you tested it? In my opinion it can't work here

 if (testClass.equals(getClass())) {
            return new CucumberJUnitRunner(testClass);
        }

because there is a filter which filters out classes not belonging to test package.

lifedemons · 2019-02-20T22:07:14Z

@lsuski cukeulator example runs for me after your committed changes from Android Studio and from gradle command connectedAndroidTest:

lifedemons · 2019-02-20T22:08:46Z

also if you execute spoonDebugAndroidTest then you should get parallel execution with a report:

lifedemons · 2019-02-20T22:14:22Z

in order to supply CucumberJUnitRunner I used CucumberJUnitRunnerBuilder inside CucumberAndroidJUnitRunner, so CucumberAndroidJUnitRunner should be defined as testInstrumentationRunner "cucumber.api.android.CucumberAndroidJUnitRunner".

So, this CucumberJUnitRunner instance is created only once per Features run and then it provides the full list of JUnit Descriptions in the function:

@Override
    public Description getDescription() {
        Description rootDescription = Description.createSuiteDescription("All tests", 1);

        for (PickleEvent pickleEvent : pickleEvents) {
            rootDescription.addChild(makeDescriptionFromPickle(pickleEvent));
        }
        return rootDescription;
    }

lifedemons · 2019-02-20T22:18:53Z

so, please, let me know does clean cukeulator example run for you on the current latest commit

lsuski · 2019-02-21T09:23:09Z

Yes, it works, I've run tests in AS for package and that's was the case. However did you tested it with Android Orchestrator (as PR title suggests) because current configuration does not use it.
Android Test Orchestrator != AndroidJunitRunner

lsuski · 2019-02-21T10:01:32Z

I wasn't able to make it work with orchestrator, however it's not critical feature for me. Sharding is much more useful with large test base

…der from duplicated code, remove AndroidObjectFactory

lifedemons · 2019-02-21T16:24:12Z

@lsuski I agree with you about importancy of sharding vs isolated running of all tests via Android Test Orchestrator, so this PR was intended to bring sharding.
But I will try today to make it compatible with the isolated/orchestrated running of all tests via Android Test Orchestrator.

lifedemons · 2019-02-21T16:25:32Z

@lsuski thank you for your feedback 👍

lifedemons · 2019-02-22T00:33:42Z

@lsuski I've made some progress in Android Test Orchestrators' isolated test Instrumentations:
lifedemons@6c328ef

What I see here:

Android Test Orchestrator(ATO) runs first time for gathering list of tests
Then ATO creates new Instrumentation for every test from the list and feeds "class" Bundle argument
The example of this Bundle Argument after test run of CucumberAndroidJUnitRunner is
class 'features_operations_subtraction.feature#Enter two digits per number and press = 2'
we can't pass this Bundle Argument to AndroidJUnitRunner because it attempts to create an instance of this test Class, so the idea is to intercept it and filter on Cucumber side

what do you think?

lsuski · 2019-02-22T08:15:16Z

Cucumber has its own filtering mechanism and I think we should use it instead. I'll try it today

lsuski · 2019-02-22T16:51:39Z

I've made some enhancements and this should finally work. I'll push my changes here after weekend

…le with test orchestrator

lsuski · 2019-02-26T12:56:45Z

Everything should work now. @lifedemons check this with your project

lifedemons · 2019-02-26T21:33:43Z

@lsuski I've checked - works fine. great job 👍

aslakhellesoy · 2019-02-27T07:48:08Z

Hi @lifedemons,

Thanks for your making your first contribution to Cucumber, and welcome to the Cucumber committers team! You can now push directly to this repo and all other repos under the cucumber organization! 🍾

In return for this generous offer we hope you will:

✅ Continue to use branches and pull requests. When someone on the core team approves a pull request (yours or someone else's), you're welcome to merge it yourself.
💚 Commit to setting a good example by following and upholding our code of conduct in your interactions with other collaborators and users.
💬 Join the community Slack channel to meet the rest of the team and make yourself at home.
ℹ️ Don't feel obliged to help, just do what you can if you have the time and the energy.
🙋 Ask if you need anything. We're looking for feedback about how to make the project more welcoming, so please tell us!

On behalf of the Cucumber core team,
Aslak Hellesøy
Creator of Cucumber

VikOles · 2021-03-10T13:44:38Z

Hi, @lifedemons, @aslakhellesoy. I have an issue with implementation of cucumber with the orchestrator. Could you provide the setting which you use for it?

Introduced CucumberJUnitRunner - a bridge between Cucumber Features a…

fcd70d3

…nd JUnit tests

lifedemons force-pushed the feature/2/android_orchestrator_integration branch from f7f347e to fcd70d3 Compare February 15, 2019 00:23

lifedemons commented Feb 15, 2019

View reviewed changes

lifedemons mentioned this pull request Feb 15, 2019

cucumber-android does not integrate very well with Android Orchestrator #2

Closed

Merge remote-tracking branch 'upstream/master' into feature/2/android…

3156074

…_orchestrator_integration

migrate to androidx, set targetSdk to 28, extract UniqueTestNameProvi…

f49f594

…der from duplicated code, remove AndroidObjectFactory

lsuski added 3 commits February 26, 2019 13:24

migrate to AndroidJunitRunner, remove CucumberExecutor, make compatib…

ef9cd1a

…le with test orchestrator

set singleInstrumentationCall = true

3c47bc2

fix AndroidResourceLoaderTest test

a42bd59

lsuski merged commit 15cbf8f into cucumber:master Feb 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduced CucumberJUnitRunner - a bridge between Cucumber Features and Android Test Orchestrator #14

Introduced CucumberJUnitRunner - a bridge between Cucumber Features and Android Test Orchestrator #14

lifedemons commented Feb 15, 2019

lifedemons Feb 15, 2019

lifedemons Feb 15, 2019

lifedemons Feb 15, 2019

lifedemons Feb 15, 2019

lifedemons Feb 15, 2019

lifedemons commented Feb 15, 2019 •

edited

Loading

mpkorstanje commented Feb 17, 2019

lsuski commented Feb 17, 2019

lsuski commented Feb 19, 2019

lsuski commented Feb 20, 2019 •

edited

Loading

lifedemons commented Feb 20, 2019

lifedemons commented Feb 20, 2019

lifedemons commented Feb 20, 2019 •

edited

Loading

lifedemons commented Feb 20, 2019

lsuski commented Feb 21, 2019 •

edited

Loading

lsuski commented Feb 21, 2019

lifedemons commented Feb 21, 2019

lifedemons commented Feb 21, 2019

lifedemons commented Feb 22, 2019

lsuski commented Feb 22, 2019

lsuski commented Feb 22, 2019 •

edited

Loading

lsuski commented Feb 26, 2019

lifedemons commented Feb 26, 2019

aslakhellesoy commented Feb 27, 2019

VikOles commented Mar 10, 2021 •

edited

Loading

Introduced CucumberJUnitRunner - a bridge between Cucumber Features and Android Test Orchestrator #14

Introduced CucumberJUnitRunner - a bridge between Cucumber Features and Android Test Orchestrator #14

Conversation

lifedemons commented Feb 15, 2019

lifedemons Feb 15, 2019

Choose a reason for hiding this comment

lifedemons Feb 15, 2019

Choose a reason for hiding this comment

lifedemons Feb 15, 2019

Choose a reason for hiding this comment

lifedemons Feb 15, 2019

Choose a reason for hiding this comment

lifedemons Feb 15, 2019

Choose a reason for hiding this comment

lifedemons commented Feb 15, 2019 • edited Loading

mpkorstanje commented Feb 17, 2019

lsuski commented Feb 17, 2019

lsuski commented Feb 19, 2019

lsuski commented Feb 20, 2019 • edited Loading

lifedemons commented Feb 20, 2019

lifedemons commented Feb 20, 2019

lifedemons commented Feb 20, 2019 • edited Loading

lifedemons commented Feb 20, 2019

lsuski commented Feb 21, 2019 • edited Loading

lsuski commented Feb 21, 2019

lifedemons commented Feb 21, 2019

lifedemons commented Feb 21, 2019

lifedemons commented Feb 22, 2019

lsuski commented Feb 22, 2019

lsuski commented Feb 22, 2019 • edited Loading

lsuski commented Feb 26, 2019

lifedemons commented Feb 26, 2019

aslakhellesoy commented Feb 27, 2019

VikOles commented Mar 10, 2021 • edited Loading

lifedemons commented Feb 15, 2019 •

edited

Loading

lsuski commented Feb 20, 2019 •

edited

Loading

lifedemons commented Feb 20, 2019 •

edited

Loading

lsuski commented Feb 21, 2019 •

edited

Loading

lsuski commented Feb 22, 2019 •

edited

Loading

VikOles commented Mar 10, 2021 •

edited

Loading