Non-overlapping scenarios don't reuse VUs

[RobinHoutevelts]

I tried to adapt #785 (comment) to my needs. But somehow I end up without VUs being reused between scenario's. Even though I'm using a liberal startTime to delay the second scenario.

I guess I'm doing something wrong.

Environment

• k6 version: v0.28.0 (2020-09-21T12:06:29+0000/v0.28.0-0-gdee9c4ce, go1.14.9, darwin/amd64)
• OS and version: MacOs Catalina 10.15.5 (19F101)
• Docker version and image, if applicable: n/a

Expected Behavior

When running a script with multiple non-overlapping scenario's, the VUs should be reused.

Actual Behavior

When I define 20 VUs, I end up with 40.

Steps to Reproduce the Problem

Command

./k6 run non_overlapping.js --summary-export=load-performance.json --logformat=raw

Script

// non_overlapping.js
import {fail} from "k6";
import { Counter } from 'k6/metrics';

let VUsCount = 20;
const vuInitTimeoutSecs = 5;
const loadTestDurationSecs = 5;
const loadTestGracefulStopSecs = 2;

let vuSetupsDone = new Counter('vu_setups_done');
let vuLoadTestDone = new Counter('vu_loadtests_done');

export let options = {
    noCookiesReset: true,
    scenarios: {
        // This is the per-VU setup/init equivalent:
        vu_setup: {
            executor: 'per-vu-iterations',
            vus: VUsCount,
            iterations: 1,
            maxDuration: `${vuInitTimeoutSecs}s`,
            gracefulStop: '0s',

            exec: 'vuSetup',
        },

        addStuffInBasketLoadTest: {
            executor: 'per-vu-iterations',
            startTime: `${vuInitTimeoutSecs + 2}s`, // start only after the init is done ( even added a buffer )
            vus: VUsCount,
            iterations: 1,
            maxDuration: `${loadTestDurationSecs}s`,
            // Add extra tags to emitted metrics from this scenario. This way
            // our thresholds below can only be for them. We can also filter by
            // the `scenario:my_api_test` tag for that, but setting a custom tag
            // here allows us to set common thresholds for multi-scenario tests.
            tags: {type: 'loadtest'},
            exec: 'apiTest',
        },

        // This is the per-VU teardown/cleanup equivalent:
        vu_teardown: {
            executor: 'per-vu-iterations',
            startTime: `${vuInitTimeoutSecs + loadTestDurationSecs + loadTestGracefulStopSecs}s`,
            vus: VUsCount,
            iterations: 1,
            maxDuration: `${vuInitTimeoutSecs}s`,
            exec: 'vuTeardown',
        },
    },
    thresholds: {
        // Make sure all of the VUs finished their setup successfully, so we can
        // ensure that the load test won't continue with broken VU "setup" data
        'vu_setups_done': [{
            threshold: `count==${VUsCount}`,
            abortOnFail: true,
            delayAbortEval: `${vuInitTimeoutSecs}s`,
        }],
        // Also make sure all of the VU teardown calls finished uninterrupted:
        'iterations{scenario:vu_teardown}': [`count==${VUsCount}`],
        'vu_loadtests_done': [`count==${VUsCount}`],

        // Ignore HTTP requests from the VU setup or teardown here
        'http_req_duration{type:loadtest}': ['p(90)<500'],
    },
    summaryTrendStats: ['min', 'med', 'avg', 'p(90)', 'p(95)', 'p(99)', 'p(99.9)', 'max'],
};

export function vuSetup() {
    vuSetupsDone.add(0); // workaround for https://github.com/loadimpact/k6/issues/1346
    vuLoadTestDone.add(0);

    let user = `info+testuser-${__VU}@foobar.com`;

    // Throttle logins so they don't all happen at once
    // ( This assumes a login can be done in less than a second )
    // sleep(__VU - 1);

    console.log(`VU ${__VU} is being logged in with username ${user}`);

    // Login bit

    console.log(`VU ${__VU} was logged in with username ${user}`);
    vuSetupsDone.add(1);
}

export function vuTeardown() {
    console.log(`VU ${__VU} was logged out`);
}

export function apiTest() {
    console.log(`${__VU} is running.`);
    if (__VU > VUsCount) {
        fail(`Unexpected ${__VU} because higher than VUsCount ${VUsCount}`);
    }
    vuLoadTestDone.add(1);
}

Output

VU 22 is being logged in with username [email protected]
VU 11 is being logged in with username [email protected]
VU 11 was logged in with username [email protected]
VU 16 is being logged in with username [email protected]
VU 16 was logged in with username [email protected]
VU 7 is being logged in with username [email protected]
VU 7 was logged in with username [email protected]
VU 22 was logged in with username [email protected]
VU 1 is being logged in with username [email protected]
VU 21 is being logged in with username [email protected]
VU 21 was logged in with username [email protected]
VU 14 is being logged in with username [email protected]
VU 3 is being logged in with username [email protected]
VU 3 was logged in with username [email protected]
VU 2 is being logged in with username [email protected]
VU 6 is being logged in with username [email protected]
VU 1 was logged in with username [email protected]
VU 9 is being logged in with username [email protected]
VU 9 was logged in with username [email protected]
VU 5 is being logged in with username [email protected]
VU 5 was logged in with username [email protected]
VU 17 is being logged in with username [email protected]
VU 17 was logged in with username [email protected]
VU 10 is being logged in with username [email protected]
VU 10 was logged in with username [email protected]
VU 15 is being logged in with username [email protected]
VU 2 was logged in with username [email protected]
VU 15 was logged in with username [email protected]
VU 13 is being logged in with username [email protected]
VU 13 was logged in with username [email protected]
VU 8 is being logged in with username [email protected]
VU 4 is being logged in with username [email protected]
VU 4 was logged in with username [email protected]
VU 19 is being logged in with username [email protected]
VU 19 was logged in with username [email protected]
VU 12 is being logged in with username [email protected]
VU 8 was logged in with username [email protected]
VU 6 was logged in with username [email protected]
VU 14 was logged in with username [email protected]
VU 12 was logged in with username [email protected]
39 is running.
40 is running.
35 is running.
    GoError: Unexpected 39 because higher than VUsCount 20
GoError: Unexpected 40 because higher than VUsCount 20
29 is running.
27 is running.
30 is running.
34 is running.
GoError: Unexpected 29 because higher than VUsCount 20
GoError: Unexpected 34 because higher than VUsCount 20
18 is running.
23 is running.
38 is running.
37 is running.
GoError: Unexpected 27 because higher than VUsCount 20
GoError: Unexpected 37 because higher than VUsCount 20
33 is running.
26 is running.
28 is running.
GoError: Unexpected 38 because higher than VUsCount 20
31 is running.
25 is running.
GoError: Unexpected 35 because higher than VUsCount 20
32 is running.
36 is running.
GoError: Unexpected 23 because higher than VUsCount 20
GoError: Unexpected 30 because higher than VUsCount 20
20 is running.
24 is running.
GoError: Unexpected 33 because higher than VUsCount 20
GoError: Unexpected 24 because higher than VUsCount 20
GoError: Unexpected 26 because higher than VUsCount 20
GoError: Unexpected 25 because higher than VUsCount 20
GoError: Unexpected 28 because higher than VUsCount 20
GoError: Unexpected 31 because higher than VUsCount 20
GoError: Unexpected 32 because higher than VUsCount 20
GoError: Unexpected 36 because higher than VUsCount 20
VU 19 was logged out
VU 6 was logged out
VU 5 was logged out
VU 3 was logged out
VU 13 was logged out
VU 4 was logged out
VU 12 was logged out
VU 9 was logged out
VU 15 was logged out
VU 7 was logged out
VU 14 was logged out
VU 22 was logged out
VU 16 was logged out
VU 10 was logged out
VU 1 was logged out
VU 21 was logged out
VU 8 was logged out
VU 2 was logged out
VU 17 was logged out
VU 11 was logged out

[RobinHoutevelts]

I was missing a gracefulStop option in my second scenario 🙈

After adding that I ended up with the expected behaviour 👍

I'm leaving it here, as others might hit the same issue.