Using GENERATE macro in a for loop

[Sidelobe]

When using a simple GENERATE expression inside a 'for' loop, the number of iterations increases dramatically

#include "catch/catch.hpp"

TEST_CASE("TESTTEST")
{
    for (int i=0; i<5; ++i) {
        int d = GENERATE(1, 2);
        REQUIRE(true);
    }
}

This results in 160 assertions (according to the catch console output). I would expect 10.
If I put the GENERATE() outside the for loop, I get 10 assertions.

Is this expected behaviour or a bug?

Using Catch v2.11.3 with clang-1103.0.32.29 on macos.

[horenmar]

That's definitely unintended behaviour. At a glance, what happens is that the for loop creates 5 distinct instances of the generators. Then, after the loop ends, the test is entered again because there are still unexhausted generators, but only the last one of the 5 is advanced. Then, the next time, the second to last one is advanced, and the last one is restarted...

What this boils down to is a 5-bit binary counter, where each number leads to 5 assertions: 5 * 2**5 == 160.

[Sidelobe]

ah great, thanks for you reply, Martin - I was indeed wondering how the 160 came to be :-)
I guess, this could have a significant performance impact on many generator-based tests out there.

[horenmar]

I have a prototype fix for this issue. The problem is that it is kinda brittle, and also that it silently breaks code like this:

TEST_CASE() {
    int i = GENERATE(1, 2); int j = GENERATE(3, 4);
}

because they are both on the same line. I can probably fix that too, but it will take a bit longer.

[pisoir]

I have this code:

#include <catch2/catch_test_macros.hpp>
#include <catch2/generators/catch_generators.hpp>
#include <catch2/generators/catch_generators_adapters.hpp>
#include <catch2/generators/catch_generators_random.hpp>

#include <iostream>

TEST_CASE("TESTTEST")
{
    for (int n=0; n<2; ++n) {
        int i = GENERATE(1, 2); 
        int j = GENERATE(3, 4);

        std::cout << i << ":" << j << std::endl;

        REQUIRE(true);
    }
}

I would like to understand why there is 32 assertions and actually quite a lot of e.g. pairs (1,3). I would expect 2*2*2 = 8 assertions. Basically twice the set of {(1,3), (1, 4), (2, 3), (2, 4) } (same when the loop is only till n < 1).