Op/invocation fail to increase due to [IterationCleanup]

[ycrumeyrolle]

Similar to #994

This sample benchmark does not increase the op/invocation on warmup when a method is marked with the [IterationCleanup] attribute:

public class TestBenchmark
    {
        [Benchmark]
        [ArgumentsSource(nameof(GetValues))]
        public int Case1(string value)
        {
            return value.Length;
        }

        [Benchmark]
        [ArgumentsSource(nameof(GetValues))]
        public int Case2(string value)
        {
            return value.Length;
        }

        [IterationCleanup]
        public void Clear()
        {
        }

        public IEnumerable<string> GetValues()
        {
            yield return "Value 1";
            yield return "Value 2";
        }
    }

The workload stall at 1.

When removing the [IterationCleanup] attribute, the workload increase as usual.

[abelbraaksma]

Seems related to #1338. In our, I noticed that the cause seems to be dependent on the situation, but esp the JIT time, causing the algorithm to incorrectly assume it's a long running test.

[abelbraaksma]

@adamsitnik, this too may be caused by #760. Any thoughts?

[adamsitnik]

Hi @ycrumeyrolle

This is by design, as described in our docs

A method which is marked by the [IterationCleanup] attribute will be executed exactly once after each invocation. This attribute has the same set of constraint with [IterationSetup]: it's not recommended to use [IterationCleanup] in microbenchmarks.

You can read more about why we made such decision in #730

[ycrumeyrolle]

@adamsitnik thanks for your reply, I had seen the recommendation to avoid such attribute, but missed the "exactly once".