Statistics

Cucumber Performance supports a number of statistics out of the box. While some are included in every execution others are completely optional.

Strict results

Some statistics are also effected by disabling the strict option.

By default all statistics are limited to passing tests only.

You can disable strict by passing a "no-strict" in the command line or set "strict = false" when using annotations.

Statistic populations

All statistics come from the execution duration of each group, scenario and step.

• Group: Is the duration of a group.
  • Each group is its own cucumber execution. So this duration includes any overhead associated with running cucumber.
  • A group can be a single or multiple features depending on how you configure your simulation.
  • Groups should not be considered a perfect representation of a system.
• Scenario: Is the duration of a single scenario.
  • A scenario is a single cucumber scenario.
  • Each scenario can contain multiple steps.
• Step: Is the duration of a single step.
  • A step is a single cucumber step.
  • This is the most granular population.

Default Statistics

• Average (Median): The average total execution time for the population.
  • Population: All
• Minimum: The minimum or lower duration for a population.
  • Population: All
• Maximum: The maximum or highest duration for a population.
  • Population: All
• Count: The total size of a population.
  • Population: All
• Failed: The total number of failed tests in the population.
  • Population: All
  • Disabled Strict Required: True
• Passed: The total number of passed tests in the population.
  • Population: All
  • Disabled Strict Required: True
• Errored: The total number of tests with errors in the population.
  • Population: All
  • Disabled Strict Required: True
• Concurrent: The average concurrency of the population.
  • Population: Group

Additional Statistics

More statistics can be added to any execution by including their name in the options of the statistics formatter.

• Percentile: The percentile for the desired percent.
  • Population: All
  • Plugin Name: prcntl, prctl
  • Options: (double) percent
• Standard Deviation: The standard deviation for a population.
  • Population: All
  • Plugin Name: stddev, stdev

You can also create and add your own statistic creator plugins!

• You add these to your statistics as you would any plugin pg=statistics:class.path.to.your.plugin