Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Print property seeds and add docs about reproducing test failures #90

Merged
merged 3 commits into from
Apr 1, 2020
Merged

Print property seeds and add docs about reproducing test failures #90

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1abd10d
Print property seeds and add docs about reproducing test failures
gabro Mar 29, 2020
1fb68ca
Add configurable scalaCheckInitialSeed
gabro Apr 1, 2020
bf76941
Implement review suggestion
gabro Apr 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/integrations/scalacheck.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,38 @@ class IntegerSuite extends ScalaCheckSuite {
}
```

## Reproducing a property failure

In some cases, you may find that ScalaCheck properties fail
non-deterministically. This can happen due to the randomness of the input values
for each test run.

When this happens, you can deterministally reproduce a property failure by using
its seed. Whenever a failure occurs, MUnit prints the offending seed along with
a suggestion on how to reproduce it:

```
Failing seed: CTH6hXj8ViScMmsO78-k4_RytXHPK_wSJYNH2h4dCpB=
You can reproduce this failure by adding this to your suite:

override val scalaCheckInitialSeed = "CTH6hXj8ViScMmsO78-k4_RytXHPK_wSJYNH2h4dCpB="

```

To reproduce the failure you can follow the suggestion to fix the seed:

```diff
class MySuite extends ScalaCheckSuite {

+ override val scalaCheckInitialSeed = "CTH6hXj8ViScMmsO78-k4_RytXHPK_wSJYNH2h4dCpB="

// ...
}
```

Re-running the test will now fail deterministically, which allows you to work on
an appropriate fix without worrying about randomness.

## Migrating from ScalaTest

ScalaTest provides two styles for writing property-based tests, which are both
Expand Down
30 changes: 24 additions & 6 deletions munit-scalacheck/shared/src/main/scala/munit/ScalaCheckSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package munit
import org.scalacheck.Prop
import org.scalacheck.{Test => ScalaCheckTest}
import org.scalacheck.util.Pretty
import org.scalacheck.rng.Seed
import scala.util.Success
import scala.util.Failure
import scala.util.Try
Expand Down Expand Up @@ -34,21 +35,38 @@ trait ScalaCheckSuite extends FunSuite {

protected def scalaCheckPrettyParameters = Pretty.defaultParams

protected def scalaCheckInitialSeed: String = Seed.random().toBase64

private val scalaCheckPropTransform: TestTransform =
new TestTransform("ScalaCheck Prop", t => {
t.withBodyMap[TestValue](
_.transformCompat {
case Success(prop: Prop) => propToTry(prop, t.location)
case Success(prop: Prop) => propToTry(prop, t)
case r => r
}(munitExecutionContext)
)
})

private def propToTry(prop: Prop, testLocation: Location): Try[Unit] = {
private def propToTry(prop: Prop, test: Test): Try[Unit] = {
import ScalaCheckTest._
val result = check(scalaCheckTestParameters, prop)
def renderResult(r: Result) =
Pretty.pretty(r, scalaCheckPrettyParameters)
val seed =
scalaCheckTestParameters.initialSeed.getOrElse(
Seed.fromBase64(scalaCheckInitialSeed).get
)
val result = check(scalaCheckTestParameters, prop.useSeed(test.name, seed))
def renderResult(r: Result) = {
val resultMessage = Pretty.pretty(r, scalaCheckPrettyParameters)
if (r.passed) {
resultMessage
} else {
val seedMessage = s"""|Failing seed: ${seed.toBase64}
|You can reproduce this failure by adding this to your suite:
|
| override val scalaCheckInitialSeed = "${seed.toBase64}"
|""".stripMargin
seedMessage + "\n" + resultMessage
}
}

result.status match {
case Passed | Proved(_) =>
Expand All @@ -60,7 +78,7 @@ trait ScalaCheckSuite extends FunSuite {
Failure(e.withMessage(e.getMessage() + "\n\n" + renderResult(r)))
case _ =>
// Fail using the test location
Try(fail("\n" + renderResult(result))(testLocation))
Try(fail("\n" + renderResult(result))(test.location))
}
}

Expand Down
33 changes: 21 additions & 12 deletions tests/shared/src/main/scala/munit/ScalaCheckFrameworkSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
package munit

import org.scalacheck.Prop.forAll
import org.scalacheck.rng.Seed

class ScalaCheckFrameworkSuite extends ScalaCheckSuite {

// NOTE(gabro): this is needed for making the test output stable for the failed test below.
// It also serves as a test for overriding these parameters.
override def scalaCheckTestParameters =
super.scalaCheckTestParameters.withInitialSeed(Seed(123L))
// It also serves as a test for overriding this parameter
override val scalaCheckInitialSeed =
"CTH6hXj8ViScMmsO78-k4_RytXHPK_wSJYNH2h4dCpB="

property("boolean check (true)") {
forAll { (l1: List[Int], l2: List[Int]) =>
Expand Down Expand Up @@ -45,27 +44,37 @@ object ScalaCheckFrameworkSuite
extends FrameworkTest(
classOf[ScalaCheckFrameworkSuite],
"""|==> success munit.ScalaCheckFrameworkSuite.boolean check (true)
|==> failure munit.ScalaCheckFrameworkSuite.boolean check (false) - /scala/munit/ScalaCheckFrameworkSuite.scala:19
|18:
|19: property("boolean check (false)") {
|20: forAll { (n: Int) => scala.math.sqrt(n * n) == n }
|==> failure munit.ScalaCheckFrameworkSuite.boolean check (false) - /scala/munit/ScalaCheckFrameworkSuite.scala:18
|17:
|18: property("boolean check (false)") {
|19: forAll { (n: Int) => scala.math.sqrt(n * n) == n }
|
|Failing seed: CTH6hXj8ViScMmsO78-k4_RytXHPK_wSJYNH2h4dCpB=
|You can reproduce this failure by adding this to your suite:
|
| override val scalaCheckInitialSeed = "CTH6hXj8ViScMmsO78-k4_RytXHPK_wSJYNH2h4dCpB="
|
|Falsified after 0 passed tests.
|> ARG_0: -1
|> ARG_0_ORIGINAL: 2147483647
|==> success munit.ScalaCheckFrameworkSuite.tagged
|==> success munit.ScalaCheckFrameworkSuite.assertions (true)
|==> failure munit.ScalaCheckFrameworkSuite.assertions (false) - /scala/munit/ScalaCheckFrameworkSuite.scala:37
|36: assertEquals(n * 1, n)
|37: assertEquals(n * n, n)
|38: assertEquals(n + 0, n)
|==> failure munit.ScalaCheckFrameworkSuite.assertions (false) - /scala/munit/ScalaCheckFrameworkSuite.scala:36
|35: assertEquals(n * 1, n)
|36: assertEquals(n * n, n)
|37: assertEquals(n + 0, n)
|values are not the same
|=> Obtained
|1
|=> Diff (- obtained, + expected)
|-1
|+-1
|
|Failing seed: CTH6hXj8ViScMmsO78-k4_RytXHPK_wSJYNH2h4dCpB=
|You can reproduce this failure by adding this to your suite:
|
| override val scalaCheckInitialSeed = "CTH6hXj8ViScMmsO78-k4_RytXHPK_wSJYNH2h4dCpB="
|
|Falsified after 0 passed tests.
|> ARG_0: -1
|> ARG_0_ORIGINAL: 2147483647
Expand Down