testthat 2.3.2

• Fix R CMD check issues

testthat 2.3.1

• The last version of testthat introduced a performance regression in
  error assertions (#963). To fix it, you need to install rlang 0.4.2.

• Fixed error assertions with rJava errors (#964).

• Fixed issue where error and warning messages were not retrieved with
  conditionMessage() under certain circumstances.

testthat 2.3.0

Conditions

This release mostly focusses on an overhaul of how testthat works with conditions (i.e. errors, warnings and messages). There are relatively few user-facing changes, although you should now see more informative backtraces from errors and failures.

• Unexpected errors are now printed with a simplified backtrace.

• expect_error() and expect_condition() now display a backtrace
  when the error doesn't conform to expectations (#729).

• expect_error(), expect_warning() and expect_message() now call
  conditionMessage() to get the condition message. This generic
  makes it possible to generate messages at print-time rather than
  signal-time.

• expect_error() gets a better warning message when you test for a custom
  error class with regexp.

• New exp_signal() function is a condition signaller that
  implements the testthat protocol (signal with stop() if the
  expectation is broken, with a continue_test restart).

• Existence of restarts is first checked before invokation. This makes
  it possible to signal warnings or messages with a different
  condition signaller (#874).

• ListReporter now tracks expectations and errors, even when they occur
  outside of tests. This ensures that stop_on_failure matches the results
  displayed by the reporter (#936).

• You can silence warnings about untested error classes by
  implementing a method for is_uninformative_warning(). This method
  should be lazily registered, e.g. with vctrs::s3_register(). This
  is useful for introducing an experimental error class without
  encouraging users to depend on the class in their tests.

• Respect options(warn = -1) to ignore all warnings (@jeroen #958).

Expectations

• Expectations can now be explicitly subclassed with
  new_expectation(). This constructor follows our new conventions
  for S3 classes and takes an optional subclass and optional
  attributes.

• Unquoted inputs no longer potentially generate multiple test messages (#929).

• verify_output() no longer uses quasiquotation, which fixes issues
  when verifying the output of tidy eval functions (#945).

• verify_output() gains a unicode parameter to turn on or off the
  use of Unicode characters by the cli package. It is disabled by
  default to prevent the tests from failing on platforms like Windows
  that don't support UTF-8 (which could be your contributors' or your
  CI machines).

• verify_output() now correctly handles multi-line condition
  messages.

• verify_output() now adds spacing after condition messages,
  consistent with the spacing added after normal output.

• verify_output() has a new syntax for inserting headers in output
  files: insert a "# Header" string (starting with # as in
  Markdown) to add a header to a set of outputs.

Other minor improvements and bug fixes

• compare.numeric() uses a more sophisticated default tolerance that will
  automatically skip tests that rely on numeric tolerance if long doubles are
  not available (#940).

• JunitReporter now reports tests in ISO 8601 in the UTC timezone and
  uses the maximum precision of 3 decimal places (#923).

testthat 2.2.1

• Repair regression in test_rd() and add a couple of tests to hopefully
  detect the problem earlier in the future.

testthat 2.2.0

New features

• New verify_output() is designed for testing output aimed at humans
  (most commonly print methods and error messages). It is a regression
  test that saves output in a way that makes it easy to review. It is
  automatically skipped on CRAN (#782, #834).

Minor improvements and bug fixes

• as.data.frame.testthat_results() now always returns a data frame with 13
  columns (@jozefhajnala, #887).

• auto_test_package() now correctly handles helper files
  (tests/testthat/helper-*.R), automatically reloading all code and
  rerunning all tests (@CorradoLanera, #376, #896).

• expect_match() now displays info even when match length is 0 (#867).

• expect_s3_class() gains new expect argument that allows you to check
  for an exact class match, not just inheritance (#885).

• fail() and succeed() gain info argument, which is passed along to
  expect().

• test_examples() gets some minor fixes: it now returns the results
  invisibly, doesn't assume that examples should contain tests, and
  documents that you shouldn't be using it routinely (#841).

• test_file() only calls Reporter$end_context() if a context was started,
  fixing an error in TeamcityReporter (@atheriel, #883).

• skip() now reports reason for skipping as: Reason: {skip condition}
  (@patr1ckm, #868).

• skip_if() and skip_if_not() now report Reason: {skip condition} is TRUE
  and Reason: {skip condition} is not TRUE respectively (@ patr1ckm, #868).

• skip_if_translated() now tests for translation of a specific message.
  This is more robust than the previous approach because translation
  happens message-by-message, not necessarily for the entire session (#879)
  (and in general, it's impossible to determine what language R is currently
  using).

• skip_on_covr() allows you to skip tests when covr is running.
  (@ianmcook, #895)

• expect_known_value() gains a new serialisation version argument,
  defaulting to 2. Prevents the .rds files created to hold reference objects
  from making a package appear to require R >= 3.5 (#888 @jennybc).

testthat 2.1.1

• Fix test failures in strict latin1 locale

testthat 2.1.0

New expectations

• New expect_visible() and expect_invisible() make it easier to check if
  a function call returns its result visibly or invisibly (#719).

• New expect_mapequal(x, y) checks that x and y have the same names,
  and the same value associated with each name (i.e. they compare the values
  of the vector standardising the order of the names) (#863).

• New expect_vector() is a wrapper around vctrs::vec_assert() making it
  easy to test against the vctrs definitions of prototype and size (#846).
  (Currently requires development version of vctrs.)

Improvements to existing expectations

• All expectations give clearer error messages if you forget the object
  or expected arguments (#743).

• expect_equal() now correctly compares infinite values (#789).

• In expect_equal_to_reference(), the default value for update is
  now FALSE (@brodieG, #683).

• expect_error() now returns the error object as documentated (#724).
  It also now warns if you're using a classed expectation and you're
  not using the class argument. This is good practice as it decouples the
  error object (which tends to be stable) from its rendering to the user
  (which tends to be fragile) (#816).

• expect_identical() gains a ... argument to pass additional arguments
  down to identical() (#714).

• expect_lt(), expect_lte(), expect_gt() expect_gte() now handle Inf
  and NA arguments appropriately (#732), and no longer require the inputs
  to be numeric.

• expect_output() gains a width argument, allowing you to control the
  output width. This does not inherit from getOption("width"), ensuring
  that tests return the same results regardless of environment (#805).

• expect_setequal() now works with more vector types (including lists),
  because it uses %in%, rather than sort(). It also warns if the inputs
  are named, as this suggests that your mental model of how expect_setequal()
  works is wrong (#750).

• is_true() and is_false() have been deprecated because they conflict
  with other functions in the tidyverse.

Reporters

• Reporter documentation has been considerably improved (#657).

• CheckReporter, used by R CMD check, now includes a count of warnings.

• JUnitReporter no longer replaces . in class names (#753), and
  creates ouput that should be more compatible with Jenkins (#806, @comicfans).

• ListReporter now records number of passed tests and original results in
  new columns (#675).

• ProgressReporter, the default reporter, now:

  • Automatically generates a context from the file name. We no longer
    recommend the use of context() and instead encourage you to delete it,
    allowing the context to be autogenerated from the file name.

    This also eliminates the error that occured if tests can before the
    first context() (#700, #705).

  • Gains a update_interval parameter to control how often updates are
    printed (default 0.1 s). This prevents large printing overhead
    for very fast tests. (#701, @jimhester)

  • Uses a 3 character wide column to display test successes, so up to
    999 successful tests can be displayed without changing the alignment
    (#712).

• reporter$end_reporter() is now only called when testing completes
  successfully. This ensures that you don't get unnecessary output when the
  test fails partway through (#727).

Skips

• skip_if_offline() skips tests if an internet connection is not available
  (#685).

• skip_on_ci() skips tests on continuous integration systems
  (@mbjoseph, #825) by looking for a CI env var..

Other new features

• New testthat_examples() and testthat_example() make it easy to access
  new test files bundled with the package. These are used in various examples
  to make it easier to understand how to use the package.

• New local_mock() which allows you to mock a function without having to
  add an additional layer of indentation as with with_mock() (#856).

Other minor improvements and bug fixes

• auto_test_package() works better with recent devtools and also watches
  src/ for changes (#809).

• expect_s3_class() now works with unquoting (@jalsalam, #771).

• expectation objects now contain the failure message, even when successful
  (#836)

• devtools::test() no longer fails if run multiple times within the same R
  session for a package containing Catch tests.
  (devtools #1832)

• New testing_package() retrieves the name of the package currently being
  tested (#699).

• run_testthat_tests C entrypoint is registered more robustly.

• skip() now always produces a message of length 1, as expected elsewhere
  in testthat (#791).

• Warnings are passed through even when options(warn = 2) is set
  (@yutannihilation, #721).

testthat v2.0.1

• Fix failing tests with devtools 2.0.0

testthat 2.0.0

Breaking API changes

• "Can't mock functions in base packages": You can no longer use with_mock()
  to mock functions in base packages, because this no longer works in
  R-devel due to changes with the byte code compiler. I recommend using
  mockery or mockr instead.

• The order of arguments to expect_equivalent() and expect_error() has
  changed slightly as both now pass ... on another function. This reveals
  itself with a number of different errors, like:

  • 'what' must be a character vector
  • 'check.attributes' must be logical
  • 'tolerance' should be numeric
  • argument is not interpretable as logical
  • threw an error with unexpected class
  • argument "quo" is missing, with no default
  • argument is missing, with no default

  If you see one of these errors, check the number, order, and names of
  arguments to the expectation.

• "Failure: (unknown)". The last release mistakenly failed to test
  bare expectations not wrapped inside test_that(). If you see "(unknown)"
  in a failure message, this is a failing expectation that you previously
  weren't seeing. As well as fixing the failure, please also wrap inside
  a test_that() with an informative name.

• "Error: the argument has already been evaluated": the way in which
  expectations now need create labels has changed, which caused a couple
  of failures with unusual usage when combined with Reduce, lapply(),
  and Map(). Avoid these functions in favour of for loops. I also recommend
  reading the section below on quasiquotation support in order to create more
  informative failure messages.

Expectations

New and improved expectations

• expect_condition() works like expect_error() but captures any
  condition, not just error conditions (#621).

• expect_error() gains a class argument that allows you to make an
  assertion about the class of the error object (#530).

• expect_reference() checks if two names point to the same object (#622).

• expect_setequal() compares two sets (stored in vectors), ignoring
  duplicates and differences in order (#528).

New and improved skips

• skip_if() makes it easy to skip a test when a condition is true (#571).
  For example, use skip_if(getRversion() <= 3.1) to skip a test in older
  R versions.

• skip_if_translated() skips tests if you're running in an locale
  where translations are likely to occur (#565). Use this to avoid
  spurious failures when checking the text of error messages in non-English
  locales.

• skip_if_not_installed() gains new minimum_version argument (#487, #499).

Known good values

We have identified a useful family of expectations that compares the results of an expression to a known good value stored in a file. They are designed to be use in conjunction with git so that you can see what precisely has changed, and revert it if needed.

• expect_known_output() replaces expect_output_file(), which has
  been soft-deprecated. It now defaults to update = TRUE and warn, rather
  than failing on the first run. It gains a print argument to automatically
  print the input (#627). It also sets the width option to 80 to ensure
  consistent output across environments (#514)

• expect_known_value() replaces expect_equal_to_reference(), which
  has been soft-deprecated. It gains an update argument defaulting to TRUE.
  This changes behaviour from the previous version, and soft-deprecated
  expect_equal_to_reference() gets update = FALSE.

• expect_known_failure() stored and compares the failure message from
  an expectation. It's a useful regression test when developing informative
  failure messges for your own expectations.

Quasiquotation support

All expectations can now use unquoting (#626). This makes it much easier to generate informative failure messages when running tests in a for loop.

For example take this test:

f <- function(i) if (i > 3) i * 9 else i * 10

for (i in 1:5) {
  expect_equal(f(i), i * 10)
}

When it fails, you'll see the message Error: `f(i)` not equal to `i * 10` .
That's hard to diagnose because you don't know which iteration caused the problem!

for (i in 1:5) {
  expect_equal(f(!!i), !!(i * 10))
}

If you unquote the values using !!, you get the failure message `f(4L)` not equal to 40.. This is much easier to diagnose! See ?quasi_label() for more details.

(Note that this is not tidy evaluation per se, but is closely related. At this time you can not unquote quosures.)

New features

Setup and teardown

• New setup() and teardown() functions allow you to run at the start and
  end of each test file. This is useful if you want to pair cleanup code
  with the code that messes up state (#536).

• Two new prefixes are recognised in the test/ directory. Files starting
  with setup are run before tests (but unlike helpers are not run in
  devtools::load_all()). Files starting with teardown are run after all
  tests are completed (#589).

Other new features

• All files are now read and written as UTF-8 (#510, #605).

• is_testing() allows you to tell if your code is being run inside a
  testing environment (#631). Rather than taking a run-time dependency on testthat
  you may want to inline the function into your own package:

  is_testing <- function() {
    identical(Sys.getenv("TESTTHAT"), "true")
  }

  It's frequently useful to combine with interactive().

New default reporter

A new default reporter, ReporterProgress, produces more aesthetically pleasing output and makes the most important information available upfront (#529). You can return to the previous default by setting option(testthat.default_reporter = "summary").

Reporters

• Output colours have been tweaked to be consistent with clang:
  warnings are now in magenta, and skips in blue.

• New default_reporter() and check_reporter() which returns the default
  reporters for interactive and check environments (#504).

• New DebugReporter that calls a better version of recover() in case of
  failures, errors, or warnings (#360, #470).

• New JunitReporter generates reports in JUnit compatible format.
  (#481, @lbartnik; #640, @nealrichardson; #575)

• New LocationReporter which just prints the location of every expectation.
  This is useful for locating segfaults and C/C++ breakpoints (#551).

• SummaryReporter recieved a number of smaller tweaks

  • Aborts testing as soon the limit given by the option
    testthat.summary.max_reports (default 10) is reached (#520).

  • New option testthat.summary.omit_dots = TRUE hides the progress dots
    speeding up tests by a small amount (#502).

  • Bring back random praise and encouragement which I accidentally dropped
    (#478).

• New option testthat.default_check_reporter, defaults to "check".
  Continuous Integration system can set this option before evaluating
  package test sources in order to direct test result details to known
  location.

• All reporters now accept a file argument on initialization. If provided,
  reporters will write the test results to that path. This output destination
  can also be controlled with the option testthat.output_file
  (#635, @nealrichardson).

Deprecated functions

• is_null() and matches() have been deprecated because they conflict
  with other functions in the tidyverse (#523).

Minor improvements and bug fixes

• Updated Catch to 1.9.6. testthat now understands and makes use of the package
  routine registration mechanism required by CRAN with R >= 3.4.0.
  (@kevinushey)

• Better reporting for deeply nested failures, limiting the stack trace to the
  first and last 10 entries (#474).

• Bare expectations notify the reporter once again. This is achieved by running
  all tests inside test_code() by default (#427, #498). This behaviour can be
  overridden by setting wrap = FALSE in test_dir() and friends (#586).

• auto_test() and auto_test_package() provide hash parameter to enable
  switching to faster, time-stamp-based modification detection
  (#598, @katrinleinweber). auto_test_package() works correctly on windows
  (#465).

• capture_output_lines() is now exported (#504).

• compare.character() works correctly for vectors of length > 5 (#513, @brodieG)

• compare.default() gains a max_diffs argument and defaults to printing
  out only the first 9 differences (#538).

• compare.numeric() respects check.attributes() so expect_equivalent()
  correctly ignores attributes of numeric vectors (#485).

• Output expectations (expect_output(), expect_message(),
  expect_warning(), and expect_silent()) all invisibly return the first
  argument to be consistent with the other expectations (#615).

• expect_length() works with any object that has a length method, not
  just vectors (#564, @nealrichardson)

• expect_match() now accepts explicit perl and fixed arguments, and adapts
  the failure message to the value of fixed. This also affects other expectations
  that forward to expect_match(), like expect_output(), expect_message(),
  expect_warning(), and expect_error().

• expect_match() escapes special regular expression characters when printing
  (#522, @jimhester).

• expect_message(), expect_warning() and expect_error() produce clearer
  failure messages.

• find_test_scripts() only looks for \.[rR] in the extension
  (#492, @brodieG)

• test_dir(), test_package(), test_check() unset the R_TESTS env var
  (#603)

• test_examples() now works ...

testthat 1.0.2

• Ensure 'std::logic_error()' constructed with 'std::string()'
  argument, to avoid build errors on Solaris.