Skip to content

Latest commit

 

History

History
233 lines (188 loc) · 8.33 KB

fixtures.md

File metadata and controls

233 lines (188 loc) · 8.33 KB

Test Fixtures

If your fuzz test requires setup/teardown logic that is too expensive to be embedded in your property function and executed on each fuzz test iteration, you can use a test fixture. In FuzzTest, any default-constructible class can serve as a test fixture. To instantiate it, use the macro FUZZ_TEST_F, which corresponds to FUZZ_TEST in the same way that TEST_F corresponds to TEST in GoogleTest. Furthermore, you can easily adapt and reuse the existing GoogleTest fixtures in your fuzz tests.

[TOC]

When to use fixtures in fuzz tests

Most best practices for writing unit tests (https://abseil.io/tips/122) apply equally well when writing fuzz tests. In view of that, you should generally avoid fixtures and prefer standalone helper functions for your fuzz test's initialization. However, in some situations test fixtures are appropriate.

  1. If your fuzz test requires expensive setup, e.g., initializing a server, doing it in the property function may be prohibitively expensive. The fuzzing engine calls the property function once per fuzz test iteration, and running the setup each time may slow down fuzzing to a point where it becomes ineffective.

  2. If you want to add a fuzz test to an existing code base that uses test fixtures for unit tests, refactoring the code to eliminate fixtures may be impractical or even impossible.

Using a fixture for expensive setup

Suppose we wanted to test that a certain HTTP server called EchoServer echoes back the string it receives. We could do this with a fuzz test that doesn't use fixtures.

// A standalone property function that tests the property of interest.
void ReturnsTheSameString(const std::string& request) {
  // Initialize the server.
  EchoServer server;
  server.Start("localhost:9999");

  // Test the property.
  std::string response;
  SendRequest("localhost:9999", request, &response);
  EXPECT_EQ(response, request);

  // Tear down the server.
  server.Stop();
}

// Instantiate the fuzz test using the FUZZ_TEST macro with a chosen suite name
// and the property function as the test name.
FUZZ_TEST(EchoServerFuzzTest, ReturnsTheSameString)
    .WithDomains(fuzztest::Arbitrary<std::string>())
    .WithSeeds({"Hello"});

The issue with this fuzz test is that the server initialization and teardown happen in each fuzz test iteration, that is, in each call to the property function. If starting and stopping the server takes non-trivial time, the fuzz test will be slow and ineffective.

To improve the test, we would like to do the initialization and teardown once and reuse the same server across all fuzz test iterations. We can do this using a test fixture:

// In FuzzTest, any default-constructible class can be a test fixture.
class EchoServerFuzzTest {
 public:
  // The constructor initializes the fixture.
  EchoServerFuzzTest() { server_.Start("localhost:9999"); }

  // The destructor tears down the fixture.
  ~EchoServerFuzzTest() { server_.Stop(); }

  // The fuzz test's property function must be a public member of the fixture.
  void ReturnsTheSameString(const std::string& request) {
    std::string response;
    SendRequest("localhost:9999", request, &response);
    EXPECT_EQ(response, request);
  }

 private:
  EchoServer server_;
};

// Instantiate the fuzz test using the FUZZ_TEST_F macro, with the fixture as
// the suite name and the property function as the test name.
FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString)
    // The macro supports the FUZZ_TEST clauses for specifying domains and seeds.
    .WithDomains(fuzztest::Arbitrary<std::string>())
    .WithSeeds({"Hello"});

FuzzTest will instantiate the fixture once and call the property function on the same object in each fuzz test iteration.

Note on mutable test fixtures

While persisting state across fuzz test iterations can be effective, it can also have unwanted consequences. Consider the following example.

class MyFuzzTest {
 public:
  void MyProperty(int n) {
    MyApi(n, flag_);
    flag_ = !flag_;
  }

 private:
  bool flag_ = false;
};
FUZZ_TEST_F(MyFuzzTest, MyProperty);

By mutating flag_ and using it in the call to MyApi(), the test no longer depends only on the input generated by the fuzzer. This can potentially break the fuzzer's assumptions about how the inputs affect coverage and make fuzzing ineffective. Additionally, a crashing input found by the fuzzer may not be sufficient for reproducing the crash, since the crash also depends on a particular value of flag_.

BEST PRACTICE: Make sure that the property function resets the mutated fixture so that each fuzz test iteration starts in the same state.

Using existing GoogleTest fixtures in fuzz tests

Note: Only use GoogleTest fixtures if they are shared with existing unit tests, or if you need static per-fixture setup and teardown (via SetUpTestSuite and TearDownTestSuite). Otherwise, prefer fixtureless code with standalone initialization functions.

If you already use a GoogleTest fixture derived from ::testing::Test in your unit tests and you want to reuse it in fuzz tests, you first need to choose when you want the fixture to be instantiated and destroyed. GoogleTest fixtures are typically built around a guarantee that the same fixture object will never be reused for multiple tests or multiple runs of the same test. In a fuzz test, this guarantee requires instantiating the fixture once per fuzz test iteration, i.e., calling the property function each time on a fresh fixture object. However, for increased performance, it is possible to instantiate a fixture once per fuzz test (if the fixture supports such persistence) and reuse the same fixture object in all calls to the property function. As noted earlier, this comes with the risk of state mutation, which can lead to non-reproducible crashes, so design your fixture carefully.

Semantics 1: Fixture instantiated once per fuzz test iteration

Adapt a fixture called Fixture by extending ::fuzztest::PerIterationFixtureAdapter<Fixture>. This case should be the more common one, since it observes the GoogleTest invariant of not using a fixture object more than once.

As an example, suppose we have a function SumVec() that returns the sum of a vector of integers and we have the following fixture.

class SumVecTest : public testing::Test {
 public:
  SumVecTest() : vec_{1, 2, 3} {}

 protected:
  std::vector<int> vec_;
};

A fuzz test that uses the fixture would look like the following:

// Adapts the fixture using the "per-iteration" semantics.
class SumVecFuzzTest : public fuzztest::PerIterationFixtureAdapter<SumVecTest> {
 public:
  void SumsLastEntry(int last_entry) {
    int previous_sum = SumVec(vec_);
    vec_.push_back(last_entry);
    EXPECT_EQ(SumVec(vec_), previous_sum + last_entry);
  }
};
FUZZ_TEST_F(SumVecFuzzTest, SumsLastEntry);

The "per-iteration" semantics is appropriate for this fixture because the test mutates vec_.

Semantics 2: Fixture instantiated once per fuzz test

Adapt a fixture called Fixture by extending ::fuzztest::PerFuzzTestFixtureAdapter<Fixture>. This case should be the less common one: it relies on the fixture being easily resettable, which is not a typical concern when designing GoogleTest fixtures.

As an example, let's return to EchoServer. Suppose we already have a GoogleTest fixture that sets up the server.

class EchoServerTest : public testing::Test {
 public:
  EchoServerTest() { server_.Start("localhost:9999"); }
  ~EchoServerTest() override { server_.Stop(); }

 private:
  EchoServer server_;
};

We would write a test using this fixture as in the following code:

// Adapts the fixture using the "per-fuzz-test" semantics.
class EchoServerFuzzTest
    : public fuzztest::PerFuzzTestFixtureAdapter<EchoServerTest> {
 public:
  void ReturnsTheSameString(const std::string& request) {
    std::string response;
    SendRequest("localhost:9999", request, &response);
    EXPECT_EQ(response, request);
  }
};
FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString);

Here, the "per-fuzz-test" semantics is appropriate because the test doesn't mutate the server in any way that would be relevant for the test, and the server initialization is too expensive to be performed in each iteration.