Fuzzing a Compiler

This tutorial walks through the process of fuzzing a non-trivial application: the Google Closure Compiler. The closure compiler optimizes JavaScript source code; therefore, we will write a generator for strings that produces syntactically valid inputs.

Unlike the Fuzzing with Zest tutorial, we will use the JQF Maven Plugin to invoke the fuzzing engine via Apache Maven. We will also use Maven to resolve build and test dependencies.

Requirements and Background

This tutorial assumes you know how to write a simple Generator<T> for any T, how to write a JQF test driver with @Fuzz annotations, and how to interpret the status screen when running the Zest algorithm. If not, check out the tutorial on Fuzzing with Zest.

You will need Java 8+ and Apache Maven 3.5+.

Step 1: Create pom.xml

Save the following file as pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>examples</groupId>
    <artifactId>zest-tutorial</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <!-- Google Closure: we want to test this library -->
        <dependency>
            <groupId>com.google.javascript</groupId>
            <artifactId>closure-compiler</artifactId>
            <version>v20180204</version>
            <scope>test</scope>
        </dependency>

        <!-- 
            JQF: test dependency for @Fuzz annotation
        -->
        <dependency>
            <groupId>edu.berkeley.cs.jqf</groupId>
            <artifactId>jqf-fuzz</artifactId>
            <!-- confirm the latest version at: https://mvnrepository.com/artifact/edu.berkeley.cs.jqf -->
            <version>1.0-beta-1</version> 
            <scope>test</scope>
        </dependency>

        <!-- 
            JUnit-QuickCheck: API to write generators
        -->
        <dependency>
            <groupId>com.pholser</groupId>
            <artifactId>junit-quickcheck-generators</artifactId>
            <version>0.7</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <!-- The JQF plugin, for invoking the command `mvn jqf:fuzz` -->
            <plugin>
                <groupId>edu.berkeley.cs.jqf</groupId>
                <artifactId>jqf-maven-plugin</artifactId>
                <!-- confirm the latest version at: https://mvnrepository.com/artifact/edu.berkeley.cs.jqf -->
                <version>1.0-beta-1</version>
            </plugin>
        </plugins>
    </build>
</project>

Step 2: Write a test driver

Create the following file as src/test/java/examples/CompilerTest.java:

package examples;

import java.io.ByteArrayOutputStream;
import java.io.PrintStream;

import com.google.javascript.jscomp.CompilationLevel;
import com.google.javascript.jscomp.Compiler;
import com.google.javascript.jscomp.CompilerOptions;
import com.google.javascript.jscomp.Result;
import com.google.javascript.jscomp.SourceFile;
import edu.berkeley.cs.jqf.fuzz.Fuzz;
import edu.berkeley.cs.jqf.fuzz.JQF;
import org.junit.Before;
import org.junit.runner.RunWith;

import static org.junit.Assume.*;

@RunWith(JQF.class)
public class CompilerTest {

    static {
        // Disable all logging by Closure passes, to speed fuzzing
        java.util.logging.LogManager.getLogManager().reset();
    }

    private Compiler compiler = new Compiler(new PrintStream(new ByteArrayOutputStream(), false));
    private CompilerOptions options = new CompilerOptions();
    private SourceFile externs = SourceFile.fromCode("externs", "");

    @Before
    public void initCompiler() {
        // Don't use threads
        compiler.disableThreads();
        // Don't print things
        options.setPrintConfig(false);
        // Enable all safe optimizations
        CompilationLevel.SIMPLE_OPTIMIZATIONS.setOptionsForCompilationLevel(options);
    }

    private Result compile(SourceFile input) {
        Result result = compiler.compile(externs, input, options);
        assumeTrue(result.success); // Semantic validity check
        return result;
    }

    @Fuzz
    public void testWithString(String code) {
        SourceFile input = SourceFile.fromCode("input", code);
        compile(input);
    }
}

The method annotated with @Fuzz will be the entry point to the fuzzing engine. The rest of the methods are primarily set-up code to initialize the Google closure compiler and set some of its configuration options.

Step 3: Fuzz via Maven

We now fuzz using the JQF Maven Plugin, as follows:

mvn jqf:fuzz -Dclass=examples.CompilerTest -Dmethod=testWithString

Step 4: Get disappointed with coverage

Zest: Validity Fuzzing with Parametric Generators
----------------------------------------------
Test name:            examples.CompilerTest#testWithString
Results directory:    /path/to/tutorial/target/fuzz-results/examples.CompilerTest/testWithString
Elapsed time:         5m 0s (no time limit)
Number of executions: 34,312
Valid inputs:         12,414 (36.18%)
Cycles completed:     0
Unique failures:      0
Queue size:           221 (0 favored last cycle)
Current parent input: 120 (favored) {735/740 mutations}
Execution speed:      85/sec now | 114/sec overall
Total coverage:       4,615 (7.04% of map)
Valid coverage:       4,247 (6.48% of map)

After fuzzing for a while, you'll notice some modest code coverage (we saw ~4,600 branches in 5 minutes), and no failures found. This is because by default, the string inputs to the testWithString method are generated at random, without any knowledge of the syntax of JavaScript. Most of these inputs lead to execution paths corresponding to syntax errors. Although Zest tries its best to bias input generation towards semantic validity, most of these inputs will be trivially valid (e.g. 1+1 or x).

Step 5: Write a custom JavaScript generator

TODO: This article is under construction

Step 6: Fuzz again via Maven