Spec-interpreter fuzzing: modify to generate and test single-instruction or no-control-flow cases #3251
Comments
This commit removes the `differential_spec` fuzz target for now, although this removal is intended to be temporary. We have bytecodealliance#3251 to track re-enabling the spec interpreter in a way that it won't time out, and additionally the spec interpreter is also failing to build with ocaml on oss-fuzz so that will also need to be investigated when re-enabling.
This commit removes the `differential_spec` fuzz target for now, although this removal is intended to be temporary. We have #3251 to track re-enabling the spec interpreter in a way that it won't time out, and additionally the spec interpreter is also failing to build with ocaml on oss-fuzz so that will also need to be investigated when re-enabling.
alexcrichton
added
fuzzing
Subscribe to Label Actioncc @fitzgen
This issue or pull request has been labeled: "fuzzing"
Thus the following users have been cc'd because of the following labels:
To subscribe or unsubscribe from this label, edit the |
|
My intuition is that combining multiple instructions, even if it is just two or three, would give us much more bang for our buck than testing single instructions. |
|
Ah, old issue, paging back in context (I'm not sure why the labeler decided to tag you just now on a discussion from August!)... Yup, agreed, that's the "or a straight-line sequence with no control flow or calls" above :-) Adding some more thought just now: IMHO we should do it only up to the max depth of our instruction combining patterns. The point of this suggestion was specifically to target the fuzz throughput in a different way, at the input-value space rather than the input-program space. As we test straightline sequences of In other words, if my goal is to say that all cases for (say) SIMD instruction X have been covered, then separately fuzzing So, said another way, what I think would be useful is a unit-test-inspired approach where we take each individual lowering and then throw a long test-vector of argument values at it. (In other words the cc @abrown , any interest in looking at this as a way to get the spec interpreter in active fuzzing use? |
|
I would eventually like to look into this more but if anyone else has the bandwidth feel free to jump in first (read: I'm not sure when I will get to this). |
In looking into bytecodealliance/wasmtime#3251, I created a mechanism for restricting what kinds of instructions wasm-smith can generate. The [WebAssembly specification](https://webassembly.github.io/spec/core/syntax/instructions.html) organizes its instructions into several categories (e.g., numeric, vector, reference, control, etc.) and this change allows the user to configure the module generation based on these categories: ``` ``` There is some related configuration in wasm-smith to restrict what instructions are available. Currently, the wasm-smith configuration is organized around "proposals," which can be enabled or disabled. In theory, a user could be confused if the proposal was disabled and they explicitly enabled an instruction kind (e.g. reference)--"why aren't reference instructions being generated?" But this accident seems unlikely: `--allowed-instructions` defaults to enabling all kinds, so the user would have to explicitly filter out some kind, deliberately shooting themselves in the foot. Despite some risk of confusion (mitigated by the documentation in this PR), this filtering of instructions kinds ends up being useful in a general way: not only is it a start at fixing the issue above, it is useful for work I am doing to generate fuzz only parts of the spec.
In looking into bytecodealliance/wasmtime#3251, I created a mechanism for restricting what kinds of instructions wasm-smith can generate. The [WebAssembly specification](https://webassembly.github.io/spec/core/syntax/instructions.html) organizes its instructions into several categories (e.g., numeric, vector, reference, control, etc.) and this change allows the user to configure the module generation based on these categories: ``` head -c 10000 /dev/urandom | cargo run --bin wasm-smith -- --allowed-instructions memory,parametric -o test.wasm && wasm2wat test.wasm ``` There is some related configuration in wasm-smith to restrict what instructions are available. Currently, the wasm-smith configuration is organized around "proposals," which can be enabled or disabled. In theory, a user could be confused if the proposal was disabled and they explicitly enabled an instruction kind (e.g. reference)--"why aren't reference instructions being generated?" But this accident seems unlikely: `--allowed-instructions` defaults to enabling all kinds, so the user would have to explicitly filter out some kind, deliberately shooting themselves in the foot. Despite some risk of confusion (mitigated by the documentation in this PR), this filtering of instructions kinds ends up being useful in a general way: not only is it a start at fixing the issue above, it is useful for work I am doing to generate fuzz only parts of the spec.
In looking into bytecodealliance/wasmtime#3251, I created a mechanism for restricting what kinds of instructions wasm-smith can generate. The [WebAssembly specification](https://webassembly.github.io/spec/core/syntax/instructions.html) organizes its instructions into several categories (e.g., numeric, vector, reference, control, etc.) and this change allows the user to configure the module generation based on these categories: ``` head -c 10000 /dev/urandom | cargo run --bin wasm-smith -- --allowed-instructions memory,parametric -o test.wasm && wasm2wat test.wasm ``` There is some related configuration in wasm-smith to restrict what instructions are available. Currently, the wasm-smith configuration is organized around "proposals," which can be enabled or disabled. In theory, a user could be confused if the proposal was disabled and they explicitly enabled an instruction kind (e.g. reference)--"why aren't reference instructions being generated?" But this accident seems unlikely: `--allowed-instructions` defaults to enabling all kinds, so the user would have to explicitly filter out some kind, deliberately shooting themselves in the foot. Despite some risk of confusion (mitigated by the documentation in this PR), this filtering of instructions kinds ends up being useful in a general way: not only is it a start at fixing the issue above, it is useful for work I am doing to fuzz only parts of the spec.
In looking into bytecodealliance/wasmtime#3251, I created a mechanism for restricting what kinds of instructions wasm-smith can generate. The [WebAssembly specification](https://webassembly.github.io/spec/core/syntax/instructions.html) organizes its instructions into several categories (e.g., numeric, vector, reference, control, etc.) and this change allows the user to configure the module generation based on these categories: ``` head -c 10000 /dev/urandom | cargo run --bin wasm-smith -- --allowed-instructions memory,parametric -o test.wasm && wasm2wat test.wasm ``` There is some related configuration in wasm-smith to restrict what instructions are available. Currently, the wasm-smith configuration is organized around "proposals," which can be enabled or disabled. In theory, a user could be confused if the proposal was disabled and they explicitly enabled an instruction kind (e.g. reference)--"why aren't reference instructions being generated?" But this accident seems unlikely: `--allowed-instructions` defaults to enabling all kinds, so the user would have to explicitly filter out some kind, deliberately shooting themselves in the foot. Despite some risk of confusion (mitigated by the documentation in this PR), this filtering of instructions kinds ends up being useful in a general way: not only is it a start at fixing the issue above, it is useful for work I am doing to fuzz only parts of the spec.
|
@cfallin, in a separate project I did something using
I wasn't too concerned with multiple-instruction sequences and other things mentioned above but if we have a place to put this |
As proposed by @cfallin in bytecodealliance#3251, this change adds a way to generate a Wasm module for a single instruction. It captures the necessary parameter and result types so that fuzzing can not only choose which instruction to check but also generate values to pass to the instruction. Not all instructions are available yet, but a significant portion of scalar instructions are implemented in this change. This does not wire the generator up to any fuzz targets.
As proposed by @cfallin in bytecodealliance#3251, this change adds a way to generate a Wasm module for a single instruction. It captures the necessary parameter and result types so that fuzzing can not only choose which instruction to check but also generate values to pass to the instruction. Not all instructions are available yet, but a significant portion of scalar instructions are implemented in this change. This does not wire the generator up to any fuzz targets.
* fuzz: add a single instruction module generator As proposed by @cfallin in #3251, this change adds a way to generate a Wasm module for a single instruction. It captures the necessary parameter and result types so that fuzzing can not only choose which instruction to check but also generate values to pass to the instruction. Not all instructions are available yet, but a significant portion of scalar instructions are implemented in this change. This does not wire the generator up to any fuzz targets. * review: use raw string in test * review: remove once_cell, use slices * review: refactor macros to use valtype! * review: avoid cloning when choosing a SingleInstModule
In #3186 we found that the WebAssembly spec interpreter may not be suitable for high-throughput fuzzing because of certain performance characteristics. While we were able to locally patch one source of quadratic runtime, there are likely others, and the patch was not really a good fit to upstream. We are looking into other peer implementations to fuzz against for general programs (aside from existing
differential_wasmifuzzing).However, there is another way we could use the reference interpreter: we could validate single instructions' semantics more closely by generating short test cases that just execute an instruction (or a straight-line sequence with no control flow or calls) and return. This would be very valuable for instructions with more complex semantics, such as many of the SIMD instructions.
This is sort of a breadth-vs-depth situation: the "depth" comes from longer-running programs and tests interactions between instructions, and a good oracle there is a more optimized implementation; while the "breadth" comes from exhaustive many-inputs testing of very short sequences and tests mainly that we got the instruction semantics right. The combination seems like it should give pretty good coverage.
cc @abrown @alexcrichton
The text was updated successfully, but these errors were encountered: