Feature: add other fuzzing platforms to the Fuzzing check

[laurentsimon]

Our current check only looks for OSS-Fuzz integration. We want to expand our list of supported platforms.

Which fuzzing platforms to support remains an open question. Suggestions welcome!

[oliverchang]

Another one to add would be clusterfuzzlite which is launching soon: https://github.com/google/clusterfuzzlite.

[laurentsimon]

Thanks Oliver. To detect clusterfuzzlite, we need to check for a workflow which uses google/clusterfuzzlite/actions/run_fuzzers action, is this correct?
Do you expect users to enable it on PRs or push to main or something else?

[inferno-chromium]

If this is simple enough, should add this in check soon.

[naveensrinivasan]

What about results of this check from this Fuzz?

[naveensrinivasan]

On a different note should we add this check to scorecard?

[oliverchang]

Thanks Oliver. To detect clusterfuzzlite, we need to check for a workflow which uses google/clusterfuzzlite/actions/run_fuzzers action, is this correct? Do you expect users to enable it on PRs or push to main or something else?

Yep! Just check for the google/clusterfuzzlite/actions/run_fuzzers action. I think we can be flexible about where it's enabled (PR or push or otherwise) so we don't need to check that.

What about results of this check from this Fuzz?
On a different note should we add this check to scorecard?

@naveensrinivasan Could you clarify what you mean here? Are you suggesting a more detailed check beyond just checking if Fuzzing is enabled?

[naveensrinivasan]

Thanks Oliver. To detect clusterfuzzlite, we need to check for a workflow which uses google/clusterfuzzlite/actions/run_fuzzers action, is this correct? Do you expect users to enable it on PRs or push to main or something else?

Yep! Just check for the google/clusterfuzzlite/actions/run_fuzzers action. I think we can be flexible about where it's enabled (PR or push or otherwise) so we don't need to check that.

What about results of this check from this Fuzz?
On a different note should we add this check to scorecard?

@naveensrinivasan Could you clarify what you mean here? Are you suggesting a more detailed check beyond just checking if Fuzzing is enabled?

Yes , how do we report the results of these fuzz runs?

[laurentsimon]

Thanks Oliver. To detect clusterfuzzlite, we need to check for a workflow which uses google/clusterfuzzlite/actions/run_fuzzers action, is this correct? Do you expect users to enable it on PRs or push to main or something else?

Yep! Just check for the google/clusterfuzzlite/actions/run_fuzzers action. I think we can be flexible about where it's enabled (PR or push or otherwise) so we don't need to check that.

great! Do you have cycles to add it before you officially release clusterfuzzlite? Or shall we add this to v4 milestone (~EOY)?

[oliverchang]

Thanks Oliver. To detect clusterfuzzlite, we need to check for a workflow which uses google/clusterfuzzlite/actions/run_fuzzers action, is this correct? Do you expect users to enable it on PRs or push to main or something else?

Yep! Just check for the google/clusterfuzzlite/actions/run_fuzzers action. I think we can be flexible about where it's enabled (PR or push or otherwise) so we don't need to check that.

great! Do you have cycles to add it before you officially release clusterfuzzlite? Or shall we add this to v4 milestone (~EOY)?

I can probably knock this one out in the next week or so.

[laurentsimon]

Awesome. I've created #1148 and added to v4 milestone.

[github-actions]

Stale issue message

[laurentsimon]

@naveensrinivasan is there a special command/keyword we can use to say "this issue should not be closed automatically by the bot"?

[georgettica]

I wanted to also not that go1.18 has fuzzing out of the box. didn't know if it should be included in this issue or a different one

[laurentsimon]

I think it fits in this issue. If you know how to check for its use (there's a command for it?) please let us know. Feel free to send. aPR if you have time for it. Thanks!

[georgettica]

I'll see what I can do :)

[naveensrinivasan]

I like the idea. In the future scorecard plans to provide fuzzing coverage. Having something like oss-fuzz provides an API to get the coverage metrics. I don't know how we can get it from go 1.18 and verify it is correct.

[georgettica]

I think oss-fuzz integrates with the native fizzing, but not sure how exactly.
Worth checking unrelated to my investigation

[laurentsimon]

We need to look for the --fuzz command in the workflow https://go.dev/doc/fuzz/. The assumption is that the command is used directly in the workflow, and not in a script that's invoked via the workflow. We can start simple and then iterate, though

[azeemshaikh38]

@georgettica @laurentsimon could we start with checking for *_test.go files in the repository which contain functions matching the regex func Fuzz* (* \*testing.F) ? Would that be a good start for adding this support?

[georgettica]

I think go test -fuzz takes any string and it's convention to use the prefix Fuzz
Not that I am opposed to the ideas, but letting you know it's based on convention rather than validation

[azeemshaikh38]

https://go.dev/doc/fuzz/ mentions this as a requirement. But maybe this is not a hard requirement?

[azeemshaikh38]

Did some local testing - go test -fuzz ignores functions not starting with Fuzz*.

[georgettica]

yeah, same (verified aswell)
expected the og test -fuzz=Bar to at least. warn me

[laurentsimon]

I think it's fair to start with the regex @azeemshaikh38 provided. The warning is something Go team should fix, rather than us, no? Note that a while back, we were contemplating having a Unit-Test checks (or part of an existing check) so the work you're doing could be useful in this context too.

[georgettica]

Yup, it's a go team thing. Noting it here as I didn't start the conversation there

[oliverchang]

If we are extending the Fuzzing check to include looking for fuzzer definitions for Go (which I think is a good idea!), we should also extend this to other languages, which all have a consistent way to do this that we can detect.

e.g. for C/C++, we can look for instances of LLVMFuzzerTestOneInput in c/cc/cxx/cpp files.

[inferno-chromium]

@oliverchang - can someone from fuzzing team pick this, it will be nice to have this scorecard check comprehensive.

[oliverchang]

@Navidem -- this (#897 (comment)) might be a interesting side project to tackle!

[oliverchang]

@Navidem will start working on the C/C++ side of things this quarter. Is anyone working on the Go part?

[laurentsimon]

@Navidem feel free to ask questions if you need some pointers

[oliverchang]

Fuzzing patterns to add:

• C/C++: https://llvm.org/docs/LibFuzzer.html#fuzz-target
• Other languages: See the language subguides on the canonical ways to write fuzzers in other languages on https://google.github.io/oss-fuzz/getting-started/new-project-guide/

[laurentsimon]

Thanks @oliverchang !

/cc @aidenwang9867

[aidenwang9867]

Thanks @oliverchang !

/cc @aidenwang9867

Thanks Laurent & Oliver, I'll take the Go side. @laurentsimon @oliverchang

Also, cc @Navidem :P

[aidenwang9867]

@oliverchang @Navidem

In the merged PR#1979, I added support for checking Go built-in fuzzers as a part of scorecard's fuzzing check. Most of the important changes have been made in checks/raw/fuzzing.go, please refer to the file changes in this PR to see them.

Built-in fuzzer patterns for other programming languages can be added here. I'll work on another feature for a while but I might implement some fuzzing supports for other languages if I have time. I'll ping you here if I start to do that. :)

cc @laurentsimon

[balteravishay]

what about One-Fuzz? Can it be supported by Scorecard? Is there anything we can help with to support that?

[laurentsimon]

Absolutely. What does the integration in a repository look like?
For example, this is how we check for clusterfuzzlite:

scorecard/checks/raw/fuzzing.go

Line 120 in 69eb1cc

func checkCFLite(c *checker.CheckRequest) (bool, error) {

If you tell us what we should be looking for in thee repo, we can add a function to check for it in the file linked above.

[balteravishay]

@laurentsimon, sorry for the delay, how about something like that: #2141 😄

[DavidKorczynski]

Added support for C, CPP, Python, Java and Rust fuzzing in #3473

Also opened up a discussion for splitting fuzzing checks in two categories: (1) a check for the presence of source code for a fuzzer and (2) a check for whether a project is continuously fuzzed. Issue is here: #3475

[github-actions]

This issue is stale because it has been open for 60 days with no activity.