Continuous Fuzzing Integration with Fuzzit #7509

yevgenypats · 2019-07-09T18:16:16Z

Please do not merge yet. This is work in progress.
The contribution was discuss in the mailing-list with @htuch , @mattklein123 .

Description:
This will introduce another platform (apart from oss) fuzz that
will run the long-running fuzzers as well as will introduce
"sanity fuzzers" that will run the accumlated corpus and crashes
on every Pull-Request to detect bugs early-on in the development
cycle.

Risk Level:
Low - as this will introduce only another step in CircleCI where the fuzzers will be uploaded to Fuzzit and the heavy lifting will be there.

Testing:
No code is added just a CI code in Circle

Docs Changes:
Will be updated later.

Release Notes: None
[Optional Fixes #Issue] None
[Optional Deprecated:] None

htuch · 2019-07-09T19:28:43Z

ci/run_fuzzit.sh

+export ENVOY_SRC_PATH=`pwd`
+cd oss-fuzz
+python infra/helper.py build_image --pull envoy
+python infra/helper.py build_fuzzers --sanitizer=address envoy ${ENVOY_SRC_PATH}


One key issue here is build performance; if this takes a significant amount of time (which anecdotally it does from local runs), e.g. > 1 hour, it will become a CI bottleneck. Many of our CI jobs now benefit from Bazel caching and (in the future) RBE enablement, this one is going to be inherently stuck without this, since the oss-fuzz image doesn't have any plumbing for this. CC @lizan

@htuch This is exactly what I'm trying to do just wanted to see if it works with the current oss-fuzz images. Anyway I was able to compile the targets with bazel for example

using bazel build //test/server:server_fuzz_test the build completes successfully but it seems that it doesn't link against libFuzzer. what would be the right command to link it against libfuzzer (i.e -fsanitize=fuzzer) as I don't wont to use the oss docker both to use the caching and also not to use the nested docker in Circle.

Yeah, I'm afraid you're going to have to do some gnarly Bazel hacking to get it to link against libfuzzer under Bazel. We would be super appreciative of this work, but it will involve diving into https://github.com/envoyproxy/envoy/blob/master/bazel/envoy_test.bzl#L66 and figuring out how to cleanly offer both the corpus run (using the existing test driver) and a way to on the Bazel CLI configure a link against libfuzzer (and somehow getting libfuzzer included in the Envoy build).

Thanks! I'll find the way to do that. I just wanted to make sure that I'm not doing double work and it's not implemented yet.

htuch · 2019-07-09T19:29:14Z

ci/run_fuzzit.sh

+git clone https://github.com/google/oss-fuzz.git
+export ENVOY_SRC_PATH=`pwd`
+cd oss-fuzz
+python infra/helper.py build_image --pull envoy


Are we doing nested Docker here? I.e. running oss-fuzz Docker build inside a CircleCI Docker env?

I'm exactly working on this to make it work not inside the oss-fuzz docker.

yevgenypats · 2019-07-16T12:37:13Z

@htuch looks like I get some no-credits error at CircleCI? any particular reason for this to happen? I can test my changes.

htuch · 2019-07-16T13:52:31Z

@yevgenypats this is a general Envoy CI issue, we are working to resolve.

htuch · 2019-07-16T22:23:50Z

/retest

repokitteh-read-only · 2019-07-16T22:23:54Z

🔨 rebuilding ci/circleci: Build Error (failed build)

🐱

Caused by: a #7509 (comment) was created by @htuch.

see: more, trace.

yevgenypats · 2019-07-17T05:14:43Z

@htuch thx. just an update. look like it works now. I'll update when the PR will be ready for review. should be soon. cheers.

lizan · 2019-09-20T09:56:16Z

ci/do_ci.sh

+  FUZZ_TEST_TARGETS="$(bazel query "attr('tags','fuzzer',${TEST_TARGETS})")"
+  echo "bazel ASAN libFuzzer build with fuzz tests ${FUZZ_TEST_TARGETS}"
+  echo "Building envoy fuzzers and executing 100 fuzz iterations..."
+  bazel_with_collection build ${BAZEL_BUILD_OPTIONS} --config=asan-fuzzer ${FUZZ_TEST_TARGETS} --test_arg="-runs=10"


no need --test_arg

lizan · 2019-09-20T09:56:26Z

ci/do_ci.sh

+  FUZZ_TEST_TARGETS="$(bazel query "attr('tags','fuzzer',${TEST_TARGETS})")"
+  echo "bazel ASAN libFuzzer build with fuzz tests ${FUZZ_TEST_TARGETS}"
+  echo "Building envoy fuzzers and executing 100 fuzz iterations..."
+  bazel_with_collection build ${BAZEL_BUILD_OPTIONS} --config=asan-fuzzer ${FUZZ_TEST_TARGETS} --test_arg="-runs=10"


lizan · 2019-09-20T09:58:06Z

.azure-pipelines/linux.yml

@@ -12,6 +12,8 @@ jobs:
        CI_TARGET: 'bazel.compile_time_options'
      fuzz:
        CI_TARGET: 'bazel.fuzz'
+      fuzzit:
+        CI_TARGET: 'bazel.fuzzit_regression'


are you intended to do this in every PR? This doesn't provide anything in addition to bazel.fuzz above, does it?

yeah. I think it is and this was one of the main reasons integrating fuzzit in addition to OSS Fuzz. this downloads the current corpus+fixed_crashes from Fuzzit Servers and runs the fuzzers through those test-cases which is stronger then just a run for 10 seconds with empty corpus (which is fine but that just checks if the fuzzers compiled successfully)

can we merge those two jobs into one?

lizan · 2019-09-20T09:59:51Z

.azure-pipelines/linux.yml

+        sudo service docker restart
+      displayName: "Enable IPv6"
+
+    - script: ci/run_envoy_docker.sh 'ci/do_ci.sh bazel.fuzzit_fuzzing'


Add FUZZIT_API_KEY env (it can be in FUZZIT_API_KEY=$(FuzzitApiKey) before ci/do_ci.sh, or env below and add propagate it in ci/run_envoy_docker.sh.

lizan · 2019-09-20T10:26:01Z

ci/do_ci.sh

+  setup_clang_toolchain
+  FUZZ_TEST_TARGETS="$(bazel query "attr('tags','fuzzer',${TEST_TARGETS})")"
+  echo "bazel ASAN libFuzzer build with fuzz tests ${FUZZ_TEST_TARGETS}"
+  echo "Building envoy fuzzers and executing 100 fuzz iterations..."


update comment as well.

lizan · 2019-09-20T10:26:59Z

.azure-pipelines/linux.yml

@@ -12,6 +12,8 @@ jobs:
        CI_TARGET: 'bazel.compile_time_options'
      fuzz:
        CI_TARGET: 'bazel.fuzz'
+      fuzzit:
+        CI_TARGET: 'bazel.fuzzit_regression'


can we merge those two jobs into one?

yevgenypats · 2019-09-20T17:16:32Z

@lizan fixed the review:)

lizan · 2019-09-20T21:05:26Z

@htuch?

htuch

/wait

htuch · 2019-09-23T15:05:49Z

.azure-pipelines/linux.yml

@@ -53,3 +53,36 @@ jobs:
      pathtoPublish: "$(Build.StagingDirectory)/envoy"
      artifactName: $(CI_TARGET)
    condition: always()
+
+- job: fuzzit_fuzzing


Can you add some comments here explaining when this runs? Does this happen on every PR, does it block PRs, is this contributing to the PR critical path on CI? I think this PR generally looks awesome, but this needs some clarification, preferably in source comments. Thanks!

Done, as explained in the comments there are two workflows:

Fuzzing - This will run on every push/merge to master, will build the fuzzers and will upload them to Fuzzit where they will run asynchronous. This will ensure the latest version of the code is always being fuzzed and new bugs are found as new code is added.

Regression - This will run on every commit/PR and will run the fuzzers inline in the CI together with the corpus generated on Fuzzit as well as previous fixed crashes. This will ensure bugs are found BEFORE merge.

Signed-off-by: Yevgeny Pats <[email protected]>

yevgenypats · 2019-09-23T16:22:28Z

@htuch why this fails now ?https://circleci.com/gh/envoyproxy/envoy/279964?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

yevgenypats · 2019-09-23T16:59:53Z

/retest

repokitteh-read-only · 2019-09-23T16:59:57Z

🔨 rebuilding ci/circleci: docs (failed build)

🐱

Caused by: a #7509 (comment) was created by @yevgenypats.

see: more, trace.

yevgenypats · 2019-09-23T17:27:00Z

@htuch passing!:)

htuch

Thanks @yevgenypats. Appreciate your patience and work on iterating to find the best fit with Envoy CI and fuzzing. Also thanks @lizan for all the help with the CI aspects.

yevgenypats · 2019-09-23T21:29:17Z

@htuch @lizan sure thing, thank you! and feel free to ping me if there are any issues/changes you need.

also @lizan are you sure the FUZZIT_API_KEY is available because I see that the script didn't find it (I couldn't check it before merge as this is environment variable is not available for forked-PRs)

Also looks there are some crashes already https://app.fuzzit.dev/orgs/envoyproxy/dashboard

and of course feel free to RT https://twitter.com/fuzzitdev/status/1176246876821217283

Thanks!

htuch · 2019-09-23T21:45:28Z

@yevgenypats are these crashes limited in visibility? We would like to maintain embargo on any zero days and only have https://github.com/google/oss-fuzz/blob/master/projects/envoy/project.yaml#L3 see these.

yevgenypats · 2019-09-23T21:48:31Z

@htuch yes of course, like we discussed. currently only @lizan has access I can add you to the email notifications but you need to sign-up at https://app.fuzzit.dev so I can add you to the envoy account so you will have access to the crashes and other data.

lizan · 2019-09-23T21:54:38Z

I set the api key right now, should be fixed in future builds

yevgenypats · 2019-09-23T21:55:58Z

Great.

…

On Mon, Sep 23, 2019, 5:55 PM Lizan Zhou ***@***.***> wrote: I set it right now, should be fixed in future builds — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7509?email_source=notifications&email_token=AD52CDUYCVHBV4TWPG5AYHTQLE3LVA5CNFSM4H7ICZEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7MMOZY#issuecomment-534300519>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD52CDRJ4YHQ2IFSKL2UCEDQLE3LVANCNFSM4H7ICZEA> .

lizan · 2019-09-24T04:33:25Z

@yevgenypats now it is failing with

+ ./fuzzit create target --skip-if-exists --public-corpus envoyproxy/access-log-formatter
2019/09/23 23:20:23 Creating target...
2019/09/23 23:20:23 target can only contain lowercase characetrs, numbers and hypens

This will introduce another platform (apart from oss) fuzz that will run the long-running fuzzers as well as will introduce "sanity fuzzers" that will run the accumlated corpus and crashes on every Pull-Request to detect bugs early-on in the development cycle. Risk Level: Low - as this will introduce only another step in CircleCI where the fuzzers will be uploaded to Fuzzit and the heavy lifting will be there. Testing: No code is added just a CI code in Circle Signed-off-by: Yevgeny Pats <[email protected]>

yevgenypats force-pushed the fuzzit_integration branch 4 times, most recently from 13d63ca to 206ec5f Compare July 9, 2019 18:35

htuch reviewed Jul 9, 2019

View reviewed changes

htuch added the waiting label Jul 9, 2019

alyssawilk assigned htuch Jul 9, 2019

yevgenypats force-pushed the fuzzit_integration branch from 206ec5f to 5594a38 Compare July 9, 2019 20:40

repokitteh-read-only bot removed the waiting label Jul 9, 2019

yevgenypats force-pushed the fuzzit_integration branch 3 times, most recently from e1b4af6 to 1178f97 Compare July 10, 2019 06:22

htuch added the waiting:any label Jul 12, 2019

repokitteh-read-only bot removed the waiting:any label Jul 12, 2019

htuch added the waiting label Jul 12, 2019

yevgenypats force-pushed the fuzzit_integration branch from 1178f97 to a0cb5b4 Compare July 16, 2019 12:27

repokitteh-read-only bot removed the waiting label Jul 16, 2019

yevgenypats force-pushed the fuzzit_integration branch from a0cb5b4 to b704a74 Compare July 16, 2019 12:36

yevgenypats force-pushed the fuzzit_integration branch from b704a74 to 958e2c6 Compare July 16, 2019 13:40

htuch added the waiting label Jul 16, 2019

yevgenypats force-pushed the fuzzit_integration branch from 958e2c6 to 3c3ec56 Compare July 17, 2019 05:13

repokitteh-read-only bot removed the waiting label Jul 17, 2019

yevgenypats force-pushed the fuzzit_integration branch 3 times, most recently from 59876ab to e1b1881 Compare July 17, 2019 07:56

lizan reviewed Sep 20, 2019

View reviewed changes

yevgenypats force-pushed the fuzzit_integration branch from 61d474a to 562831f Compare September 20, 2019 10:23

lizan reviewed Sep 20, 2019

View reviewed changes

yevgenypats force-pushed the fuzzit_integration branch from 562831f to 5c127f8 Compare September 20, 2019 12:00

lizan previously approved these changes Sep 20, 2019

View reviewed changes

lizan requested a review from htuch September 20, 2019 21:05

htuch suggested changes Sep 23, 2019

View reviewed changes

repokitteh-read-only bot added the waiting label Sep 23, 2019

Migrate to AZP

54dc342

Signed-off-by: Yevgeny Pats <[email protected]>

yevgenypats dismissed lizan’s stale review via 54dc342 September 23, 2019 15:40

yevgenypats force-pushed the fuzzit_integration branch from 5c127f8 to 54dc342 Compare September 23, 2019 15:40

repokitteh-read-only bot removed the waiting label Sep 23, 2019

htuch approved these changes Sep 23, 2019

View reviewed changes

htuch merged commit bbae4a7 into envoyproxy:master Sep 23, 2019

yevgenypats mentioned this pull request Sep 24, 2019

update fuzzit CLI to v2.4.57 #8342

Merged

Continuous Fuzzing Integration with Fuzzit #7509

Continuous Fuzzing Integration with Fuzzit #7509

Conversation

yevgenypats commented Jul 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yevgenypats commented Jul 16, 2019

htuch commented Jul 16, 2019 • edited Loading

htuch commented Jul 16, 2019

repokitteh-read-only bot commented Jul 16, 2019

yevgenypats commented Jul 17, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yevgenypats commented Sep 20, 2019

lizan commented Sep 20, 2019

htuch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yevgenypats commented Sep 23, 2019

yevgenypats commented Sep 23, 2019

repokitteh-read-only bot commented Sep 23, 2019

yevgenypats commented Sep 23, 2019

htuch left a comment

Choose a reason for hiding this comment

yevgenypats commented Sep 23, 2019

htuch commented Sep 23, 2019

yevgenypats commented Sep 23, 2019

lizan commented Sep 23, 2019 • edited Loading

yevgenypats commented Sep 23, 2019 via email

lizan commented Sep 24, 2019

htuch commented Jul 16, 2019 •

edited

Loading

lizan commented Sep 23, 2019 •

edited

Loading