Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Continuous Fuzzing Integration with Fuzzit #7509

Merged
merged 4 commits into from
Sep 23, 2019
Merged

Continuous Fuzzing Integration with Fuzzit #7509

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c8f345b
Continuous Fuzzing Integration with Fuzzit
Jul 9, 2019
157a821
migrate fuzzer build to bazel.fuzz target
Aug 23, 2019
85e990f
final fixes
Aug 23, 2019
54dc342
Migrate to AZP
Aug 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 40 additions & 2 deletions .azure-pipelines/linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,10 @@ jobs:
CI_TARGET: 'bazel.gcc'
compile_time_options:
CI_TARGET: 'bazel.compile_time_options'
fuzz:
CI_TARGET: 'bazel.fuzz'
# This will run on every commit/PR and will make sure the corpus generated by the fuzzers as well as fixed crashes
# (on Fuzzit) is not crashing envoy. This will help find bugs BEFORE merging and not after.
fuzzit:
CI_TARGET: 'bazel.fuzzit_regression'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you intended to do this in every PR? This doesn't provide anything in addition to bazel.fuzz above, does it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah. I think it is and this was one of the main reasons integrating fuzzit in addition to OSS Fuzz. this downloads the current corpus+fixed_crashes from Fuzzit Servers and runs the fuzzers through those test-cases which is stronger then just a run for 10 seconds with empty corpus (which is fine but that just checks if the fuzzers compiled successfully)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we merge those two jobs into one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

dependsOn: [] # this removes the implicit dependency on previous stage and causes this to run in parallel.
timeoutInMinutes: 360
pool:
Expand Down Expand Up @@ -53,3 +55,39 @@ jobs:
pathtoPublish: "$(Build.StagingDirectory)/envoy"
artifactName: $(CI_TARGET)
condition: always()

- job: fuzzit_fuzzing
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add some comments here explaining when this runs? Does this happen on every PR, does it block PRs, is this contributing to the PR critical path on CI? I think this PR generally looks awesome, but this needs some clarification, preferably in source comments. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, as explained in the comments there are two workflows:

  • Fuzzing - This will run on every push/merge to master, will build the fuzzers and will upload them to Fuzzit where they will run asynchronous. This will ensure the latest version of the code is always being fuzzed and new bugs are found as new code is added.
  • Regression - This will run on every commit/PR and will run the fuzzers inline in the CI together with the corpus generated on Fuzzit as well as previous fixed crashes. This will ensure bugs are found BEFORE merge.

dependsOn: [] # this removes the implicit dependency on previous stage and causes this to run in parallel.
timeoutInMinutes: 360
# this runs on every push to master / merge to master. this will build the fuzzers and will update them on Fuzzit where
# they will run asynchronously. Essentially this will make sure that the latest master version is always being fuzzed
# continuously.
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'), ne(variables['Build.Reason'], 'PullRequest'))
pool:
vmImage: 'Ubuntu 16.04'
steps:
- bash: |
echo "disk space at beginning of build:"
df -h
displayName: "Check disk space at beginning"

- bash: |
sudo mkdir -p /etc/docker
echo '{
"ipv6": true,
"fixed-cidr-v6": "2001:db8:1::/64"
}' | sudo tee /etc/docker/daemon.json
sudo service docker restart
displayName: "Enable IPv6"

- script: ci/run_envoy_docker.sh 'ci/do_ci.sh bazel.fuzzit_fuzzing'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add FUZZIT_API_KEY env (it can be in FUZZIT_API_KEY=$(FuzzitApiKey) before ci/do_ci.sh, or env below and add propagate it in ci/run_envoy_docker.sh.

workingDirectory: $(Build.SourcesDirectory)
env:
FUZZIT_API_KEY: $(FuzzitApiKey)
ENVOY_DOCKER_BUILD_DIR: $(Build.StagingDirectory)
ENVOY_RBE: "true"
BAZEL_BUILD_EXTRA_OPTIONS: "--config=remote-ci --config=remote --jobs=100 --curses=no"
BAZEL_REMOTE_CACHE: grpcs://remotebuildexecution.googleapis.com
BAZEL_REMOTE_INSTANCE: projects/envoy-ci/instances/default_instance
GCP_SERVICE_ACCOUNT_KEY: $(GcpServiceAccountKey)
displayName: "Fuzzit Regression"
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ involved and how Envoy plays a role, read the CNCF
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1266/badge)](https://bestpractices.coreinfrastructure.org/projects/1266)
[![Azure Pipelines](https://dev.azure.com/cncf/envoy/_apis/build/status/envoyproxy.envoy.mac?branchName=master)](https://dev.azure.com/cncf/envoy/_build/latest?definitionId=2&branchName=master)
[![CircleCI](https://circleci.com/gh/envoyproxy/envoy/tree/master.svg?style=shield)](https://circleci.com/gh/envoyproxy/envoy/tree/master)
[![fuzzit](https://app.fuzzit.dev/badge?org_id=envoyproxy)](https://app.fuzzit.dev/orgs/envoyproxy/dashboard)
[![Jenkins](https://img.shields.io/jenkins/s/https/powerci.osuosl.org/job/build-envoy-master/badge/icon/.svg?label=ppc64le%20build)](http://powerci.osuosl.org/job/build-envoy-master/)

## Documentation
Expand Down
16 changes: 16 additions & 0 deletions ci/do_ci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,22 @@ elif [[ "$CI_TARGET" == "bazel.fuzz" ]]; then
echo "Building envoy fuzzers and executing 100 fuzz iterations..."
bazel_with_collection test ${BAZEL_BUILD_OPTIONS} --config=asan-fuzzer ${FUZZ_TEST_TARGETS} --test_arg="-runs=10"
exit 0
elif [[ "$CI_TARGET" == "bazel.fuzzit_regression" ]]; then
setup_clang_toolchain
FUZZ_TEST_TARGETS="$(bazel query "attr('tags','fuzzer',${TEST_TARGETS})")"
echo "bazel ASAN libFuzzer build with fuzz tests ${FUZZ_TEST_TARGETS}"
echo "Building fuzzers and run a regression with corpus from Fuzzit"
bazel_with_collection build ${BAZEL_BUILD_OPTIONS} --config=asan-fuzzer ${FUZZ_TEST_TARGETS}
./ci/run_fuzzit.sh local-regression
exit 0
elif [[ "$CI_TARGET" == "bazel.fuzzit_fuzzing" ]]; then
setup_clang_toolchain
FUZZ_TEST_TARGETS="$(bazel query "attr('tags','fuzzer',${TEST_TARGETS})")"
echo "bazel ASAN libFuzzer build with fuzz tests ${FUZZ_TEST_TARGETS}"
echo "Build fuzzers and push them to Fuzzit servers for continuous fuzzing"
bazel_with_collection build ${BAZEL_BUILD_OPTIONS} --config=asan-fuzzer ${FUZZ_TEST_TARGETS}
./ci/run_fuzzit.sh fuzzing
exit 0
elif [[ "$CI_TARGET" == "fix_format" ]]; then
echo "fix_format..."
./tools/check_format.py fix
Expand Down
4 changes: 2 additions & 2 deletions ci/run_envoy_docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ USER_GROUP=root
mkdir -p "${ENVOY_DOCKER_BUILD_DIR}"
# Since we specify an explicit hash, docker-run will pull from the remote repo if missing.
docker run --rm ${DOCKER_TTY_OPTION} -e HTTP_PROXY=${http_proxy} -e HTTPS_PROXY=${https_proxy} \
-u "${USER}":"${USER_GROUP}" -v "${ENVOY_DOCKER_BUILD_DIR}":/build ${GIT_VOLUME_OPTION} \
-u "${USER}":"${USER_GROUP}" -v "${ENVOY_DOCKER_BUILD_DIR}":/build -v /var/run/docker.sock:/var/run/docker.sock ${GIT_VOLUME_OPTION} \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch, thanks

-e BAZEL_BUILD_EXTRA_OPTIONS -e BAZEL_EXTRA_TEST_OPTIONS -e BAZEL_REMOTE_CACHE \
-e BAZEL_REMOTE_INSTANCE -e GCP_SERVICE_ACCOUNT_KEY -e NUM_CPUS -e ENVOY_RBE \
-e BAZEL_REMOTE_INSTANCE -e GCP_SERVICE_ACCOUNT_KEY -e NUM_CPUS -e ENVOY_RBE -e FUZZIT_API_KEY \
-v "$PWD":/source --cap-add SYS_PTRACE --cap-add NET_RAW --cap-add NET_ADMIN "${IMAGE_NAME}":"${IMAGE_ID}" \
/bin/bash -lc "groupadd --gid $(id -g) -f envoygroup && useradd -o --uid $(id -u) --gid $(id -g) --no-create-home \
--home-dir /source envoybuild && usermod -a -G pcap envoybuild && su envoybuild -c \"cd source && $*\""
33 changes: 33 additions & 0 deletions ci/run_fuzzit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/bash -eux
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you really want -x here? This might expose credentials.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no credentials are passed in this file (it is taken straight from the environment variables. I removed the need for fuzzit auth ${FUZZIT_API_KEY))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still would like to play it safe and not have -x except for temporary debug situations on CI.


# Dynamically source fuzzing targets
declare -r FUZZER_TARGETS_CC=$(find . -name *_fuzz_test.cc)
declare -r FUZZER_TARGETS="$(for t in ${FUZZER_TARGETS_CC}; do echo "${t:2:-3}"; done)"

declare BAZEL_BUILD_TARGETS=""
for t in ${FUZZER_TARGETS}
do
declare BAZEL_PATH="//"$(dirname "$t")":"$(basename "$t")
declare TAGGED=$(bazel query "attr('tags', 'no_fuzz', ${BAZEL_PATH})")
if [ -z "${TAGGED}" ]
then
FILTERED_FUZZER_TARGETS+="$t "
fi
done
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is OK. I think what @asraa did in https://github.com/envoyproxy/envoy/blob/master/ci/do_ci.sh#L258 is even shorter and more maintainable though.



# run fuzzing regression or upload to Fuzzit for long running fuzzing job ($1 is either local-regression or fuzzing)
wget -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v2.4.55/fuzzit_Linux_x86_64
chmod a+x fuzzit

PREFIX=$(realpath /build/tmp/_bazel_bazel/*/execroot/envoy/bazel-out/k8-fastbuild/bin)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$(bazel info output_path) for the path until bazel-out

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like it doesn't work:( https://circleci.com/gh/envoyproxy/envoy/269968?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, maybe because the env vars is different outside of do_ci.sh, fine to keep as is.

for t in ${FILTERED_FUZZER_TARGETS}
do
TARGET_BASE="$(expr "$t" : '.*/\(.*\)_fuzz_test')"
# Fuzzit target names can't contain underscore
FUZZIT_TARGET_NAME=${TARGET_BASE//_/-}
if [ $1 == "fuzzing" ]; then
./fuzzit create target --skip-if-exists --public-corpus envoyproxy/"${FUZZIT_TARGET_NAME}"
fi
./fuzzit create job --skip-if-not-exists --type $1 envoyproxy/"${FUZZIT_TARGET_NAME}" "${PREFIX}"/"${t}"_with_libfuzzer
done