Skip to content

Commit

Permalink
sync (#5)
Browse files Browse the repository at this point in the history
* [cryptofuzz] Add Microsoft SymCrypt (google#3826)

* [gRPC-gateway] Initial integration (google#3807)

* [doc] Update new_project_guide.md (google#3828)

* [knot-dns] link against liblmdb statically (google#3797)

Relates to https://gitlab.labs.nic.cz/knot/knot-dns/-/commit/d4ec3a3aa8b7e2ba4d196e2f7984173069e3d91b

Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22191

* Simplify rust project setup. (google#3830)

* Simplify rust project setup.

- Add rust and cargo-fuzz in base builder.
- Set RUSTC_BOOSTRAP to make ASan available.
- Set RUSTFLAGS and C,CXXFLAGS properly.

* Quick fix for helper.py build_fuzzers. (google#3832)

* Parse project language and use in infra/helper.py (google#3834)

* Parse project language and use in infra/helper.py

* Fix exception message.

* Add FUZZING_LANGUAGE to build step. (google#3835)

* Add fuzzing language correctly during compile step. (google#3837)

* Remove unneeded rust hacks after google#3830. (google#3840)

* Fix missing FUZZING_LANGUAGE in coverage build. (google#3843)

* Use hardcoded FUZZING_LANGUAGE to unbreak CIFuzz. (google#3844)

* solidity: Disable Z3 build in dockerfile (google#3831)

* cifuzz: use pull_request.number instead of GITHUB_REF (google#3845)

Closes google#3732

* wasmtime: build fuzz targets with --all-features (google#3850)

This enables not only the binaryen-using fuzz targets, but also the peepmatic fuzz targets (which is necessary after bytecodealliance/wasmtime#1727).

* Relax project language requirement. (google#3846)

This is necessary for the bisector to be able to build older revisions.

Print a warning instead.

* libzmq: add maintainer, add UBSAN (google#3829)

* Add another maintainer to libzmq's CC list

* Enable UBSAN for libzmq

* libzmq: disable afl fuzzer, CI fails

* libsodium: do not let libsodium's autogen.sh download files from gnu.org

There is no sanity check and if the download fails because gnu.org is down
the build fails with unhelpful errors

* Fixed changes added in clang-11.0. (google#3852)

* Bump httplib2 from 0.11.3 to 0.18.0 in /infra/gcb (google#3853)

Bumps [httplib2](https://github.com/httplib2/httplib2) from 0.11.3 to 0.18.0.
- [Release notes](https://github.com/httplib2/httplib2/releases)
- [Changelog](https://github.com/httplib2/httplib2/blob/master/CHANGELOG)
- [Commits](httplib2/httplib2@v0.11.3...v0.18.0)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [TiDB] initial integration (google#3849)

* [Minify] Initial integration (google#3848)

* Set rust flags always in compile step. (google#3859)

This is needed for projects that mix both rust and c/c++.

* Remove unneeded RUSTC_BOOTSTRAP since nightly is used

* [fasthttp] Initial integration (google#3858)

* [libfido2] bump libcbor version from 0.5.0 to 0.7.0 (google#3861)

* Fix cras build failure, use default rust nightly.

* Don't add Rust sanitizer flags when using ubsan or i386. (google#3862)

Rust sanitizer flags are not supported for these configs.
Fixes Cras and ecc-diff-fuzzer builds.

* [hermes] Initial setup for Hermes (google#3698)

* [poppler] Exercise a few more methods (google#3857)

* [minify] Moved build script upstream (google#3867)

* Use available nightly rust for spidermonkey build

* Use available rust nightly in spidermonkey-ufi

* Use available rust nightly in firefox

* [python3-libraries] Fix build (google#3869)

* Use latest git version. (google#3870)

Adds features needed by firefox and spidermonkey-ufi builds.

* libzmq: fix MSAN (google#3868)

* libzmq: build libsodium with --disable-asm to avoid false positives in MSAN checks

* libzmq: set autoconf install prefix to avoid /usr/local being added to the linker library paths

A non-working version of the standard library is installed in oss-fuzz's images
under /usr/local/lib. Using it breaks MSAN's tests with false positives.
Set the prefix in autoconf to something different from the default
/usr/local, as /lib is added by autoconf automatically to the linker
flags, which means this broken standard library is used instead of the
instrumented one in /usr/lib.

* build_specified_commit: revert to using cp. (google#3871)


rsync seems to have disappeared in the latest images somehow, and we
can't rely on it always existing.

* Don't add sanitizer=coverage in RUSTFLAGS

* Remove libpcap patch merged upstream (google#3865)

* Install Python 3.8.3 on builder image. (google#3874)

Part of fixing google#3756.

* [qt] Get a shallow clone of qtbase (google#3872)

Reduces the size by nearly 200 MB

* [qt] Add fuzzer for QImage::loadFromData() (google#3873)

* Fix jsoncpp build

* build_specified_commit: Don't replace gitdir if already relative. (google#3875)

Upgraded git broke this.

* [skia] Speculative build fix (google#3847)

* [stb] extend code coverage for stb_image (google#3784)

* [stb] extend code coverage for stb_image

* remove wget's from build.sh

* add wget's to Dockerfile

* fix Dockerfile

* [hermes] Fix coverage build (google#3877)

* Update email address to non-alias for clusterfuzz login (google#3878)

* add hosts and interns (google#3879)

Signed-off-by: Asra Ali <[email protected]>

* [qt] Don't install build-essential (google#3880)

It's in base-builder now.

* [hermes] Add CC for issues (google#3882)

* [systemd] turn off hongfuzz (google#3889)

See google#3887

* Remove libbz2-dev dependency installed in base-builder for python3 (google#3890)

* Remove libbz2-dev dependency installed in base-builder for python3

Fixes google#3888

* Revert "[systemd] turn off hongfuzz (google#3889)" (google#3891)

This reverts commit d638fac.

* Set up python in one docker layer, remove python deps. (google#3893)

* Set up python in one docker layer.

Address review comment in google#3890 (review)

* Remove all python dependencies, keep build-essential.

* Makes suricata work with rust sanitizers (google#3897)

* [firefox] Fix libFuzzer cloning process (google#3896)

* Fix spidermonkey-ufi build.

* [nanopb] Use $(which python3) instead of hardcoding /usr/bin/python3 (google#3895)

After the addition of Python 3.8 in google#3874, the default python3 is
now /usr/local/bin/python3. The nanopb Dockerfile hardcoded a path
of /usr/bin/python3, causing again problems of Python packages being
installed for different version than attempting to run with.

* Fix CIFuzz issue where targets assumed in OSS-Fuzz build if exists (google#3817)

Make some other changes:
1. Refactor fuzz_target.py and fuzz_target_test.py
2. Introduce pyfakefs and parameterized as dependencies and use them
in tests.
3. Fix infra-tests in CI so that they use installed dependencies

* Fix spidermonkey-ufi build

* Extend Zeek project auto_ccs (google#3898)

* [libgd] Add another fuzzer to libgd (google#3892)

* [qt] Add qtdeclarative (google#3901)

* [qt] Add qtdeclarative

* [qt] Ignore stdout from make

"The job exceeded the maximum log length, and has been terminated."

* Updating jsoncpp primary contact (google#3900)

* [php] Remove --enable-json (google#3899)

JSON is now always compiled in, and cannot be explicitly enabled.

* Don't create empty corpus archive as it wont unpack. (google#3903)

* Update email addresses for Prometheus (google#3905)

Signed-off-by: Julius Volz <[email protected]>

* [cryptofuzz] wolfCrypt: Build with support for AES-CFB, AES-OFB (google#3904)

* libzmq: enable honggfuzz (google#3909)

* Update Prometheus maintainer in Dockerfile (google#3906)

...to myself (Prometheus co-creator).

Signed-off-by: Julius Volz <[email protected]>

* Revert "[systemd] remove my non-gmail address (google#3606)" (google#3910)

This reverts commit 65dcabc.

* [GDAL] Move clone of dependencies to upstream repository (google#3911)

Cloning of build dependencies and patching is now done by
https://github.com/OSGeo/gdal/blob/master/gdal/fuzzers/build.sh

* bump libtorrent to fuzz the 2.0 release branch (google#3913)

* [FFmpeg] install rsync as it is required for the seed corpus (google#3912)

Fixes: Issue 22500 (build failure)

* Fix builds_status cron exception. (google#3914)

When build log is not found, skip it and avoid 404.
Fixes exception
```
Traceback (most recent call last):
  File "oss-fuzz/infra/gcb/builds_status.py", line 243, in <module>
    main()
  File "oss-fuzz/infra/gcb/builds_status.py", line 228, in main
    status_filename='status.json')
  File "oss-fuzz/infra/gcb/builds_status.py", line 159, in update_build_status
    last_build = find_last_build(builds, project, build_tag_suffix)
  File "oss-fuzz/infra/gcb/builds_status.py", line 102, in find_last_build
    log.download_to_filename(f.name)
  File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 565, in download_to_filename
    file_obj, client=client, start=start, end=end)
  File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 537, in download_to_file
    _raise_from_invalid_response(exc)
  File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 1873, in _raise_from_invalid_response
    raise exceptions.from_http_response(error.response)
google.api_core.exceptions.NotFound: 404 GET https://www.googleapis.com/download/storage/v1/b/oss-fuzz-gcb-logs/o/log-9fca5dab-72bf-4970-9557-86b93ede51e0.txt?alt=media: No such object: oss-fuzz-gcb-logs/log-9fca5dab-72bf-4970-9557-86b93ede51e0.txt
Build step 'Execute shell' marked build as failure
Finished: FAILURE
```

* Retry building project images in build_specified_commit. (google#3915)

To mitigate transient network issues.

* Prometheus: Don't use WORKDIR $SRC (google#3908)

This makes build_fuzzers work with local checkouts.

Without this, it complains with 'Cannot use local checkout with "WORKDIR
/src".'

Signed-off-by: Julien Pivotto <[email protected]>

* [systemd] add DaanDeMeyer's address (google#3916)

* [muparser] initial integration (google#3814)

* [haproxy] Fix build failures. (google#3884)

* Updated the haproxy fuzzers to build again.

* The frame decoder needs additional updates since we need to call init_h2 to initialise a memory pool. Disabling this for now as this is a larger change in the code base and will fix up in the coming week.

* Fixed hpack decode.

* Updated the yaml since we dont want memory sanitizer.

* Update project.yaml (google#3917)

* install rsync (google#3920)

* [LibRaw] Initial integration (google#3918)

* Init integration

* Fix formatting

* Add size check

* Update maintainers

* PR Feedback

* Add newline

* Disable UBSan vptr

* Disable building examples

* Remove disabling UBSan vptr

Co-authored-by: Jamie Pinheiro <[email protected]>

* ecc-diff-fuzzer buils nettle without openssl (google#3922)

* Update Dockerfile (google#3923)

* Added contact to zlib-ng project auto_ccs. (google#3924)

* [qt] Add pdf to image corpus (google#3926)

* [haproxy] Minor fix (google#3928)

* Revert "[qt] Add pdf to image corpus (google#3926)" (google#3929)

To handle PDFs, QImage requires qtwebengine
which doesn't build statically.

This reverts commit 934c770.

* Fix TensorFlow build (google#3930)

* Remove [email protected] as he left Google

* Use python3 as python2 is deprecated

* Need to manually install numpy now as it is used in the toolchain

* Use bazelisk instead of the grep configure -> get Bazel version -> curl hack

* Remove C++11 constraint as TF now builds and uses C++14

* Handle review

* Remove source sed/replace as it is no longer needed

Co-authored-by: Guido Vranken <[email protected]>
Co-authored-by: AdamKorcz <[email protected]>
Co-authored-by: Daniel Salzman <[email protected]>
Co-authored-by: Abhishek Arya <[email protected]>
Co-authored-by: Oliver Chang <[email protected]>
Co-authored-by: Bhargava Shastry <[email protected]>
Co-authored-by: Evgeny Vereshchagin <[email protected]>
Co-authored-by: Nick Fitzgerald <[email protected]>
Co-authored-by: Luca Boccassi <[email protected]>
Co-authored-by: DavidKorczynski <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Gabriel Kihlman <[email protected]>
Co-authored-by: neildhar <[email protected]>
Co-authored-by: tsdgeos <[email protected]>
Co-authored-by: Catena cyber <[email protected]>
Co-authored-by: Robert Löhning <[email protected]>
Co-authored-by: Kevin Lubick <[email protected]>
Co-authored-by: Randy <[email protected]>
Co-authored-by: Will <[email protected]>
Co-authored-by: asraa <[email protected]>
Co-authored-by: Christian Holler (:decoder) <[email protected]>
Co-authored-by: Petteri Aimonen <[email protected]>
Co-authored-by: jonathanmetzman <[email protected]>
Co-authored-by: Jon Siwek <[email protected]>
Co-authored-by: Google AutoFuzz Team <[email protected]>
Co-authored-by: kabeer27 <[email protected]>
Co-authored-by: Nikita Popov <[email protected]>
Co-authored-by: Julius Volz <[email protected]>
Co-authored-by: Even Rouault <[email protected]>
Co-authored-by: Arvid Norberg <[email protected]>
Co-authored-by: Michael Niedermayer <[email protected]>
Co-authored-by: Julien Pivotto <[email protected]>
Co-authored-by: Ingo Berg <[email protected]>
Co-authored-by: Teju Nareddy <[email protected]>
Co-authored-by: Jamie Pinheiro <[email protected]>
Co-authored-by: Jamie Pinheiro <[email protected]>
Co-authored-by: Nathan Moinvaziri <[email protected]>
Co-authored-by: Mihai Maruseac <[email protected]>
  • Loading branch information
Show file tree
Hide file tree
Showing 104 changed files with 1,188 additions and 482 deletions.
4 changes: 1 addition & 3 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ install:
matrix:
include:
- name: "presubmit"
install:
- pip install -r infra/dev-requirements.txt
script: ./infra/presubmit.py
- name: "libfuzzer address x86_64"
env:
Expand Down Expand Up @@ -58,7 +56,7 @@ matrix:
- TRAVIS_SANITIZER=dataflow
- TRAVIS_ARCHITECTURE=x86_64
- name: "infra-tests"
script: sudo ./infra/presubmit.py infra-tests
script: sudo /bin/bash -c 'source $HOME/virtualenv/python3.6/bin/activate && ./infra/presubmit.py infra-tests'

script: ./infra/travis/travis_build.py

Expand Down
10 changes: 4 additions & 6 deletions docs/getting-started/new_project_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ Before you can start setting up your new project for fuzzing, you must do the fo
[docker-cleanup](https://gist.github.com/mikea/d23a839cba68778d94e0302e8a2c200f)
periodically to garbage-collect unused images.

- (optional) [Install gsutil](https://cloud.google.com/storage/docs/gsutil_install) for local code coverage sanity check.
For Google internal (gLinux) machines, please refer [here](https://cloud.google.com/storage/docs/gsutil_install#deb) instead.

## Creating the file structure

Each OSS-Fuzz project has a subdirectory
Expand Down Expand Up @@ -306,12 +309,7 @@ You can build your docker image and fuzz targets locally, so you can test them b
```

4. We recommend taking a look at your code coverage as a sanity check to make sure that your
fuzz targets get to the code you expect:

```bash
$ python infra/helper.py build_fuzzers --sanitizer coverage $PROJECT_NAME
$ python infra/helper.py coverage $PROJECT_NAME <fuzz_target>
```
fuzz targets get to the code you expect. Please refer to [code coverage]({{ site.baseurl }}/advanced-topics/code-coverage/).

**Note:** Currently, we only support AddressSanitizer (address) and UndefinedBehaviorSanitizer (undefined)
configurations. MemorySanitizer is recommended, but needs to be enabled manually once you verify
Expand Down
53 changes: 45 additions & 8 deletions infra/base-images/base-builder/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,44 @@
################################################################################

FROM gcr.io/oss-fuzz-base/base-clang
RUN apt-get install -y git \
subversion \
jq \
python3 \
zip \
make \
binutils-dev \
libc6-dev-i386

RUN apt-get update && \
apt-get install -y software-properties-common && \
add-apt-repository ppa:git-core/ppa && \
apt-get update && \
apt-get install -y \
binutils-dev \
build-essential \
curl \
git \
jq \
libc6-dev-i386 \
subversion \
zip

# Build and install latest Python 3 (3.8.3).
ENV PYTHON_VERSION 3.8.3
RUN export PYTHON_DEPS="\
zlib1g-dev \
libncurses5-dev \
libgdbm-dev \
libnss3-dev \
libssl-dev \
libsqlite3-dev \
libreadline-dev \
libffi-dev \
libbz2-dev \
liblzma-dev" && \
apt-get install -y $PYTHON_DEPS && \
cd /tmp/ && \
curl -O https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tar.xz && \
tar -xvf Python-$PYTHON_VERSION.tar.xz && \
cd Python-$PYTHON_VERSION && \
./configure --enable-optimizations && \
make -j install && \
cd .. && \
rm -r /tmp/Python-$PYTHON_VERSION.tar.xz /tmp/Python-$PYTHON_VERSION && \
apt-get remove -y $PYTHON_DEPS # https://github.com/google/oss-fuzz/issues/3888

# Download and install the latest stable Go.
ADD https://storage.googleapis.com/golang/getgo/installer_linux $SRC/
Expand All @@ -41,6 +71,13 @@ ENV PATH $PATH:/root/.go/bin:$GOPATH/bin
RUN go get -u github.com/mdempsky/go114-fuzz-build && \
ln -s $GOPATH/bin/go114-fuzz-build $GOPATH/bin/go-fuzz

# Install Rust and cargo-fuzz for libFuzzer instrumentation.
ENV CARGO_HOME=/rust
ENV RUSTUP_HOME=/rust/rustup
ENV PATH=$PATH:/rust/bin
RUN curl https://sh.rustup.rs | sh -s -- -y --default-toolchain=nightly
RUN cargo install cargo-fuzz

# Default build flags for various sanitizers.
ENV SANITIZER_FLAGS_address "-fsanitize=address -fsanitize-address-use-after-scope"

Expand Down
17 changes: 16 additions & 1 deletion infra/base-images/base-builder/compile
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,29 @@ if [ $FUZZING_ENGINE = "none" ]; then
export COVERAGE_FLAGS=
fi

# Rust does not support sanitizers and coverage flags via CFLAGS/CXXFLAGS, so
# use RUSTFLAGS.
# FIXME: Support code coverage once support is in.
# See https://github.com/rust-lang/rust/issues/34701.
if [ "$SANITIZER" != "undefined" ] && [ "$SANITIZER" != "coverage" ] && [ "$ARCHITECTURE" != 'i386' ]; then
export RUSTFLAGS="--cfg fuzzing -Zsanitizer=${SANITIZER} -Cdebuginfo=1 -Cforce-frame-pointers"
else
export RUSTFLAGS="--cfg fuzzing -Cdebuginfo=1 -Cforce-frame-pointers"
fi

# Add Rust libfuzzer flags.
# See https://github.com/rust-fuzz/libfuzzer/blob/master/build.rs#L12.
export CUSTOM_LIBFUZZER_PATH="$LIB_FUZZING_ENGINE_DEPRECATED"
export CUSTOM_LIBFUZZER_STD_CXX=c++

export CFLAGS="$CFLAGS $SANITIZER_FLAGS $COVERAGE_FLAGS"
export CXXFLAGS="$CFLAGS $CXXFLAGS_EXTRA"

echo "---------------------------------------------------------------"
echo "CC=$CC"
echo "CXX=$CXX"
echo "CFLAGS=$CFLAGS"
echo "CXXFLAGS=$CXXFLAGS"

echo "---------------------------------------------------------------"

BUILD_CMD="bash -eux $SRC/build.sh"
Expand Down
1 change: 1 addition & 0 deletions infra/base-images/base-runner/coverage
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ function run_fuzz_target {
local profraw_file_mask="$DUMPS_DIR/$target.*.profraw"
local profdata_file="$DUMPS_DIR/$target.profdata"
local corpus_real="/corpus/${target}"
mkdir -p $corpus_real

# -merge=1 requires an output directory, create a dummy dir for that.
local corpus_dummy="$OUT/dummy_corpus_dir_for_${target}"
Expand Down
2 changes: 1 addition & 1 deletion infra/base-images/base-runner/download_corpus
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ fi

for pair in "$@"; do
read path url <<< "$pair"
wget -q -O $path $url
wget -q -O $path $url || rm $path
done

# Always exit with 0 as we do not track wget return codes and should not rely
Expand Down
82 changes: 55 additions & 27 deletions infra/build_specified_commit.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
import collections
import logging
import re
import shutil
import time

import helper
import repo_manager
Expand All @@ -31,6 +33,8 @@
'BuildData', ['project_name', 'engine', 'sanitizer', 'architecture'])

_GIT_DIR_MARKER = 'gitdir: '
_IMAGE_BUILD_TRIES = 3
_IMAGE_BUILD_RETRY_SLEEP = 30.0


class BaseBuilderRepo:
Expand All @@ -54,6 +58,34 @@ def find_digest(self, timestamp):
raise ValueError('Failed to find suitable base-builder.')


def _replace_gitdir(src_dir, file_path):
"""Replace gitdir with a relative path."""
with open(file_path) as handle:
lines = handle.readlines()

new_lines = []
for line in lines:
if line.startswith(_GIT_DIR_MARKER):
absolute_path = line[len(_GIT_DIR_MARKER):].strip()
if not os.path.isabs(absolute_path):
# Already relative.
return

current_dir = os.path.dirname(file_path)
# Rebase to /src rather than the host src dir.
base_dir = current_dir.replace(src_dir, '/src')
relative_path = os.path.relpath(absolute_path, base_dir)
logging.info('Replacing absolute submodule gitdir from %s to %s',
absolute_path, relative_path)

line = _GIT_DIR_MARKER + relative_path

new_lines.append(line)

with open(file_path, 'w') as handle:
handle.write(''.join(new_lines))


def _make_gitdirs_relative(src_dir):
"""Make gitdirs relative."""
for root_dir, _, files in os.walk(src_dir):
Expand All @@ -62,26 +94,7 @@ def _make_gitdirs_relative(src_dir):
continue

file_path = os.path.join(root_dir, filename)
with open(file_path) as handle:
lines = handle.readlines()

new_lines = []
for line in lines:
if line.startswith(_GIT_DIR_MARKER):
absolute_path = line[len(_GIT_DIR_MARKER):].strip()
current_dir = os.path.dirname(file_path)
# Rebase to /src rather than the host src dir.
base_dir = current_dir.replace(src_dir, '/src')
relative_path = os.path.relpath(absolute_path, base_dir)
logging.info('Replacing absolute submodule gitdir from %s to %s',
absolute_path, relative_path)

line = _GIT_DIR_MARKER + relative_path

new_lines.append(line)

with open(file_path, 'w') as handle:
handle.write(''.join(new_lines))
_replace_gitdir(src_dir, file_path)


def _replace_base_builder_digest(dockerfile_path, digest):
Expand All @@ -104,26 +117,41 @@ def copy_src_from_docker(project_name, host_dir):
"""Copy /src from docker to the host."""
# Copy /src to host.
image_name = 'gcr.io/oss-fuzz/' + project_name
src_dir = os.path.join(host_dir, 'src')
if os.path.exists(src_dir):
shutil.rmtree(src_dir, ignore_errors=True)

docker_args = [
'-v',
host_dir + ':/out',
image_name,
'rsync',
'-aW',
'--delete',
'cp',
'-r',
'-p',
'/src',
'/out',
]
helper.docker_run(docker_args)

# Submodules can have gitdir entries which point to absolute paths. Make them
# relative, as otherwise we can't do operations on the checkout on the host.
src_dir = os.path.join(host_dir, 'src')
_make_gitdirs_relative(src_dir)

return src_dir


def _build_image_with_retries(project_name):
"""Build image with retries."""

for _ in range(_IMAGE_BUILD_TRIES):
result = helper.build_image_impl(project_name)
if result:
return result

time.sleep(_IMAGE_BUILD_RETRY_SLEEP)

return result


def build_fuzzers_from_commit(commit,
build_repo_manager,
host_src_path,
Expand Down Expand Up @@ -192,7 +220,7 @@ def cleanup():
base_builder_digest)

# Rebuild image and re-copy src dir since things in /src could have changed.
if not helper.build_image_impl(build_data.project_name):
if not _build_image_with_retries(build_data.project_name):
raise RuntimeError('Failed to rebuild image.')

cleanup()
Expand Down Expand Up @@ -226,7 +254,7 @@ def detect_main_repo(project_name, repo_name=None, commit=None):

# Change to oss-fuzz main directory so helper.py runs correctly.
utils.chdir_to_root()
if not helper.build_image_impl(project_name):
if not _build_image_with_retries(project_name):
logging.error('Error: building %s image failed.', project_name)
return None, None
docker_image_name = 'gcr.io/oss-fuzz/' + project_name
Expand Down
21 changes: 14 additions & 7 deletions infra/cifuzz/actions/build_fuzzers/build_fuzzers_entrypoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Builds and runs specific OSS-Fuzz project's fuzzers for CI tools."""
import json
import logging
import os
import sys
Expand Down Expand Up @@ -40,16 +41,17 @@ def main():
OSS_FUZZ_PROJECT_NAME: The name of OSS-Fuzz project.
GITHUB_REPOSITORY: The name of the Github repo that called this script.
GITHUB_SHA: The commit SHA that triggered this script.
GITHUB_REF: The pull request reference that triggered this script.
GITHUB_EVENT_NAME: The name of the hook event that triggered this script.
GITHUB_EVENT_PATH:
The path to the file containing the POST payload of the webhook:
https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#filesystems-on-github-hosted-runners
GITHUB_WORKSPACE: The shared volume directory where input artifacts are.
Returns:
0 on success or 1 on Failure.
"""
oss_fuzz_project_name = os.environ.get('OSS_FUZZ_PROJECT_NAME')
github_repo_name = os.path.basename(os.environ.get('GITHUB_REPOSITORY'))
pr_ref = os.environ.get('GITHUB_REF')
commit_sha = os.environ.get('GITHUB_SHA')
event = os.environ.get('GITHUB_EVENT_NAME')
workspace = os.environ.get('GITHUB_WORKSPACE')
Expand All @@ -73,11 +75,16 @@ def main():
logging.error('Error building fuzzers for project %s with commit %s.',
oss_fuzz_project_name, commit_sha)
return returncode
if event == 'pull_request' and not cifuzz.build_fuzzers(
oss_fuzz_project_name, github_repo_name, workspace, pr_ref=pr_ref):
logging.error('Error building fuzzers for project %s with pull request %s.',
oss_fuzz_project_name, pr_ref)
return returncode
if event == 'pull_request':
with open(os.environ.get('GITHUB_EVENT_PATH'), encoding='utf-8') as file:
event = json.load(file)
pr_ref = 'refs/pull/{0}/merge'.format(event['pull_request']['number'])
if not cifuzz.build_fuzzers(
oss_fuzz_project_name, github_repo_name, workspace, pr_ref=pr_ref):
logging.error(
'Error building fuzzers for project %s with pull request %s.',
oss_fuzz_project_name, pr_ref)
return returncode
out_dir = os.path.join(workspace, 'out')
if cifuzz.check_fuzzer_build(out_dir):
return 0
Expand Down
24 changes: 18 additions & 6 deletions infra/cifuzz/cifuzz.py
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,16 @@ def build_fuzzers(project_name,

# Build Fuzzers using docker run.
command = [
'--cap-add', 'SYS_PTRACE', '-e', 'FUZZING_ENGINE=' + DEFAULT_ENGINE, '-e',
'SANITIZER=' + DEFAULT_SANITIZER, '-e',
'ARCHITECTURE=' + DEFAULT_ARCHITECTURE
'--cap-add',
'SYS_PTRACE',
'-e',
'FUZZING_ENGINE=' + DEFAULT_ENGINE,
'-e',
'SANITIZER=' + DEFAULT_SANITIZER,
'-e',
'ARCHITECTURE=' + DEFAULT_ARCHITECTURE,
'-e',
'FUZZING_LANGUAGE=c++', # FIXME: Add proper support.
]
container = utils.get_container_name()
if container:
Expand Down Expand Up @@ -244,9 +251,14 @@ def check_fuzzer_build(out_dir):
return False

command = [
'--cap-add', 'SYS_PTRACE', '-e', 'FUZZING_ENGINE=' + DEFAULT_ENGINE, '-e',
'SANITIZER=' + DEFAULT_SANITIZER, '-e',
'ARCHITECTURE=' + DEFAULT_ARCHITECTURE
'--cap-add',
'SYS_PTRACE',
'-e',
'FUZZING_ENGINE=' + DEFAULT_ENGINE,
'-e',
'SANITIZER=' + DEFAULT_SANITIZER,
'-e',
'ARCHITECTURE=' + DEFAULT_ARCHITECTURE,
]

# Set ALLOWED_BROKEN_TARGETS_PERCENTAGE in docker if specified by user.
Expand Down
Loading

0 comments on commit da66a25

Please sign in to comment.