libzmq: add maintainer, add UBSAN #3829
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
looks like libFuzzer MSan and AFL builds are broken in travis, can you fix those or disable those configs in project.yaml |
These are working fine in oss-fuzz runs, when reproducing locally and in the repo's CI, any idea what could be the issue? |
|
For MSan build, |
I already tried as the page suggests, and even running the check_build option of infra/helper.py, and it all looks fine: $ infra/helper.py check_build libzmq --engine libfuzzer --sanitizer memory |
|
The AFL issue seems to be just a timeout that is too short. I do not see a way in the documentation to change it. |
Yes just disable that fuzzing engine, afl one we keep a low one for efficiency. |
|
Can easily reproduce the failure, probably you didn't pull latest images (with newer clang) ? |
|
I did pull - tried again, same result, works fine |
|
Tried purging everything in docker, still works just fine. I'll just ignore this, the important thing is that it works in oss-fuzz itself. I set the fuzzing_engines to libfuzzer in project.yaml, which I think should disable afl. |
Travis tests have failedHey @bluca, 4th Build./infra/travis/travis_build.pyTravisBuddy Request Identifier: 1709ab80-9929-11ea-860e-87b20eb63714 |
There is no sanity check and if the download fails because gnu.org is down the build fails with unhelpful errors
Looks like it's failing in oss-fuzz too now. Still can't reproduce, but I can get test builds on travis-ci.com now so I'll try a few things there |
|
MSan can be fixed in another CL, merging this. |
|
MSAN failures seem to be false positive in Clang, see: #3855 |
zchcai
added a commit
to zchcai/oss-fuzz
that referenced
this pull request
Jun 3, 2020
* [cryptofuzz] Add Microsoft SymCrypt (google#3826) * [gRPC-gateway] Initial integration (google#3807) * [doc] Update new_project_guide.md (google#3828) * [knot-dns] link against liblmdb statically (google#3797) Relates to https://gitlab.labs.nic.cz/knot/knot-dns/-/commit/d4ec3a3aa8b7e2ba4d196e2f7984173069e3d91b Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22191 * Simplify rust project setup. (google#3830) * Simplify rust project setup. - Add rust and cargo-fuzz in base builder. - Set RUSTC_BOOSTRAP to make ASan available. - Set RUSTFLAGS and C,CXXFLAGS properly. * Quick fix for helper.py build_fuzzers. (google#3832) * Parse project language and use in infra/helper.py (google#3834) * Parse project language and use in infra/helper.py * Fix exception message. * Add FUZZING_LANGUAGE to build step. (google#3835) * Add fuzzing language correctly during compile step. (google#3837) * Remove unneeded rust hacks after google#3830. (google#3840) * Fix missing FUZZING_LANGUAGE in coverage build. (google#3843) * Use hardcoded FUZZING_LANGUAGE to unbreak CIFuzz. (google#3844) * solidity: Disable Z3 build in dockerfile (google#3831) * cifuzz: use pull_request.number instead of GITHUB_REF (google#3845) Closes google#3732 * wasmtime: build fuzz targets with --all-features (google#3850) This enables not only the binaryen-using fuzz targets, but also the peepmatic fuzz targets (which is necessary after bytecodealliance/wasmtime#1727). * Relax project language requirement. (google#3846) This is necessary for the bisector to be able to build older revisions. Print a warning instead. * libzmq: add maintainer, add UBSAN (google#3829) * Add another maintainer to libzmq's CC list * Enable UBSAN for libzmq * libzmq: disable afl fuzzer, CI fails * libsodium: do not let libsodium's autogen.sh download files from gnu.org There is no sanity check and if the download fails because gnu.org is down the build fails with unhelpful errors * Fixed changes added in clang-11.0. (google#3852) * Bump httplib2 from 0.11.3 to 0.18.0 in /infra/gcb (google#3853) Bumps [httplib2](https://github.com/httplib2/httplib2) from 0.11.3 to 0.18.0. - [Release notes](https://github.com/httplib2/httplib2/releases) - [Changelog](https://github.com/httplib2/httplib2/blob/master/CHANGELOG) - [Commits](httplib2/httplib2@v0.11.3...v0.18.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [TiDB] initial integration (google#3849) * [Minify] Initial integration (google#3848) * Set rust flags always in compile step. (google#3859) This is needed for projects that mix both rust and c/c++. * Remove unneeded RUSTC_BOOTSTRAP since nightly is used * [fasthttp] Initial integration (google#3858) * [libfido2] bump libcbor version from 0.5.0 to 0.7.0 (google#3861) * Fix cras build failure, use default rust nightly. * Don't add Rust sanitizer flags when using ubsan or i386. (google#3862) Rust sanitizer flags are not supported for these configs. Fixes Cras and ecc-diff-fuzzer builds. * [hermes] Initial setup for Hermes (google#3698) * [poppler] Exercise a few more methods (google#3857) * [minify] Moved build script upstream (google#3867) * Use available nightly rust for spidermonkey build * Use available rust nightly in spidermonkey-ufi * Use available rust nightly in firefox * [python3-libraries] Fix build (google#3869) * Use latest git version. (google#3870) Adds features needed by firefox and spidermonkey-ufi builds. * libzmq: fix MSAN (google#3868) * libzmq: build libsodium with --disable-asm to avoid false positives in MSAN checks * libzmq: set autoconf install prefix to avoid /usr/local being added to the linker library paths A non-working version of the standard library is installed in oss-fuzz's images under /usr/local/lib. Using it breaks MSAN's tests with false positives. Set the prefix in autoconf to something different from the default /usr/local, as /lib is added by autoconf automatically to the linker flags, which means this broken standard library is used instead of the instrumented one in /usr/lib. * build_specified_commit: revert to using cp. (google#3871) rsync seems to have disappeared in the latest images somehow, and we can't rely on it always existing. * Don't add sanitizer=coverage in RUSTFLAGS * Remove libpcap patch merged upstream (google#3865) * Install Python 3.8.3 on builder image. (google#3874) Part of fixing google#3756. * [qt] Get a shallow clone of qtbase (google#3872) Reduces the size by nearly 200 MB * [qt] Add fuzzer for QImage::loadFromData() (google#3873) * Fix jsoncpp build * build_specified_commit: Don't replace gitdir if already relative. (google#3875) Upgraded git broke this. * [skia] Speculative build fix (google#3847) * [stb] extend code coverage for stb_image (google#3784) * [stb] extend code coverage for stb_image * remove wget's from build.sh * add wget's to Dockerfile * fix Dockerfile * [hermes] Fix coverage build (google#3877) * Update email address to non-alias for clusterfuzz login (google#3878) * add hosts and interns (google#3879) Signed-off-by: Asra Ali <[email protected]> * [qt] Don't install build-essential (google#3880) It's in base-builder now. * [hermes] Add CC for issues (google#3882) * [systemd] turn off hongfuzz (google#3889) See google#3887 * Remove libbz2-dev dependency installed in base-builder for python3 (google#3890) * Remove libbz2-dev dependency installed in base-builder for python3 Fixes google#3888 * Revert "[systemd] turn off hongfuzz (google#3889)" (google#3891) This reverts commit d638fac. * Set up python in one docker layer, remove python deps. (google#3893) * Set up python in one docker layer. Address review comment in google#3890 (review) * Remove all python dependencies, keep build-essential. * Makes suricata work with rust sanitizers (google#3897) * [firefox] Fix libFuzzer cloning process (google#3896) * Fix spidermonkey-ufi build. * [nanopb] Use $(which python3) instead of hardcoding /usr/bin/python3 (google#3895) After the addition of Python 3.8 in google#3874, the default python3 is now /usr/local/bin/python3. The nanopb Dockerfile hardcoded a path of /usr/bin/python3, causing again problems of Python packages being installed for different version than attempting to run with. * Fix CIFuzz issue where targets assumed in OSS-Fuzz build if exists (google#3817) Make some other changes: 1. Refactor fuzz_target.py and fuzz_target_test.py 2. Introduce pyfakefs and parameterized as dependencies and use them in tests. 3. Fix infra-tests in CI so that they use installed dependencies * Fix spidermonkey-ufi build * Extend Zeek project auto_ccs (google#3898) * [libgd] Add another fuzzer to libgd (google#3892) * [qt] Add qtdeclarative (google#3901) * [qt] Add qtdeclarative * [qt] Ignore stdout from make "The job exceeded the maximum log length, and has been terminated." * Updating jsoncpp primary contact (google#3900) * [php] Remove --enable-json (google#3899) JSON is now always compiled in, and cannot be explicitly enabled. * Don't create empty corpus archive as it wont unpack. (google#3903) * Update email addresses for Prometheus (google#3905) Signed-off-by: Julius Volz <[email protected]> * [cryptofuzz] wolfCrypt: Build with support for AES-CFB, AES-OFB (google#3904) * libzmq: enable honggfuzz (google#3909) * Update Prometheus maintainer in Dockerfile (google#3906) ...to myself (Prometheus co-creator). Signed-off-by: Julius Volz <[email protected]> * Revert "[systemd] remove my non-gmail address (google#3606)" (google#3910) This reverts commit 65dcabc. * [GDAL] Move clone of dependencies to upstream repository (google#3911) Cloning of build dependencies and patching is now done by https://github.com/OSGeo/gdal/blob/master/gdal/fuzzers/build.sh * bump libtorrent to fuzz the 2.0 release branch (google#3913) * [FFmpeg] install rsync as it is required for the seed corpus (google#3912) Fixes: Issue 22500 (build failure) * Fix builds_status cron exception. (google#3914) When build log is not found, skip it and avoid 404. Fixes exception ``` Traceback (most recent call last): File "oss-fuzz/infra/gcb/builds_status.py", line 243, in <module> main() File "oss-fuzz/infra/gcb/builds_status.py", line 228, in main status_filename='status.json') File "oss-fuzz/infra/gcb/builds_status.py", line 159, in update_build_status last_build = find_last_build(builds, project, build_tag_suffix) File "oss-fuzz/infra/gcb/builds_status.py", line 102, in find_last_build log.download_to_filename(f.name) File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 565, in download_to_filename file_obj, client=client, start=start, end=end) File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 537, in download_to_file _raise_from_invalid_response(exc) File "/var/jenkins_home/workspace/infra/builds_status/ENV/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 1873, in _raise_from_invalid_response raise exceptions.from_http_response(error.response) google.api_core.exceptions.NotFound: 404 GET https://www.googleapis.com/download/storage/v1/b/oss-fuzz-gcb-logs/o/log-9fca5dab-72bf-4970-9557-86b93ede51e0.txt?alt=media: No such object: oss-fuzz-gcb-logs/log-9fca5dab-72bf-4970-9557-86b93ede51e0.txt Build step 'Execute shell' marked build as failure Finished: FAILURE ``` * Retry building project images in build_specified_commit. (google#3915) To mitigate transient network issues. * Prometheus: Don't use WORKDIR $SRC (google#3908) This makes build_fuzzers work with local checkouts. Without this, it complains with 'Cannot use local checkout with "WORKDIR /src".' Signed-off-by: Julien Pivotto <[email protected]> * [systemd] add DaanDeMeyer's address (google#3916) * [muparser] initial integration (google#3814) * [haproxy] Fix build failures. (google#3884) * Updated the haproxy fuzzers to build again. * The frame decoder needs additional updates since we need to call init_h2 to initialise a memory pool. Disabling this for now as this is a larger change in the code base and will fix up in the coming week. * Fixed hpack decode. * Updated the yaml since we dont want memory sanitizer. * Update project.yaml (google#3917) * install rsync (google#3920) * [LibRaw] Initial integration (google#3918) * Init integration * Fix formatting * Add size check * Update maintainers * PR Feedback * Add newline * Disable UBSan vptr * Disable building examples * Remove disabling UBSan vptr Co-authored-by: Jamie Pinheiro <[email protected]> * ecc-diff-fuzzer buils nettle without openssl (google#3922) * Update Dockerfile (google#3923) * Added contact to zlib-ng project auto_ccs. (google#3924) * [qt] Add pdf to image corpus (google#3926) * [haproxy] Minor fix (google#3928) * Revert "[qt] Add pdf to image corpus (google#3926)" (google#3929) To handle PDFs, QImage requires qtwebengine which doesn't build statically. This reverts commit 934c770. * Fix TensorFlow build (google#3930) * Remove [email protected] as he left Google * Use python3 as python2 is deprecated * Need to manually install numpy now as it is used in the toolchain * Use bazelisk instead of the grep configure -> get Bazel version -> curl hack * Remove C++11 constraint as TF now builds and uses C++14 * Handle review * Remove source sed/replace as it is no longer needed Co-authored-by: Guido Vranken <[email protected]> Co-authored-by: AdamKorcz <[email protected]> Co-authored-by: Daniel Salzman <[email protected]> Co-authored-by: Abhishek Arya <[email protected]> Co-authored-by: Oliver Chang <[email protected]> Co-authored-by: Bhargava Shastry <[email protected]> Co-authored-by: Evgeny Vereshchagin <[email protected]> Co-authored-by: Nick Fitzgerald <[email protected]> Co-authored-by: Luca Boccassi <[email protected]> Co-authored-by: DavidKorczynski <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Gabriel Kihlman <[email protected]> Co-authored-by: neildhar <[email protected]> Co-authored-by: tsdgeos <[email protected]> Co-authored-by: Catena cyber <[email protected]> Co-authored-by: Robert Löhning <[email protected]> Co-authored-by: Kevin Lubick <[email protected]> Co-authored-by: Randy <[email protected]> Co-authored-by: Will <[email protected]> Co-authored-by: asraa <[email protected]> Co-authored-by: Christian Holler (:decoder) <[email protected]> Co-authored-by: Petteri Aimonen <[email protected]> Co-authored-by: jonathanmetzman <[email protected]> Co-authored-by: Jon Siwek <[email protected]> Co-authored-by: Google AutoFuzz Team <[email protected]> Co-authored-by: kabeer27 <[email protected]> Co-authored-by: Nikita Popov <[email protected]> Co-authored-by: Julius Volz <[email protected]> Co-authored-by: Even Rouault <[email protected]> Co-authored-by: Arvid Norberg <[email protected]> Co-authored-by: Michael Niedermayer <[email protected]> Co-authored-by: Julien Pivotto <[email protected]> Co-authored-by: Ingo Berg <[email protected]> Co-authored-by: Teju Nareddy <[email protected]> Co-authored-by: Jamie Pinheiro <[email protected]> Co-authored-by: Jamie Pinheiro <[email protected]> Co-authored-by: Nathan Moinvaziri <[email protected]> Co-authored-by: Mihai Maruseac <[email protected]>
|
@bluca - just fyi, libzmq build is failing https://oss-fuzz-build-logs.storage.googleapis.com/index.html#libzmq because when check_build runs the fuzz targets, they create lot of junk/temporary files in that dir and then zip operation to create build archive fails. This seems bad, fuzz targets should clean their temp files before exit or ideally just operate on data in memory, can you please fix that, that will fix the build archiving. |
|
@inferno-chromium strange, I'm not sure which files would be created, I'll have a look |
|
there are no files created by the tests, I'm really not sure what the ci is complaining about |
|
corpora for the tests is built and installed alongside the binaries, as the documentation suggests - could that be the problem? |
|
same for dictionaries |
|
@bluca - i looked more and found that this fuzzer is the issue ctrl+c after a 10-20 sec, |
|
Ah, I see: it's socket files. Thanks to the dict, the test is interpreting the input as AF_UNIX paths, so socket files are created. I'll change the test so that they go into an ephemeral directory. Thanks for pointing this out. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.