[Envoy] Add project. #1130

htuch · 2018-02-01T17:48:19Z

Following the steps at
https://github.com/google/oss-fuzz/blob/master/docs/new_project_guide.md.

Signed-off-by: Harvey Tuch [email protected]

Following the steps at https://github.com/google/oss-fuzz/blob/master/docs/new_project_guide.md. Signed-off-by: Harvey Tuch <[email protected]>

inferno-chromium · 2018-02-01T17:57:08Z

@jonathanmetzman - can you please help to review.

jonathanmetzman

I have a few nits about Dockerfile, but more importantly something weird is going on with coverage when using ASan.
If I build a coverage build of the fuzzer I see coverage reported as 10148 but when I do an ASan build I see it reported as 34.

I don't know what is causing this but I will need to look into it further. It's hard to tell how the fuzzer's performance is affected by this since the target appears pretty simple.

jonathanmetzman · 2018-02-01T18:28:16Z

projects/envoy/Dockerfile

+RUN apt-get update && apt-get install -y software-properties-common python-software-properties
+RUN add-apt-repository ppa:webupd8team/java
+RUN apt-get update && apt-get -y install  \
+	vim             \


Do you actually need vim?

jonathanmetzman · 2018-02-01T18:40:59Z

projects/envoy/Dockerfile

+        curl            \
+        autoconf        \
+        libtool
+RUN apt-get -y install cmake wget golang


Nit: can you add these to the apt-get -y install command started on line 22 (like curl, autoconf , libtool etc.)?

jonathanmetzman · 2018-02-01T18:59:51Z

projects/envoy/Dockerfile

+MAINTAINER [email protected]
+
+RUN apt-get update && apt-get install -y software-properties-common python-software-properties
+RUN add-apt-repository ppa:webupd8team/java


I don't think this line is needed. openjdk-8-jdk isn't downloaded from there.
I think line 20 is only needed for line 21 so please remove that as well if you remove this line.

htuch · 2018-02-01T20:22:59Z

Yep, the Docker stuff is largely because we started with a copy+paste of projects/grpc. Will fix.

On coverage, this is similar to my local experience, I was a bit confused about how I'm supposed to validate locally with pure ASAN and without LLVM coverage generation; see #1126 for the thread. Any assistance here appreciated, it's a bit mysterious to me still how we get coverage without instrumentation.

kcc · 2018-02-01T21:37:11Z

I've built the asan version and it looks fine.
The coverage is indeed tiny, but that's totally expected from a target that fuzzes base64.
I assume this is just a test target and you will have more. :)

Any assistance here appreciated,

I am bit lost. Which part doesn't work after the fix for #1126 ?

kcc · 2018-02-01T21:51:33Z

Interesting.
Running

./infra/helper.py build_fuzzers envoy

works and produces good asan binary.

But then running

./infra/helper.py build_fuzzers --sanitizer coverage envoy

fails:

external/envoy_deps/thirdparty_build/lib/libcrypto.a(bcm.c.o):bcm.c:function RAND_bytes_with_additional_data: error: undefined reference to '__asan_stack_malloc_4'

could this be caused by #1125 (edited) ?

kcc · 2018-02-01T22:00:34Z

Yes, undoing #1125 helps.
I don't know enough about bazel, but maybe doing bazel clean at the top of build.sh would be sufficient

The coverage build also looks sane. Yes, it shows crazy INITED cov: 10124 (this is most likely caused different coverage instrumentations used. Should be safe to ignore).

kcc · 2018-02-01T22:05:43Z

Running ./infra/helper.py coverage envoy base64_fuzz_test
leads to

Navigate to see coverage: http://127.0.0.1:8001/

and a working top-level coverage page, but then

192.168.9.1 - - [01/Feb/2018 22:02:56] "GET /proc/self/cwd/source/common/common/base64.cc HTTP/1.1" 404 -

Looks like the local coverage collection is busted (apparently, noone have been using it for a while).

Anyway, this PR currently looks good to me (modulo comments about docker above).
Looking forward to seeing real fuzz targets (not just base64).

kcc · 2018-02-01T22:23:54Z

The problem with coverage may also be caused by the pesky "/proc/self/cwd/" which is created by bazel and thus be specific to this PR (and also affect grpc, but not others)

Signed-off-by: Harvey Tuch <[email protected]>

htuch · 2018-02-02T13:59:23Z

OK, great. Thanks for the review feedback @jonathanmetzman and @kcc. I've addressed the Dockerfile nits. Yes, we plan on doing more than base64 encoder fuzzing ;) The idea was to get a simple example working end-to-end and then building more complex control and data plane fuzzer binaries once the infrastructure was in place.

inferno-chromium · 2018-02-02T19:08:48Z

Build is failing
https://oss-fuzz-build-logs.storage.googleapis.com/index.html
https://oss-fuzz-build-logs.storage.googleapis.com/log-a2a5bec8-1d76-47b4-8972-bb35e582ded2.txt

@htuch - can you please take a look.

kcc · 2018-02-02T20:46:05Z

I suspect this is related to my comment #1130 (comment) above:
We either need to revert #1125 (helps) or add bazel clean to the build.sh (unverified guess)

This reverts commit d0fc020.

htuch · 2018-02-02T20:50:13Z

#1133; this was just a convenience for local use, let's drop if broken on build machines.

* Revert "[Envoy] Add project. (#1130)" This reverts commit d0fc020. * Revert "Fix Skia compile (#1132)" This reverts commit 4bf9e7f. * Revert "Propose graphics magick for inclusion in OSS-Fuzz (#1131)" This reverts commit cb277cc. * Revert "[json-c] Add project (#1123)" This reverts commit 31b0046. * Revert "[infra] Update upload URL timeout to be the same as build timeout (#1112)" This reverts commit 9215296. * Revert "infra/helper: persist /root directory via bind mount. (#1125)" This reverts commit b77745a.

kcc · 2018-02-03T03:20:53Z

ubsan build fails (--sanitizer undefined):

clang-6.0: error: invalid argument '-fsanitize=vptr' not allowed with '-fno-rtti'

This is a common problem. We probably need to fix it eventually in the clang driver.
for now please add -fno-sanitize=vptr as some other projects do.

inferno-chromium · 2018-02-03T17:55:30Z

@htuch - build is still failing, please take a look on Monday. we need to get a first working build, otherwise it raises a ton of exceptions on our infrastructure.

htuch · 2018-02-04T04:44:18Z

@inferno-chromium is it failing on ASAN or UBSAN? I noticed that you've set the project.yaml to only ASAN now.

I built with python infra/helper.py build_fuzzers --sanitizer=undefined envoy after your recent changes to build.sh to fix CFLAGS/CXXFLAGS, it seems to work fine and I can run the fuzzer locally.

Beyond what I can see locally, are there any build debug artifacts you can point me at?

inferno-chromium · 2018-02-04T06:09:14Z

@htuch - i looked at wrong log incorrectly, this seemed to be failure during coverage build (see https://oss-fuzz-build-logs.storage.googleapis.com/index.html), so i have reenabled ubsan now. Also, right now i can build coverage locally fine, so triggered another envoy build on builder, lets see if it succeeds.

htuch · 2018-02-05T20:47:38Z

This looks like it's still failing. I think I understand the underlying issue. Our external dependencies are built outside of Bazel, and pick up on CFLAGS/CXXFLAGS. Our top-level build driver isn't smart enough right now to force a rebuild when they change. So, we do an ASAN build first, that's fine, then when we do UBSAN, we attempt to link a bunch of external deps built with the wrong flags. I will fix this in Envoy, it's a small change. Should be merged by later today or tomorrow. CC: @jmillikin-stripe

inferno-chromium · 2018-02-05T21:20:09Z

@htuch - should we do bazel clean in build.sh ?

htuch · 2018-02-05T21:23:54Z

As a temporary workaround, "bazel clean --expunge" would work, I think we should fix this properly in Envoy though.

…

On Mon, Feb 5, 2018 at 4:20 PM Abhishek Arya ***@***.***> wrote: @htuch <https://github.com/htuch> - should we do bazel clean in build.sh ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1130 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKaLv6tb-mAXXorAXjBFVF6bZmlPhzYAks5tR3COgaJpZM4R2BXQ> .

inferno-chromium · 2018-02-05T21:28:01Z

@htuch - ok upto you, we are fine to wait another day till this is fixed properly.

This was a longstanding problem with external deps that manifested in google/oss-fuzz#1130. Signed-off-by: Harvey Tuch <[email protected]>

htuch · 2018-02-06T18:25:58Z

@inferno-chromium can we kick off another run, now that envoyproxy/envoy#2534 has merged?

inferno-chromium · 2018-02-06T19:30:02Z

@htuch - done and got first green build, yippee, build page should update in < 30 min, but i checked the results on builder.

* [Envoy] Add project. Following the steps at https://github.com/google/oss-fuzz/blob/master/docs/new_project_guide.md. Signed-off-by: Harvey Tuch <[email protected]> * Dockerfile review feedback. Signed-off-by: Harvey Tuch <[email protected]>

…e#1133) * Revert "[Envoy] Add project. (google#1130)" This reverts commit d0fc020. * Revert "Fix Skia compile (google#1132)" This reverts commit 4bf9e7f. * Revert "Propose graphics magick for inclusion in OSS-Fuzz (google#1131)" This reverts commit cb277cc. * Revert "[json-c] Add project (google#1123)" This reverts commit 31b0046. * Revert "[infra] Update upload URL timeout to be the same as build timeout (google#1112)" This reverts commit 9215296. * Revert "infra/helper: persist /root directory via bind mount. (google#1125)" This reverts commit b77745a.

[Envoy] Add project.

ddc27fd

Following the steps at https://github.com/google/oss-fuzz/blob/master/docs/new_project_guide.md. Signed-off-by: Harvey Tuch <[email protected]>

inferno-chromium requested a review from jonathanmetzman February 1, 2018 17:56

jonathanmetzman reviewed Feb 1, 2018

View reviewed changes

htuch added 2 commits February 2, 2018 08:55

Dockerfile review feedback.

8c10223

Signed-off-by: Harvey Tuch <[email protected]>

Merge remote-tracking branch 'upstream/master' into envoy

0824f40

inferno-chromium requested a review from kcc February 2, 2018 15:15

kcc merged commit d0fc020 into google:master Feb 2, 2018

htuch added a commit to htuch/oss-fuzz that referenced this pull request Feb 2, 2018

Revert "[Envoy] Add project. (google#1130)"

0eb15f7

This reverts commit d0fc020.

inferno-chromium added a commit that referenced this pull request Feb 3, 2018

Disable UBSan vptr (#1130)

2998db4

inferno-chromium added a commit that referenced this pull request Feb 3, 2018

Disable UBSan to get first build (#1130)

565cbbe

htuch deleted the envoy branch February 4, 2018 04:36

htuch added a commit to htuch/envoy that referenced this pull request Feb 5, 2018

build: retrigger external builds when CFLAGS/CXXFLAGS changes.

dc55da2

This was a longstanding problem with external deps that manifested in google/oss-fuzz#1130. Signed-off-by: Harvey Tuch <[email protected]>

htuch mentioned this pull request Feb 5, 2018

build: retrigger external builds when CFLAGS/CXXFLAGS changes. envoyproxy/envoy#2534

Merged

tmatth pushed a commit to tmatth/oss-fuzz that referenced this pull request Oct 22, 2018

Disable UBSan vptr (google#1130)

47f6b9f

tmatth pushed a commit to tmatth/oss-fuzz that referenced this pull request Oct 22, 2018

Disable UBSan to get first build (google#1130)

53d65d6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Envoy] Add project. #1130

[Envoy] Add project. #1130

htuch commented Feb 1, 2018

inferno-chromium commented Feb 1, 2018

jonathanmetzman left a comment •

edited

Loading

jonathanmetzman Feb 1, 2018

jonathanmetzman Feb 1, 2018

jonathanmetzman Feb 1, 2018

htuch commented Feb 1, 2018

kcc commented Feb 1, 2018

kcc commented Feb 1, 2018 •

edited

Loading

kcc commented Feb 1, 2018

kcc commented Feb 1, 2018

kcc commented Feb 1, 2018

htuch commented Feb 2, 2018

inferno-chromium commented Feb 2, 2018

kcc commented Feb 2, 2018

htuch commented Feb 2, 2018

kcc commented Feb 3, 2018

inferno-chromium commented Feb 3, 2018

htuch commented Feb 4, 2018

inferno-chromium commented Feb 4, 2018

htuch commented Feb 5, 2018

inferno-chromium commented Feb 5, 2018

htuch commented Feb 5, 2018 via email

inferno-chromium commented Feb 5, 2018

htuch commented Feb 6, 2018

inferno-chromium commented Feb 6, 2018

[Envoy] Add project. #1130

[Envoy] Add project. #1130

Conversation

htuch commented Feb 1, 2018

inferno-chromium commented Feb 1, 2018

jonathanmetzman left a comment • edited Loading

Choose a reason for hiding this comment

jonathanmetzman Feb 1, 2018

Choose a reason for hiding this comment

jonathanmetzman Feb 1, 2018

Choose a reason for hiding this comment

jonathanmetzman Feb 1, 2018

Choose a reason for hiding this comment

htuch commented Feb 1, 2018

kcc commented Feb 1, 2018

kcc commented Feb 1, 2018 • edited Loading

kcc commented Feb 1, 2018

kcc commented Feb 1, 2018

kcc commented Feb 1, 2018

htuch commented Feb 2, 2018

inferno-chromium commented Feb 2, 2018

kcc commented Feb 2, 2018

htuch commented Feb 2, 2018

kcc commented Feb 3, 2018

inferno-chromium commented Feb 3, 2018

htuch commented Feb 4, 2018

inferno-chromium commented Feb 4, 2018

htuch commented Feb 5, 2018

inferno-chromium commented Feb 5, 2018

htuch commented Feb 5, 2018 via email

inferno-chromium commented Feb 5, 2018

htuch commented Feb 6, 2018

inferno-chromium commented Feb 6, 2018

jonathanmetzman left a comment •

edited

Loading

kcc commented Feb 1, 2018 •

edited

Loading