Add db benchmark script #1928

matthewmturner · 2022-03-05T07:46:38Z

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

matthewmturner · 2022-03-05T08:10:22Z

meh, i think im close but im having some issues due to docker memory swapping which isnt allowed by some db-benchmark helpers. will come back to this.

matthewmturner · 2022-03-05T18:38:10Z

Lol. I unknowingly created a shell script that had a name conflict with a db-benchmark script which caused issue.

houqp · 2022-03-05T20:07:06Z

benchmarks/db-benchmark/db-benchmark.dockerfile

+RUN apt-get update && \
+    apt-get install -y git build-essential
+
+# Install R, curl, and python deps
+RUN apt-get update && apt-get -y install --no-install-recommends --no-install-suggests \


the apt-get update seems to be redundant? looks like we can merge these two apt runs to reduce build time.

Good catch. Thx.

houqp · 2022-03-05T20:08:01Z

benchmarks/db-benchmark/db-benchmark.dockerfile

+
+# Clone datafusion-python and build python library
+# Not sure if the wheel will be the same on all computers
+RUN git clone https://github.com/datafusion-contrib/datafusion-python \


would be good to clone a particular tag/commit to make this more reproducible.

matthewmturner · 2022-03-06T04:43:00Z

I have this working now.

Would at least one other person be able to give it a try. I do the following from root directory of arrow-datafusion

$ docker build -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .
# I used privileged as its supposed to improve with CPU intensive scripts
$ docker run --privileged db-benchmark

So far im not seeing as good of results in docker as i was natively but i think thats expected, even using --privileged. I havent had to performance tune docker too much in the past though so if anyone else has other ideas im open to it.

Right now, this only works with official datafusion release (version 7). However, I added option to pull in local datafusion so that its easier to benchmark local changes. Even if the docker performance isnt optimal it should at least give a baseline for making local changes and seeing if there are improvements - which i think is the intent of this script.

let me know if any questions or if you see any areas for improvement.

houqp · 2022-03-09T05:51:31Z

@matthewmturner you are on a mac machine right?

matthewmturner · 2022-03-09T05:52:23Z

@matthewmturner you are on a mac machine right?

Yes, M1 Mac

houqp · 2022-03-09T06:23:53Z

that's expected then. the overhead should go away with when it's executed on a linux box.

bobtins · 2022-03-11T22:56:49Z

Is this specific to M1 Mac? I am running Ubuntu 20.04 on AMD64 and I get this error on docker build:

⚠️  Warning: No compatible platform tag found, using the linux tag instead. You won't be able to upload those wheels to PyPI.
📦 Built wheel for abi3 Python ≥ 3.6 to /datafusion-python/target/wheels/datafusion-0.4.0-cp36-abi3-linux_x86_64.whl
WARNING: Requirement 'target/wheels/datafusion-0.4.0-cp36-abi3-linux_aarch64.whl' looks like a filename, but the file does not exist
ERROR: datafusion-0.4.0-cp36-abi3-linux_aarch64.whl is not a supported wheel on this platform.
The command '/bin/sh -c cd datafusion-python     && maturin build --release     && python3 -m pip install target/wheels/datafusion-0.4.0-cp36-abi3-linux_aarch64.whl     && cd ..' returned a non-zero code: 1

Will try s/aarch64/x86_64/...

matthewmturner · 2022-03-11T23:28:30Z

@bobtins thanks for trying it out! You'll just need to update the wheel file name in the docker file. Looks like the below is what you'll need.

/datafusion-python/target/wheels/datafusion-0.4.0-cp36-abi3-linux_x86_64.whl

bobtins · 2022-03-11T23:34:41Z

Yeah, that got it to work.
I found this for populating env variables automatically, but it requires BuildKit, which means I have to do export DOCKER_BUILDKIT=1 before running the build step.

matthewmturner · 2022-03-11T23:52:09Z

Nice find - I can look into integrating that

matthewmturner · 2022-03-15T13:00:44Z

@houqp If no other feedback I think this is good to merge for now.

@bobtins im a bit constrained on time right now to work on getting the architectures / wheels integrated properly into the script. given youve shown interest in the benchmarking area would you be willing to handle that as a follow on PR?

yjshen · 2022-03-15T16:09:33Z

/datafusion-python/target/wheels/datafusion-0.4.0-cp36-abi3-linux_x86_64.whl

Possible to adjust the docker file to make it more generic? Meet the same problem here.

After some search, Buildx might do the trick?

bobtins · 2022-03-15T20:24:24Z

Almost got it...will post a diff as soon as it works.

matthewmturner · 2022-03-15T20:32:58Z

@bobtins thanks so much!

bobtins · 2022-03-15T20:48:11Z

benchmarks/db-benchmark/README.md

+Run the following from root `arrow-datafusion` directory
+
+```bash
+$ docker build -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .


Suggested change

$ docker build -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .

$ docker buildx build -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .

bobtins

Got it working with changes suggested.
Looking forward to playing around with this and maybe adding some more features.

bobtins · 2022-03-15T20:54:59Z

benchmarks/db-benchmark/db-benchmark.dockerfile

+# 1. datafusion-python that builds from datafusion version referenced datafusion-python
+RUN cd datafusion-python \
+    && maturin build --release \
+    && python3 -m pip install target/wheels/datafusion-0.4.0-cp36-abi3-linux_aarch64.whl \


Suggested change

&& python3 -m pip install target/wheels/datafusion-0.4.0-cp36-abi3-linux_aarch64.whl \

&& case "${TARGETPLATFORM}" in \

*/amd64) CPUARCH=x86_64 ;; \

*/arm64) CPUARCH=aarch64 ;; \

*) exit 1 ;; \

esac \

&& python3 -m pip install target/wheels/datafusion-0.4.0-cp36-abi3-linux_${CPUARCH}.whl \

&& case "${TARGETPLATFORM}" in \

*/amd64) CPUARCH=x86_64 ;; \

*/arm64) CPUARCH=aarch64 ;; \

*) exit 1 ;; \

esac \

&& python3 -m pip install target/wheels/datafusion-0.4.0-cp36-abi3-linux_${CPUARCH}.whl \

that's weird, didn't mean to put it in twice; haven't used this github feature before

OK, ignore those suggestions...I just did a diff and attached it.

diff.txt

matthewmturner · 2022-03-15T21:45:18Z

@bobtins thank you for the help! I am going to review and integrate tomorrow morning.

matthewmturner · 2022-03-16T14:03:55Z

@bobtins thank you again, for the help. I have updated.

@yjshen can you give it a try now?

yjshen · 2022-03-16T15:04:05Z

benchmarks/db-benchmark/README.md

+Run the following from root `arrow-datafusion` directory
+
+```bash
+$ docker buildx -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .


Suggested change

$ docker buildx -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .

$ docker buildx build -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .

Ah sry, did that local but forgot to add there

yjshen

LGTM. Thanks @matthewmturner !

matthewmturner added 2 commits February 23, 2022 18:40

Initial

4ddb279

Create Dockerfile

cb50c80

houqp reviewed Mar 5, 2022

View reviewed changes

matthewmturner added 3 commits March 5, 2022 22:38

Fix entry and script names

71cdbb6

Remove blank lines

6e51937

Add local datafusion option

876427e

Fix comment

eaec91b

matthewmturner marked this pull request as ready for review March 6, 2022 04:47

Add readme

ecf209b

matthewmturner added 2 commits March 14, 2022 10:10

Add license

d885ef2

More licenses

7c23302

bobtins reviewed Mar 15, 2022

View reviewed changes

bobtins suggested changes Mar 15, 2022

View reviewed changes

bobtins reviewed Mar 15, 2022

View reviewed changes

Arch agnostic

b6c304e

yjshen reviewed Mar 16, 2022

View reviewed changes

Fix readme

fde6f53

yjshen closed this Mar 16, 2022

yjshen reopened this Mar 16, 2022

yjshen approved these changes Mar 16, 2022

View reviewed changes

yjshen merged commit 8b249ae into apache:master Mar 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add db benchmark script #1928

Add db benchmark script #1928

matthewmturner commented Mar 5, 2022

matthewmturner commented Mar 5, 2022

matthewmturner commented Mar 5, 2022

houqp Mar 5, 2022

matthewmturner Mar 5, 2022

houqp Mar 5, 2022

matthewmturner commented Mar 6, 2022

houqp commented Mar 9, 2022

matthewmturner commented Mar 9, 2022

houqp commented Mar 9, 2022

bobtins commented Mar 11, 2022 •

edited

Loading

matthewmturner commented Mar 11, 2022

bobtins commented Mar 11, 2022

matthewmturner commented Mar 11, 2022

matthewmturner commented Mar 15, 2022

yjshen commented Mar 15, 2022 •

edited

Loading

bobtins commented Mar 15, 2022

matthewmturner commented Mar 15, 2022

bobtins Mar 15, 2022 •

edited

Loading

bobtins left a comment

bobtins Mar 15, 2022

bobtins Mar 15, 2022

bobtins Mar 15, 2022

matthewmturner commented Mar 15, 2022

matthewmturner commented Mar 16, 2022

yjshen Mar 16, 2022

matthewmturner Mar 16, 2022

yjshen left a comment

	$ docker build -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .
	$ docker buildx build -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .

-    && python3 -m pip install target/wheels/datafusion-0.4.0-cp36-abi3-linux_aarch64.whl \
+    && case "${TARGETPLATFORM}" in \
+         */amd64)  CPUARCH=x86_64  ;; \
+         */arm64)  CPUARCH=aarch64  ;; \
+         *) exit 1 ;; \
+    esac \
+    && python3 -m pip install target/wheels/datafusion-0.4.0-cp36-abi3-linux_${CPUARCH}.whl \
+    && case "${TARGETPLATFORM}" in \
+         */amd64)  CPUARCH=x86_64  ;; \
+         */arm64)  CPUARCH=aarch64  ;; \
+         *) exit 1 ;; \
+    esac \
+    && python3 -m pip install target/wheels/datafusion-0.4.0-cp36-abi3-linux_${CPUARCH}.whl \

	$ docker buildx -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .
	$ docker buildx build -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile .

Add db benchmark script #1928

Add db benchmark script #1928

Conversation

matthewmturner commented Mar 5, 2022

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

matthewmturner commented Mar 5, 2022

matthewmturner commented Mar 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewmturner commented Mar 6, 2022

houqp commented Mar 9, 2022

matthewmturner commented Mar 9, 2022

houqp commented Mar 9, 2022

bobtins commented Mar 11, 2022 • edited Loading

matthewmturner commented Mar 11, 2022

bobtins commented Mar 11, 2022

matthewmturner commented Mar 11, 2022

matthewmturner commented Mar 15, 2022

yjshen commented Mar 15, 2022 • edited Loading

bobtins commented Mar 15, 2022

matthewmturner commented Mar 15, 2022

bobtins Mar 15, 2022 • edited Loading

Choose a reason for hiding this comment

bobtins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewmturner commented Mar 15, 2022

matthewmturner commented Mar 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yjshen left a comment

Choose a reason for hiding this comment

bobtins commented Mar 11, 2022 •

edited

Loading

yjshen commented Mar 15, 2022 •

edited

Loading

bobtins Mar 15, 2022 •

edited

Loading