Skip to content

Commit dbc0397

Browse files
ch-scBrent Gardnerandygrovedependabot[bot]yahoNanJing
authored
Rebase to the latest Ballista version (#42)
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * feat: update script such that ballista-cli image is built as well (apache#601) * Fix Cargo.toml format issue (apache#616) * Refactor executor main (apache#614) * Refactor executor main * copy all configs * toml fmt * Refactor scheduler main (apache#615) * refactor scheduler main * toml fmt * Python: add method to get explain output as a string (apache#593) * Update contributor guide (apache#617) * Cluster state refactor part 1 (apache#560) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> * replace master with main (apache#621) * implement new release process (apache#623) * add docs on who can release (apache#632) * Upgrade to DataFusion 16 (again) (apache#636) * Update datafusion dependency to the latest version (apache#612) * Update datafusion dependency to the latest version * Fix python * Skip ut of test_window_lead due to apache/datafusion-python#135 * Fix clippy --------- Co-authored-by: yangzhong <[email protected]> * Upgrade to DataFusion 17 (apache#639) * Upgrade to DF 17 * Restore original error handling functionality * check in benchmark image (apache#647) * Remove `python` dir & python-related workflows (apache#654) * refactor: remove python dir & python-related workflows * remove brackets * Handle job resubmission (apache#586) * Handle job resubmission * Make resubmission configurable and add test * Fix debug log * Add executor self-registration mechanism in the heartbeat service (apache#649) Co-authored-by: yangzhong <[email protected]> * Cluster state refactor Part 2 (apache#658) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header * WIP * Implement JobState * Tests and fixes * do not hold ref across await point * Fix clippy warnings * Fix tomlfmt github action * uncomment test --------- Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> * Upgrade to DataFusion 18.0.0-rc1 (apache#664) * Minor refactor to reduce duplicate code (apache#659) * move test_util to ballista-examples package (apache#661) * Upgrade to DataFusion 18 (apache#668) * Enable physical plan round-trip tests (apache#666) * Prep 0.11 (apache#682) * Change version to 0.11.0 * changelog * update react-timeago version * yarn upgrade * fix * fix * revert yarn change * Print versions * Print locations * Avoid github shenanigans * Try to get runners running * Try to get runners running * already root --------- Co-authored-by: Andy Grove <[email protected]> * [minor] remove todo (apache#683) * Add executor terminating status for graceful shutdown (apache#667) * Add executor terminating status for graceful shutdown * Remove empty file * Update ballista/executor/src/executor_process.rs Co-authored-by: Brent Gardner <[email protected]> --------- Co-authored-by: Brent Gardner <[email protected]> * Allow `BallistaContext::read_*` methods to read multiple paths. (apache#679) * updated dependency in cargo, added read_json method, modified read_* methods to read multiple paths. * ran cargo fmt * Added revision for proper builds. * Update scheduler.md (apache#657) * Mark `SchedulerState` as pub (apache#688) * Mark as pub * Fmt --------- Co-authored-by: Daniël Heres <[email protected]> * Update graphviz-rust requirement from 0.5.0 to 0.6.1 (apache#651) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Upgrade DataFusion to 19.0.0 (apache#691) * update release notes (apache#692) * Make task launcher pub (apache#695) Co-authored-by: Daniël Heres <[email protected]> * Make task_manager pub (apache#698) Co-authored-by: Daniël Heres <[email protected]> * Add ExecutionEngine abstraction (apache#687) * Allow accessing s3 locations in client mode (apache#700) * Allow accessing s3 locations in client mode * Removed s3 feature from test dependencies. * fixed cargo-tomlfmt issues * deployment/docker-compose.md incorrect remote ref (apache#699) * Fix for error message during testing (apache#707) * Fix cargo clippy * Fix for error message during testing * Remove unwrap for dealing with JobQueued event * log task ids when launch tasks --------- Co-authored-by: yangzhong <[email protected]> * Upgrade datafusion to 20.0.0 & sqlparser to to 0.32.0 (apache#711) * Upgrade datafusion & sqlparser * Move ballista_round_trip tests of benchmark into a separate feature to avoid stack overflow * Fix failed tests of scheduler * Update README.md (apache#729) * Update link to proto file in dev docs (apache#713) * Fix `show tables` fails (apache#715) * Remove cancelled jobs from active cache (#36) * Downgrade expected error to warning (#37) * Downgrade expected error to warning * add context * Serialize configoptions and pass them to executor (#34) * serialize configoptions and pass them to executor and allow extensions for TaskContext * use ConfigOptions::with_extensions * fix usage of ConfigOptions * clippy * Add wait_drained to SchedulerServer and Executor (#41) * Add missing code from previous commits * Fixes after merging from master * Reintroduce Executor::with_functions * Adapt prometheus histogram buckets * cargo tomlfmt * cargo fmt --all * Allow too_many_arguments lint * Cargo tomlfmt --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Tim Van Wassenhove <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Ian Alexander Joiner <[email protected]> Co-authored-by: jiangzhx <[email protected]> Co-authored-by: Yang Jiang <[email protected]> Co-authored-by: Lakkam Sai Krishna Reddy <[email protected]> Co-authored-by: Vrishabh <[email protected]> Co-authored-by: Daniël Heres <[email protected]> Co-authored-by: Daniël Heres <[email protected]> Co-authored-by: Joe Williams <[email protected]> Co-authored-by: Jaap Aarts <[email protected]> Co-authored-by: mpurins-coralogix <[email protected]>
1 parent 0637711 commit dbc0397

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+5191
-9328
lines changed

‎.github/dependabot.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ updates:
2222
schedule:
2323
interval: daily
2424
open-pull-requests-limit: 10
25-
target-branch: master
25+
target-branch: main
2626
labels: [auto-dependencies]
2727
ignore:
2828
# arrow and datafusion are bumped manually

‎.github/workflows/comment_bot.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -58,14 +58,14 @@ jobs:
5858
- uses: r-lib/actions/pr-fetch@master
5959
with:
6060
repo-token: ${{ secrets.GITHUB_TOKEN }}
61-
- name: Rebase on ${{ github.repository }} master
61+
- name: Rebase on ${{ github.repository }} main
6262
run: |
6363
set -ex
6464
git config user.name "$(git log -1 --pretty=format:%an)"
6565
git config user.email "$(git log -1 --pretty=format:%ae)"
6666
git remote add upstream https://github.com/${{ github.repository }}
67-
git fetch --unshallow upstream master
68-
git rebase upstream/master
67+
git fetch --unshallow upstream main
68+
git rebase upstream/main
6969
- uses: r-lib/actions/pr-push@master
7070
with:
7171
repo-token: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/rust.yml

+57-4
Original file line numberDiff line numberDiff line change
@@ -74,14 +74,23 @@ jobs:
7474
7575
react-build:
7676
name: React build
77-
runs-on: ubuntu-latest
77+
runs-on: ubuntu-latest # proprietary github image, not ubuntu:latest
78+
container:
79+
image: ubuntu:latest # actual ubuntu:latest that ubuntu publishes
7880
steps:
7981
- uses: actions/checkout@v3
8082
- name: Install dependencies
8183
run: |
82-
sudo apt-get -qq update
83-
sudo apt-get -qq install -y nodejs npm
84+
apt-get -qq update
85+
apt-get -qq install -y nodejs npm
8486
npm install -g yarn
87+
cat /etc/*-release
88+
which node
89+
which npm
90+
which yarn
91+
node --version
92+
npm --version
93+
yarn --version
8594
- name: Run yarn build
8695
run: |
8796
cd ballista/scheduler/ui
@@ -259,6 +268,50 @@ jobs:
259268
# do not produce debug symbols to keep memory usage down
260269
RUSTFLAGS: "-C debuginfo=0"
261270

271+
# verify that the benchmark queries return the correct results
272+
verify-benchmark-results:
273+
name: verify benchmark results (amd64)
274+
needs: [linux-build-lib]
275+
runs-on: ubuntu-latest
276+
strategy:
277+
matrix:
278+
arch: [amd64]
279+
rust: [stable]
280+
container:
281+
image: ${{ matrix.arch }}/rust
282+
env:
283+
# Disable full debug symbol generation to speed up CI build and keep memory down
284+
# "1" means line tables only, which is useful for panic tracebacks.
285+
RUSTFLAGS: "-C debuginfo=1"
286+
steps:
287+
- uses: actions/checkout@v3
288+
with:
289+
submodules: true
290+
- name: Install protobuf compiler
291+
shell: bash
292+
run: |
293+
apt-get -qq update && apt-get -y -qq install protobuf-compiler
294+
protoc --version
295+
- name: Cache Cargo
296+
uses: actions/cache@v3
297+
with:
298+
path: /github/home/.cargo
299+
# this key equals the ones on `linux-build-lib` for re-use
300+
key: cargo-cache-
301+
- name: Cache Rust dependencies
302+
uses: actions/cache@v3
303+
with:
304+
path: /github/home/target
305+
# this key equals the ones on `linux-build-lib` for re-use
306+
key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}
307+
- name: Setup Rust toolchain
308+
uses: ./.github/actions/setup-builder
309+
with:
310+
rust-version: ${{ matrix.rust }}
311+
- name: Verify that benchmark queries return expected results
312+
run: |
313+
cargo test --package ballista-benchmarks --profile release-nonlto --features=ci -- --test-threads=1
314+
262315
lint:
263316
name: Lint
264317
runs-on: ubuntu-latest
@@ -304,7 +357,7 @@ jobs:
304357
if [[ $DOCKER_TAG =~ ^[0-9\.]+-rc[0-9]+$ ]]
305358
then
306359
echo "publishing docker tag $DOCKER_TAG"
307-
docker tag arrow-ballista-standalone:latest ghcr.io/apache/arrow-ballista-standalone:$DOCKER_TAG
360+
docker tag arrow-ballista-standalone:latest ghcr.io/apache/arrow-ballista-standalone:$DOCKER_TAG
308361
docker login ghcr.io -u $DOCKER_USER -p "$DOCKER_PASS"
309362
docker push ghcr.io/apache/arrow-ballista-standalone:$DOCKER_TAG
310363
fi

‎.gitignore

+1-1
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ rusty-tags.vi
9292
.flatbuffers/
9393

9494
.vscode
95-
venv/*
95+
venv/
9696

9797
# apache release artifacts
9898
dev/dist

‎Cargo.toml

+21-9
Original file line numberDiff line numberDiff line change
@@ -16,20 +16,32 @@
1616
# under the License.
1717

1818
[workspace]
19-
members = [
20-
"ballista-cli",
21-
"ballista/client",
22-
"ballista/core",
23-
"ballista/executor",
24-
"ballista/scheduler",
25-
"benchmarks",
26-
"examples",
27-
]
2819
exclude = ["python"]
20+
members = ["ballista-cli", "ballista/client", "ballista/core", "ballista/executor", "ballista/scheduler", "benchmarks", "examples",
21+
]
22+
23+
[workspace.dependencies]
24+
arrow = { version = "34.0.0" }
25+
arrow-flight = { version = "34.0.0", features = ["flight-sql-experimental"] }
26+
datafusion = "20.0.0"
27+
datafusion-proto = "20.0.0"
2928

3029
# cargo build --profile release-lto
3130
[profile.release-lto]
3231
codegen-units = 1
3332
inherits = "release"
3433
lto = true
3534

35+
# the release profile takes a long time to build so we can use this profile during development to save time
36+
# cargo build --profile release-nonlto
37+
[profile.release-nonlto]
38+
codegen-units = 16
39+
debug = false
40+
debug-assertions = false
41+
incremental = false
42+
inherits = "release"
43+
lto = false
44+
opt-level = 3
45+
overflow-checks = false
46+
panic = 'unwind'
47+
rpath = false

‎ballista-cli/Cargo.toml

+4-4
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
[package]
1919
name = "ballista-cli"
2020
description = "Command Line Client for Ballista distributed query engine."
21-
version = "0.10.0"
21+
version = "0.11.0"
2222
authors = ["Apache Arrow <[email protected]>"]
2323
edition = "2021"
2424
keywords = ["ballista", "cli"]
@@ -29,12 +29,12 @@ rust-version = "1.63"
2929
readme = "README.md"
3030

3131
[dependencies]
32-
ballista = { path = "../ballista/client", version = "0.10.0", features = [
32+
ballista = { path = "../ballista/client", version = "0.11.0", features = [
3333
"standalone",
3434
] }
3535
clap = { version = "3", features = ["derive", "cargo"] }
36-
datafusion = { git = "https://github.com/coralogix/arrow-datafusion.git", tag = "v18.0.0-cx.2" }
37-
datafusion-cli = { git = "https://github.com/coralogix/arrow-datafusion.git", tag = "v18.0.0-cx.2" }
36+
datafusion = { workspace = true }
37+
datafusion-cli = "20.0.0"
3838
dirs = "4.0.0"
3939
env_logger = "0.10"
4040
mimalloc = { version = "0.1", default-features = false }

‎ballista/CHANGELOG.md

+82
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,88 @@
1919

2020
# Changelog
2121

22+
## [0.11.0](https://github.com/apache/arrow-ballista/tree/0.11.0) (2023-02-19)
23+
24+
[Full Changelog](https://github.com/apache/arrow-ballista/compare/0.10.0...0.11.0)
25+
26+
**Implemented enhancements:**
27+
28+
- Remove `python` since it has been moved to its own repo, `arrow-ballista-python` [\#653](https://github.com/apache/arrow-ballista/issues/653)
29+
- Add executor self-registration mechanism in the heartbeat service [\#648](https://github.com/apache/arrow-ballista/issues/648)
30+
- Upgrade to DataFusion 17 [\#638](https://github.com/apache/arrow-ballista/issues/638)
31+
- Move Python bindings to separate repo? [\#635](https://github.com/apache/arrow-ballista/issues/635)
32+
- Implement new release process [\#622](https://github.com/apache/arrow-ballista/issues/622)
33+
- Change default branch name from master to main [\#618](https://github.com/apache/arrow-ballista/issues/618)
34+
- Update latest datafusion dependency [\#610](https://github.com/apache/arrow-ballista/issues/610)
35+
- Implement optimizer rule to remove redundant repartitioning [\#608](https://github.com/apache/arrow-ballista/issues/608)
36+
- ballista-cli as \(docker\) images [\#600](https://github.com/apache/arrow-ballista/issues/600)
37+
- Update contributor guide [\#598](https://github.com/apache/arrow-ballista/issues/598)
38+
- Fix cargo clippy [\#570](https://github.com/apache/arrow-ballista/issues/570)
39+
- Support Alibaba Cloud OSS with ObjectStore [\#566](https://github.com/apache/arrow-ballista/issues/566)
40+
- Refactor `StateBackendClient` to be a higher-level interface [\#554](https://github.com/apache/arrow-ballista/issues/554)
41+
- Make it concurrently to launch tasks to executors [\#544](https://github.com/apache/arrow-ballista/issues/544)
42+
- Simplify docs [\#531](https://github.com/apache/arrow-ballista/issues/531)
43+
- Provide an in-memory StateBackend [\#505](https://github.com/apache/arrow-ballista/issues/505)
44+
- Add support for Azure blob storage [\#294](https://github.com/apache/arrow-ballista/issues/294)
45+
- Add a workflow to build the image and publish it to the package [\#71](https://github.com/apache/arrow-ballista/issues/71)
46+
47+
**Fixed bugs:**
48+
49+
- Rust / Check Cargo.toml formatting \(amd64, stable\) \(pull\_request\) Failing [\#662](https://github.com/apache/arrow-ballista/issues/662)
50+
- Protobuf parsing error [\#646](https://github.com/apache/arrow-ballista/issues/646)
51+
- jobs from python client not showing up in Scheduler UI [\#625](https://github.com/apache/arrow-ballista/issues/625)
52+
- ballista ui fails to build [\#594](https://github.com/apache/arrow-ballista/issues/594)
53+
- cargo build --release fails for ballista-scheduler [\#590](https://github.com/apache/arrow-ballista/issues/590)
54+
- docker build fails [\#589](https://github.com/apache/arrow-ballista/issues/589)
55+
- Multi-scheduler Job Starvation [\#585](https://github.com/apache/arrow-ballista/issues/585)
56+
- Cannot query file from S3 [\#559](https://github.com/apache/arrow-ballista/issues/559)
57+
- Benchmark q16 fails [\#373](https://github.com/apache/arrow-ballista/issues/373)
58+
59+
**Documentation updates:**
60+
61+
- Check in benchmark image [\#647](https://github.com/apache/arrow-ballista/pull/647) ([andygrove](https://github.com/andygrove))
62+
- MINOR: Fix benchmark image link [\#596](https://github.com/apache/arrow-ballista/pull/596) ([andygrove](https://github.com/andygrove))
63+
64+
**Merged pull requests:**
65+
66+
- Upgrade to DataFusion 18 [\#668](https://github.com/apache/arrow-ballista/pull/668) ([andygrove](https://github.com/andygrove))
67+
- Enable physical plan round-trip tests [\#666](https://github.com/apache/arrow-ballista/pull/666) ([andygrove](https://github.com/andygrove))
68+
- Upgrade to DataFusion 18.0.0-rc1 [\#664](https://github.com/apache/arrow-ballista/pull/664) ([andygrove](https://github.com/andygrove))
69+
- add test\_util to make examples work well [\#661](https://github.com/apache/arrow-ballista/pull/661) ([jiangzhx](https://github.com/jiangzhx))
70+
- Minor refactor to reduce duplicate code [\#659](https://github.com/apache/arrow-ballista/pull/659) ([andygrove](https://github.com/andygrove))
71+
- Cluster state refactor Part 2 [\#658](https://github.com/apache/arrow-ballista/pull/658) ([thinkharderdev](https://github.com/thinkharderdev))
72+
- Remove `python` dir & python-related workflows [\#654](https://github.com/apache/arrow-ballista/pull/654) ([iajoiner](https://github.com/iajoiner))
73+
- Add executor self-registration mechanism in the heartbeat service [\#649](https://github.com/apache/arrow-ballista/pull/649) ([yahoNanJing](https://github.com/yahoNanJing))
74+
- Upgrade to DataFusion 17 [\#639](https://github.com/apache/arrow-ballista/pull/639) ([avantgardnerio](https://github.com/avantgardnerio))
75+
- Upgrade to DataFusion 16 \(again\) [\#636](https://github.com/apache/arrow-ballista/pull/636) ([avantgardnerio](https://github.com/avantgardnerio))
76+
- Update release process documentation [\#632](https://github.com/apache/arrow-ballista/pull/632) ([andygrove](https://github.com/andygrove))
77+
- Implement new release process [\#623](https://github.com/apache/arrow-ballista/pull/623) ([andygrove](https://github.com/andygrove))
78+
- Update contributor guide [\#617](https://github.com/apache/arrow-ballista/pull/617) ([andygrove](https://github.com/andygrove))
79+
- Fix Cargo.toml format issue [\#616](https://github.com/apache/arrow-ballista/pull/616) ([andygrove](https://github.com/andygrove))
80+
- Refactor scheduler main [\#615](https://github.com/apache/arrow-ballista/pull/615) ([andygrove](https://github.com/andygrove))
81+
- Refactor executor main [\#614](https://github.com/apache/arrow-ballista/pull/614) ([andygrove](https://github.com/andygrove))
82+
- Update datafusion dependency to the latest version [\#612](https://github.com/apache/arrow-ballista/pull/612) ([yahoNanJing](https://github.com/yahoNanJing))
83+
- Add support for Azure Blob Storage [\#599](https://github.com/apache/arrow-ballista/pull/599) ([aidankovacic-8451](https://github.com/aidankovacic-8451))
84+
- Python: add method to get explain output as a string [\#593](https://github.com/apache/arrow-ballista/pull/593) ([andygrove](https://github.com/andygrove))
85+
- Handle job resubmission [\#586](https://github.com/apache/arrow-ballista/pull/586) ([thinkharderdev](https://github.com/thinkharderdev))
86+
- updated readme to contain correct versions of dependencies. [\#580](https://github.com/apache/arrow-ballista/pull/580) ([saikrishna1-bidgely](https://github.com/saikrishna1-bidgely))
87+
- Update graphviz-rust requirement from 0.4.0 to 0.5.0 [\#574](https://github.com/apache/arrow-ballista/pull/574) ([dependabot[bot]](https://github.com/apps/dependabot))
88+
- Super minor spelling error [\#573](https://github.com/apache/arrow-ballista/pull/573) ([jdye64](https://github.com/jdye64))
89+
- Fix cargo clippy [\#571](https://github.com/apache/arrow-ballista/pull/571) ([yahoNanJing](https://github.com/yahoNanJing))
90+
- Support Alibaba Cloud OSS with ObjectStore [\#567](https://github.com/apache/arrow-ballista/pull/567) ([r4ntix](https://github.com/r4ntix))
91+
- fix\(ui\): fix last seen [\#562](https://github.com/apache/arrow-ballista/pull/562) ([duyet](https://github.com/duyet))
92+
- Cluster state refactor part 1 [\#560](https://github.com/apache/arrow-ballista/pull/560) ([thinkharderdev](https://github.com/thinkharderdev))
93+
- Make it concurrently to launch tasks to executors [\#557](https://github.com/apache/arrow-ballista/pull/557) ([yahoNanJing](https://github.com/yahoNanJing))
94+
- Update datafusion requirement from 14.0.0 to 15.0.0 [\#552](https://github.com/apache/arrow-ballista/pull/552) ([yahoNanJing](https://github.com/yahoNanJing))
95+
- docs: fix style in the Helm readme [\#551](https://github.com/apache/arrow-ballista/pull/551) ([haoxins](https://github.com/haoxins))
96+
- Fix Helm chart's image format [\#550](https://github.com/apache/arrow-ballista/pull/550) ([haoxins](https://github.com/haoxins))
97+
- Update env\_logger requirement from 0.9 to 0.10 [\#539](https://github.com/apache/arrow-ballista/pull/539) ([dependabot[bot]](https://github.com/apps/dependabot))
98+
- only build docker images on rc tags [\#535](https://github.com/apache/arrow-ballista/pull/535) ([andygrove](https://github.com/andygrove))
99+
- Remove `--locked` when building Python wheels [\#533](https://github.com/apache/arrow-ballista/pull/533) ([andygrove](https://github.com/andygrove))
100+
- Bump actions/labeler from 4.0.2 to 4.1.0 [\#525](https://github.com/apache/arrow-ballista/pull/525) ([dependabot[bot]](https://github.com/apps/dependabot))
101+
- Provide a memory StateBackendClient [\#523](https://github.com/apache/arrow-ballista/pull/523) ([yahoNanJing](https://github.com/yahoNanJing))
102+
103+
22104
## [0.10.0](https://github.com/apache/arrow-ballista/tree/0.10.0) (2022-11-18)
23105

24106
[Full Changelog](https://github.com/apache/arrow-ballista/compare/0.9.0...0.10.0)

‎ballista/client/Cargo.toml

+7-7
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
name = "ballista"
2020
description = "Ballista Distributed Compute"
2121
license = "Apache-2.0"
22-
version = "0.10.0"
22+
version = "0.11.0"
2323
homepage = "https://github.com/apache/arrow-ballista"
2424
repository = "https://github.com/apache/arrow-ballista"
2525
readme = "README.md"
@@ -28,15 +28,15 @@ edition = "2021"
2828
rust-version = "1.63"
2929

3030
[dependencies]
31-
ballista-core = { path = "../core", version = "0.10.0" }
32-
ballista-executor = { path = "../executor", version = "0.10.0", optional = true }
33-
ballista-scheduler = { path = "../scheduler", version = "0.10.0", optional = true }
34-
datafusion = { git = "https://github.com/coralogix/arrow-datafusion.git", tag = "v18.0.0-cx.2" }
35-
datafusion-proto = { git = "https://github.com/coralogix/arrow-datafusion.git", tag = "v18.0.0-cx.2" }
31+
ballista-core = { path = "../core", version = "0.11.0" }
32+
ballista-executor = { path = "../executor", version = "0.11.0", optional = true }
33+
ballista-scheduler = { path = "../scheduler", version = "0.11.0", optional = true }
34+
datafusion = { workspace = true }
35+
datafusion-proto = { workspace = true }
3636
futures = "0.3"
3737
log = "0.4"
3838
parking_lot = "0.12"
39-
sqlparser = "0.30.0"
39+
sqlparser = "0.32.0"
4040
tempfile = "3"
4141
tokio = "1.0"
4242

‎ballista/client/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ To build a simple ballista example, add the following dependencies to your `Carg
8585
```toml
8686
[dependencies]
8787
ballista = "0.10"
88-
datafusion = "14.0.0"
88+
datafusion = "20.0.0"
8989
tokio = "1.0"
9090
```
9191

0 commit comments

Comments
 (0)