Skip to content

Commit 808606f

Browse files
fsdvhBrent Gardnerandygrovedependabot[bot]yahoNanJing
authored
SC-16350: More specific errors (#44)
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * feat: update script such that ballista-cli image is built as well (apache#601) * Fix Cargo.toml format issue (apache#616) * Refactor executor main (apache#614) * Refactor executor main * copy all configs * toml fmt * Refactor scheduler main (apache#615) * refactor scheduler main * toml fmt * Python: add method to get explain output as a string (apache#593) * Update contributor guide (apache#617) * Cluster state refactor part 1 (apache#560) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> * replace master with main (apache#621) * implement new release process (apache#623) * add docs on who can release (apache#632) * Upgrade to DataFusion 16 (again) (apache#636) * Update datafusion dependency to the latest version (apache#612) * Update datafusion dependency to the latest version * Fix python * Skip ut of test_window_lead due to apache/datafusion-python#135 * Fix clippy --------- Co-authored-by: yangzhong <[email protected]> * Upgrade to DataFusion 17 (apache#639) * Upgrade to DF 17 * Restore original error handling functionality * check in benchmark image (apache#647) * Remove `python` dir & python-related workflows (apache#654) * refactor: remove python dir & python-related workflows * remove brackets * Handle job resubmission (apache#586) * Handle job resubmission * Make resubmission configurable and add test * Fix debug log * Add executor self-registration mechanism in the heartbeat service (apache#649) Co-authored-by: yangzhong <[email protected]> * Cluster state refactor Part 2 (apache#658) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header * WIP * Implement JobState * Tests and fixes * do not hold ref across await point * Fix clippy warnings * Fix tomlfmt github action * uncomment test --------- Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> * Upgrade to DataFusion 18.0.0-rc1 (apache#664) * Minor refactor to reduce duplicate code (apache#659) * move test_util to ballista-examples package (apache#661) * Upgrade to DataFusion 18 (apache#668) * Enable physical plan round-trip tests (apache#666) * Prep 0.11 (apache#682) * Change version to 0.11.0 * changelog * update react-timeago version * yarn upgrade * fix * fix * revert yarn change * Print versions * Print locations * Avoid github shenanigans * Try to get runners running * Try to get runners running * already root --------- Co-authored-by: Andy Grove <[email protected]> * [minor] remove todo (apache#683) * Add executor terminating status for graceful shutdown (apache#667) * Add executor terminating status for graceful shutdown * Remove empty file * Update ballista/executor/src/executor_process.rs Co-authored-by: Brent Gardner <[email protected]> --------- Co-authored-by: Brent Gardner <[email protected]> * Allow `BallistaContext::read_*` methods to read multiple paths. (apache#679) * updated dependency in cargo, added read_json method, modified read_* methods to read multiple paths. * ran cargo fmt * Added revision for proper builds. * Update scheduler.md (apache#657) * Mark `SchedulerState` as pub (apache#688) * Mark as pub * Fmt --------- Co-authored-by: Daniël Heres <[email protected]> * Update graphviz-rust requirement from 0.5.0 to 0.6.1 (apache#651) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Upgrade DataFusion to 19.0.0 (apache#691) * update release notes (apache#692) * Make task launcher pub (apache#695) Co-authored-by: Daniël Heres <[email protected]> * Make task_manager pub (apache#698) Co-authored-by: Daniël Heres <[email protected]> * Add ExecutionEngine abstraction (apache#687) * Allow accessing s3 locations in client mode (apache#700) * Allow accessing s3 locations in client mode * Removed s3 feature from test dependencies. * fixed cargo-tomlfmt issues * deployment/docker-compose.md incorrect remote ref (apache#699) * Fix for error message during testing (apache#707) * Fix cargo clippy * Fix for error message during testing * Remove unwrap for dealing with JobQueued event * log task ids when launch tasks --------- Co-authored-by: yangzhong <[email protected]> * Upgrade datafusion to 20.0.0 & sqlparser to to 0.32.0 (apache#711) * Upgrade datafusion & sqlparser * Move ballista_round_trip tests of benchmark into a separate feature to avoid stack overflow * Fix failed tests of scheduler * Update README.md (apache#729) * Update link to proto file in dev docs (apache#713) * Fix `show tables` fails (apache#715) * Remove cancelled jobs from active cache (#36) * Downgrade expected error to warning (#37) * Downgrade expected error to warning * add context * Serialize configoptions and pass them to executor (#34) * serialize configoptions and pass them to executor and allow extensions for TaskContext * use ConfigOptions::with_extensions * fix usage of ConfigOptions * clippy * Add wait_drained to SchedulerServer and Executor (#41) * Add missing code from previous commits * Fixes after merging from master * Reintroduce Executor::with_functions * Adapt prometheus histogram buckets * cargo tomlfmt * cargo fmt --all * Allow too_many_arguments lint * sc-16350: introducing notion of external and internal error in the failed job status * Cargo tomlfmt * sc-16350: small test fix * sc-16350: partially implemented ballista error serialization * sc-16350: update failed job proto definition * sc-16350: cleanup * sc-16350: update protoc * sc-16350: update action * sc-16350: more action update * sc-16350: update test * sc-16350: allow optional fields * VTX-522: fix models * VTX-522: cleanup * Change error to warn * VTX-522: fixes --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Tim Van Wassenhove <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Ian Alexander Joiner <[email protected]> Co-authored-by: jiangzhx <[email protected]> Co-authored-by: Yang Jiang <[email protected]> Co-authored-by: Lakkam Sai Krishna Reddy <[email protected]> Co-authored-by: Vrishabh <[email protected]> Co-authored-by: Daniël Heres <[email protected]> Co-authored-by: Daniël Heres <[email protected]> Co-authored-by: Joe Williams <[email protected]> Co-authored-by: Jaap Aarts <[email protected]> Co-authored-by: mpurins-coralogix <[email protected]> Co-authored-by: Christoph Schulze <[email protected]> Co-authored-by: Dan Harris <[email protected]>
1 parent 57029df commit 808606f

File tree

19 files changed

+1639
-171
lines changed

19 files changed

+1639
-171
lines changed

‎.github/workflows/rust.yml

+6-6
Original file line numberDiff line numberDiff line change
@@ -180,8 +180,8 @@ jobs:
180180
run: |
181181
mkdir -p $HOME/d/protoc
182182
cd $HOME/d/protoc
183-
export PROTO_ZIP="protoc-21.4-linux-x86_64.zip"
184-
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v21.4/$PROTO_ZIP
183+
export PROTO_ZIP="protoc-22.2-linux-x86_64.zip"
184+
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v22.2/$PROTO_ZIP
185185
unzip $PROTO_ZIP
186186
export PATH=$PATH:$HOME/d/protoc/bin
187187
protoc --version
@@ -232,8 +232,8 @@ jobs:
232232
run: |
233233
mkdir -p $HOME/d/protoc
234234
cd $HOME/d/protoc
235-
export PROTO_ZIP="protoc-21.4-osx-x86_64.zip"
236-
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v21.4/$PROTO_ZIP
235+
export PROTO_ZIP="protoc-22.2-osx-x86_64.zip"
236+
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v22.2/$PROTO_ZIP
237237
unzip $PROTO_ZIP
238238
echo "$HOME/d/protoc/bin" >> $GITHUB_PATH
239239
export PATH=$PATH:$HOME/d/protoc/bin
@@ -244,8 +244,8 @@ jobs:
244244
run: |
245245
mkdir -p $HOME/d/protoc
246246
cd $HOME/d/protoc
247-
export PROTO_ZIP="protoc-21.4-win64.zip"
248-
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v21.4/$PROTO_ZIP
247+
export PROTO_ZIP="protoc-22.2-win64.zip"
248+
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v22.2/$PROTO_ZIP
249249
unzip $PROTO_ZIP
250250
export PATH=$PATH:$HOME/d/protoc/bin
251251
protoc.exe --version

‎ballista/core/build.rs

+1
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ fn main() -> Result<(), String> {
4141
println!("cargo:rerun-if-changed=proto/ballista.proto");
4242
tonic_build::configure()
4343
.extern_path(".datafusion", "::datafusion_proto::protobuf")
44+
.protoc_arg("--experimental_allow_proto3_optional")
4445
.compile(&["proto/ballista.proto"], &["proto"])
4546
.map_err(|e| format!("protobuf compilation failed: {e}"))?;
4647
let generated_source_path = out.join("ballista.protobuf.rs");

0 commit comments

Comments
 (0)