Upstream rebase #29
Merged
Upstream rebase #29
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not
* bump version * CHANGELOG
* Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy
* fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file
* Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]>
* Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]>
* Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]>
* Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]>
Co-authored-by: yangzhong <[email protected]>
Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Remove outdated script and use evergreen version of rust * Use debian protobuf
* Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]>
* Record when job is queueud in scheduler metrics * add additional buckets for exec times
github-actions
bot
added
development-process
documentation
fsdvh
added a commit
that referenced
this pull request
Feb 2, 2023
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * fmt * clippy * tomlfmt Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Dan Harris <[email protected]>
fsdvh
added a commit
that referenced
this pull request
Feb 2, 2023
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * feat: update script such that ballista-cli image is built as well (apache#601) * Fix Cargo.toml format issue (apache#616) * Refactor executor main (apache#614) * Refactor executor main * copy all configs * toml fmt * Refactor scheduler main (apache#615) * refactor scheduler main * toml fmt * Python: add method to get explain output as a string (apache#593) * Update contributor guide (apache#617) * Cluster state refactor part 1 (apache#560) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> * replace master with main (apache#621) * implement new release process (apache#623) * add docs on who can release (apache#632) * Upgrade to DataFusion 16 (again) (apache#636) * Update datafusion dependency to the latest version (apache#612) * Update datafusion dependency to the latest version * Fix python * Skip ut of test_window_lead due to apache/datafusion-python#135 * Fix clippy --------- Co-authored-by: yangzhong <[email protected]> * Upgrade to DataFusion 17 (apache#639) * Upgrade to DF 17 * Restore original error handling functionality * Customize session builder * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * Upstream rebase (#29) * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * fmt * clippy * tomlfmt Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Dan Harris <[email protected]> * Post merge update * update message formatting * post merge update * another post-merge updates * update github actions * clippy * update script * fmt --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Tim Van Wassenhove <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]>
fsdvh
added a commit
that referenced
this pull request
Feb 15, 2023
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * fmt * clippy * tomlfmt Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Dan Harris <[email protected]>
fsdvh
added a commit
that referenced
this pull request
Feb 15, 2023
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * feat: update script such that ballista-cli image is built as well (apache#601) * Fix Cargo.toml format issue (apache#616) * Refactor executor main (apache#614) * Refactor executor main * copy all configs * toml fmt * Refactor scheduler main (apache#615) * refactor scheduler main * toml fmt * Python: add method to get explain output as a string (apache#593) * Update contributor guide (apache#617) * Cluster state refactor part 1 (apache#560) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> * replace master with main (apache#621) * implement new release process (apache#623) * add docs on who can release (apache#632) * Upgrade to DataFusion 16 (again) (apache#636) * Update datafusion dependency to the latest version (apache#612) * Update datafusion dependency to the latest version * Fix python * Skip ut of test_window_lead due to apache/datafusion-python#135 * Fix clippy --------- Co-authored-by: yangzhong <[email protected]> * Upgrade to DataFusion 17 (apache#639) * Upgrade to DF 17 * Restore original error handling functionality * Customize session builder * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * Upstream rebase (#29) * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * fmt * clippy * tomlfmt Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Dan Harris <[email protected]> * Post merge update * update message formatting * post merge update * another post-merge updates * update github actions * clippy * update script * fmt --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Tim Van Wassenhove <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]>
fsdvh
added a commit
that referenced
this pull request
Feb 17, 2023
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * feat: update script such that ballista-cli image is built as well (apache#601) * Fix Cargo.toml format issue (apache#616) * Refactor executor main (apache#614) * Refactor executor main * copy all configs * toml fmt * Refactor scheduler main (apache#615) * refactor scheduler main * toml fmt * Python: add method to get explain output as a string (apache#593) * Update contributor guide (apache#617) * Cluster state refactor part 1 (apache#560) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> * replace master with main (apache#621) * implement new release process (apache#623) * add docs on who can release (apache#632) * Upgrade to DataFusion 16 (again) (apache#636) * Update datafusion dependency to the latest version (apache#612) * Update datafusion dependency to the latest version * Fix python * Skip ut of test_window_lead due to apache/datafusion-python#135 * Fix clippy --------- Co-authored-by: yangzhong <[email protected]> * Upgrade to DataFusion 17 (apache#639) * Upgrade to DF 17 * Restore original error handling functionality * check in benchmark image (apache#647) * Remove `python` dir & python-related workflows (apache#654) * refactor: remove python dir & python-related workflows * remove brackets * Handle job resubmission (apache#586) * Handle job resubmission * Make resubmission configurable and add test * Fix debug log * Add executor self-registration mechanism in the heartbeat service (apache#649) Co-authored-by: yangzhong <[email protected]> * Cluster state refactor Part 2 (apache#658) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header * WIP * Implement JobState * Tests and fixes * do not hold ref across await point * Fix clippy warnings * Fix tomlfmt github action * uncomment test --------- Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> * Upgrade to DataFusion 18.0.0-rc1 (apache#664) * Add executor terminating status for graceful shutdown * Remove empty file * Minor refactor to reduce duplicate code (apache#659) * move test_util to ballista-examples package (apache#661) * Upgrade to DataFusion 18 (apache#668) * Enable physical plan round-trip tests (apache#666) * Customize session builder * Construct Executor with functions * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Make parse_physical_expr public * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * Upstream rebase (#29) * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * fmt * clippy * tomlfmt Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Dan Harris <[email protected]> * Update from upstream (#30) * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * feat: update script such that ballista-cli image is built as well (apache#601) * Fix Cargo.toml format issue (apache#616) * Refactor executor main (apache#614) * Refactor executor main * copy all configs * toml fmt * Refactor scheduler main (apache#615) * refactor scheduler main * toml fmt * Python: add method to get explain output as a string (apache#593) * Update contributor guide (apache#617) * Cluster state refactor part 1 (apache#560) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> * replace master with main (apache#621) * implement new release process (apache#623) * add docs on who can release (apache#632) * Upgrade to DataFusion 16 (again) (apache#636) * Update datafusion dependency to the latest version (apache#612) * Update datafusion dependency to the latest version * Fix python * Skip ut of test_window_lead due to apache/datafusion-python#135 * Fix clippy --------- Co-authored-by: yangzhong <[email protected]> * Upgrade to DataFusion 17 (apache#639) * Upgrade to DF 17 * Restore original error handling functionality * Customize session builder * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * Upstream rebase (#29) * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * fmt * clippy * tomlfmt Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Dan Harris <[email protected]> * Post merge update * update message formatting * post merge update * another post-merge updates * update github actions * clippy * update script * fmt --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Tim Van Wassenhove <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]> * post merge fixes * fix branch naming in github actions * cleanup * fmt * update imports --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Tim Van Wassenhove <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Ian Alexander Joiner <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: jiangzhx <[email protected]> Co-authored-by: Dan Harris <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To fix some of the issues in DQE we need to move on to datafusion 15.0.0. I thought it would be a good idea to sync with upstream