Skip to content

Commit e6caffe

Browse files
fsdvhBrent Gardnerandygrovedependabot[bot]yahoNanJing
committed
Upstream rebase (#29)
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * fmt * clippy * tomlfmt Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Dan Harris <[email protected]>
1 parent 13572f3 commit e6caffe

File tree

17 files changed

+3941
-435
lines changed

17 files changed

+3941
-435
lines changed

ballista/core/Cargo.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -55,9 +55,9 @@ datafusion-objectstore-hdfs = { version = "0.1.1", default-features = false, opt
5555
datafusion-proto = "18.0.0"
5656
futures = "0.3"
5757
hashbrown = "0.13"
58+
itertools = "0.10"
5859

5960
lazy_static = "1.4.0"
60-
itertools = "0.10"
6161
libloading = "0.7.3"
6262
log = "0.4"
6363
lru = "0.8.1"

ballista/core/src/serde/physical_plan/from_proto.rs

+2
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,8 @@ impl TryInto<FileScanConfig> for &protobuf::FileScanExecConf {
411411
projection,
412412
limit: self.limit.as_ref().map(|sl| sl.limit as usize),
413413
table_partition_cols: vec![],
414+
// TODO add ordering info to the ballista proto file
415+
output_ordering: None,
414416
})
415417
}
416418
}

ballista/core/src/serde/physical_plan/mod.rs

+1,720
Large diffs are not rendered by default.

ballista/core/src/serde/physical_plan/to_proto.rs

+468
Large diffs are not rendered by default.

ballista/scheduler/src/cluster/storage/etcd.rs

+1-1
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ use crate::state::backend::{
3434
Keyspace, Lock, Operation, StateBackendClient, Watch, WatchEvent,
3535
};
3636

37-
/// A [`StateBackendClient`] implementation that uses etcd to save cluster configuration.
37+
/// A [`StateBackendClient`] implementation that uses etcd to save cluster state.
3838
#[derive(Clone)]
3939
pub struct EtcdClient {
4040
namespace: String,

0 commit comments

Comments
 (0)