Skip to content

Commit 5ce3c26

Browse files
fsdvhBrent Gardnerandygrovedependabot[bot]yahoNanJing
committed
Upstream rebase (#29)
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <[email protected]> Co-authored-by: Andy Grove <[email protected]> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <[email protected]> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <[email protected]> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <[email protected]> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <[email protected]> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <[email protected]> * Add ClusterState trait * Expose active job count * Remove println * Resubmit jobs when no resources available for scheduling * Make parse_physical_expr public * Reduce log spam * Fix job submitted metric by ignoring resubmissions * Record when job is queued in scheduler metrics (#28) * Record when job is queueud in scheduler metrics * add additional buckets for exec times * fmt * clippy * tomlfmt Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Andy Grove <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <[email protected]> Co-authored-by: yangzhong <[email protected]> Co-authored-by: Xin Hao <[email protected]> Co-authored-by: Duyet Le <[email protected]> Co-authored-by: r.4ntix <[email protected]> Co-authored-by: Jeremy Dyer <[email protected]> Co-authored-by: Sai Krishna Reddy Lakkam <[email protected]> Co-authored-by: Aidan Kovacic <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Dan Harris <[email protected]> Co-authored-by: Martins Purins <[email protected]> Co-authored-by: Dan Harris <[email protected]>
1 parent 96992d8 commit 5ce3c26

File tree

15 files changed

+825
-446
lines changed

15 files changed

+825
-446
lines changed

ballista/core/Cargo.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -55,9 +55,9 @@ datafusion-objectstore-hdfs = { version = "0.1.1", default-features = false, opt
5555
datafusion-proto = "17.0.0"
5656
futures = "0.3"
5757
hashbrown = "0.13"
58+
itertools = "0.10"
5859

5960
lazy_static = "1.4.0"
60-
itertools = "0.10"
6161
libloading = "0.7.3"
6262
log = "0.4"
6363
lru = "0.8.1"

ballista/core/build.rs

+2-2
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ fn main() -> Result<(), String> {
2626
println!("cargo:rerun-if-env-changed=FORCE_REBUILD");
2727

2828
let version = rustc_version::version().unwrap();
29-
println!("cargo:rustc-env=RUSTC_VERSION={version}");
29+
println!("cargo:rustc-env=RUSTC_VERSION={}", version);
3030

3131
// TODO: undo when resolved: https://github.com/intellij-rust/intellij-rust/issues/9402
3232
#[cfg(feature = "docsrs")]
@@ -42,7 +42,7 @@ fn main() -> Result<(), String> {
4242
tonic_build::configure()
4343
.extern_path(".datafusion", "::datafusion_proto::protobuf")
4444
.compile(&["proto/ballista.proto"], &["proto"])
45-
.map_err(|e| format!("protobuf compilation failed: {e}"))?;
45+
.map_err(|e| format!("protobuf compilation failed: {}", e))?;
4646
let generated_source_path = out.join("ballista.protobuf.rs");
4747
let code = std::fs::read_to_string(generated_source_path).unwrap();
4848
let mut file = std::fs::OpenOptions::new()

ballista/core/proto/ballista.proto

+1
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,7 @@ message AvroScanExecNode {
9595
enum PartitionMode {
9696
COLLECT_LEFT = 0;
9797
PARTITIONED = 1;
98+
AUTO = 2;
9899
}
99100

100101
message HashJoinExecNode {

ballista/core/src/serde/physical_plan/from_proto.rs

+2
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,8 @@ impl TryInto<FileScanConfig> for &protobuf::FileScanExecConf {
411411
projection,
412412
limit: self.limit.as_ref().map(|sl| sl.limit as usize),
413413
table_partition_cols: vec![],
414+
// TODO add ordering info to the ballista proto file
415+
output_ordering: None,
414416
})
415417
}
416418
}

ballista/core/src/serde/physical_plan/mod.rs

+7-1
Original file line numberDiff line numberDiff line change
@@ -331,6 +331,8 @@ impl AsExecutionPlan for PhysicalPlanNode {
331331
physical_window_expr,
332332
input,
333333
Arc::new((&input_schema).try_into()?),
334+
vec![],
335+
None,
334336
)?))
335337
}
336338
PhysicalPlanType::Aggregate(hash_agg) => {
@@ -528,6 +530,7 @@ impl AsExecutionPlan for PhysicalPlanNode {
528530
let partition_mode = match partition_mode {
529531
protobuf::PartitionMode::CollectLeft => PartitionMode::CollectLeft,
530532
protobuf::PartitionMode::Partitioned => PartitionMode::Partitioned,
533+
protobuf::PartitionMode::Auto => PartitionMode::Auto,
531534
};
532535
Ok(Arc::new(HashJoinExec::try_new(
533536
left,
@@ -800,7 +803,7 @@ impl AsExecutionPlan for PhysicalPlanNode {
800803
input: Some(Box::new(input)),
801804
skip: limit.skip() as u32,
802805
fetch: match limit.fetch() {
803-
Some(n) => *n as i64,
806+
Some(n) => n as i64,
804807
_ => -1, // no limit
805808
},
806809
},
@@ -875,6 +878,7 @@ impl AsExecutionPlan for PhysicalPlanNode {
875878
let partition_mode = match exec.partition_mode() {
876879
PartitionMode::CollectLeft => protobuf::PartitionMode::CollectLeft,
877880
PartitionMode::Partitioned => protobuf::PartitionMode::Partitioned,
881+
PartitionMode::Auto => protobuf::PartitionMode::Auto,
878882
};
879883

880884
Ok(protobuf::PhysicalPlanNode {
@@ -1285,6 +1289,7 @@ fn decode_scan_config(
12851289
projection,
12861290
limit: proto.limit.as_ref().map(|sl| sl.limit as usize),
12871291
table_partition_cols: vec![],
1292+
output_ordering: None,
12881293
})
12891294
}
12901295

@@ -1612,6 +1617,7 @@ mod roundtrip_tests {
16121617
projection: None,
16131618
limit: None,
16141619
table_partition_cols: vec![],
1620+
output_ordering: None,
16151621
};
16161622

16171623
let predicate = datafusion::prelude::col("col").eq(datafusion::prelude::lit("1"));

0 commit comments

Comments
 (0)