Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ExecutionEngine abstraction #687

Merged
merged 11 commits into from
Mar 2, 2023
Prev Previous commit
Next Next commit
undo change
  • Loading branch information
andygrove committed Feb 28, 2023
commit f837f502667df61e46ebc7aa27a974584ae5f192
5 changes: 2 additions & 3 deletions ballista/core/src/execution_plans/shuffle_writer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
use datafusion::physical_plan::expressions::PhysicalSortExpr;

use std::any::Any;
use std::fmt::Debug;
use std::future::Future;
use std::iter::Iterator;
use std::path::PathBuf;
Expand Down Expand Up @@ -140,7 +139,7 @@ impl ShuffleWriterExec {
self.shuffle_output_partitioning.as_ref()
}

pub fn execute_shuffle_write_internal(
pub fn execute_shuffle_write(
&self,
input_partition: usize,
context: Arc<TaskContext>,
Expand Down Expand Up @@ -336,7 +335,7 @@ impl ExecutionPlan for ShuffleWriterExec {

let schema_captured = schema.clone();
let fut_stream = self
.execute_shuffle_write_internal(partition, context)
.execute_shuffle_write(partition, context)
.and_then(|part_loc| async move {
// build metadata result batch
let num_writers = part_loc.len();
Expand Down
2 changes: 1 addition & 1 deletion ballista/executor/src/execution_engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ impl QueryStageExecutor for DefaultQueryStageExec {
context: Arc<TaskContext>,
) -> Result<Vec<ShuffleWritePartition>> {
self.shuffle_writer
.execute_shuffle_write_internal(input_partition, context)
.execute_shuffle_write(input_partition, context)
.await
}

Expand Down