Optimizer should have option to skip failing rules #2909
Conversation
github-actions
bot
added
logical-expr
Codecov Report
@@ Coverage Diff @@
## master #2909 +/- ##
==========================================
- Coverage 85.30% 85.30% -0.01%
==========================================
Files 274 274
Lines 49234 49267 +33
==========================================
+ Hits 41999 42027 +28
- Misses 7235 7240 +5
Continue to review full report at Codecov.
|
| debug!("Optimized logical plan:\n{}\n", new_plan.display_indent()); | ||
| } | ||
| Err(ref e) => { | ||
| if optimizer_config.skip_failing_rules { |
There was a problem hiding this comment.
This is the main change here - we now have the option to ignore optimization rules that fail
|
@jdye64 This is part 2 of the index out of bounds fix |
| @@ -43,6 +43,10 @@ pub const OPT_COALESCE_BATCHES: &str = "datafusion.execution.coalesce_batches"; | |||
| pub const OPT_COALESCE_TARGET_BATCH_SIZE: &str = | |||
| "datafusion.execution.coalesce_target_batch_size"; | |||
|
|
|||
| /// Configuration option "datafusion.optimizer.skip_failed_rules" | |||
There was a problem hiding this comment.
💯 for it being a config setting
There was a problem hiding this comment.
Yeah I love this too! There are a few places I would want our product to fail in case of an error here but others I would not. so the configuration is great
Co-authored-by: Andrew Lamb <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>
andygrove
changed the title
|
Benchmark runs are scheduled for baseline = 034678b and contender = 8ad3df5. 8ad3df5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #2908
Partial fix for #2907
Rationale for this change
I often see queries fail due to bugs in individual optimizer rules. I would rather just skip the buggy optimizations rather than have the query fail.
What changes are included in this PR?
Check result from each rule and skip or fail depending on the configuration.
Are there any user-facing changes?
No