From 1a37956c68c87883677264e6c1d21d427cf87acc Mon Sep 17 00:00:00 2001 From: Parth Chandra Date: Tue, 17 Dec 2024 10:59:09 -0800 Subject: [PATCH] fix: Document enabling comet explain plan usage in Spark (4.0) --- docs/source/user-guide/tuning.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/source/user-guide/tuning.md b/docs/source/user-guide/tuning.md index af722494f..f10a0dde9 100644 --- a/docs/source/user-guide/tuning.md +++ b/docs/source/user-guide/tuning.md @@ -127,3 +127,13 @@ Here is a guide to some of the native metrics. | `elapsed_compute` | Total time spent in this operator, fetching batches from a JVM iterator. | | `jvm_fetch_time` | Time spent in the JVM fetching input batches to be read by this `ScanExec` instance. | | `arrow_ffi_time` | Time spent using Arrow FFI to create Arrow batches from the memory addresses returned from the JVM. | + +## Explain Plan +### Extended Explain +With Spark 4.0.0 and newer, Comet can provide extended explain plan information in the Spark UI. Currently this lists +reasons why Comet may not have been enabled for specific operations. +To enable this, in the Spark configuration, set the following: +```shell +-c spark.sql.extendedExplainProviders=org.apache.comet.ExtendedExplainInfo +``` +This will add a section to the detailed plan displayed in the Spark SQL UI page. \ No newline at end of file