Support `EXPLAIN ... FORMAT <indent | tree | json | graphviz > ...` #15166

alamb · 2025-03-11T21:03:47Z

Which issue does this PR close?

Closes Support different EXPLAIN formats via SQL #15021
Closes Graphviz PhysicalPlan #219

Rationale for this change

I would like to be able to see more of the great explain plan code that DataFusion already has (see #15021 for details)

What changes are included in this PR?

Changes:

Here is an example

> explain format text select * from values(1) t(x);
Invalid or Unsupported Configuration: Invalid explain format: text
> explain format indent select * from values(1) t(x);
+---------------+-----------------------------------------------------+
| plan_type     | plan                                                |
+---------------+-----------------------------------------------------+
| logical_plan  | SubqueryAlias: t                                    |
|               |   Projection: column1 AS x                          |
|               |     Values: (Int64(1))                              |
| physical_plan | ProjectionExec: expr=[column1@0 as x]               |
|               |   DataSourceExec: partitions=1, partition_sizes=[1] |
|               |                                                     |
+---------------+-----------------------------------------------------+
2 row(s) fetched.
Elapsed 0.013 seconds.

> explain format tree select * from values(1) t(x);
+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │       ProjectionExec      │ |
|               | │    --------------------   │ |
|               | │        x: column1@0       │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │         bytes: 128        │ |
|               | │       format: memory      │ |
|               | │          rows: 1          │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+
1 row(s) fetched.
Elapsed 0.011 seconds.

Are these changes tested?

Are there any user-facing changes?

datafusion/sql/src/parser.rs

waynexia · 2025-03-12T18:11:10Z

datafusion/sql/src/statement.rs

                describe_alias: _,
                ..
            } => {
-                self.explain_to_plan(verbose, analyze, DFStatement::Statement(statement))
+                let format = format.map(|format| format.to_string());


Not very related to this PR, will we change sqlparser part to use a String instead of enum like here? (I'd prefer so, as it gives us more flexibility)

I think that would be a fine idea to propose upstream in sqlparser

I am not aware of any plans at the moment

waynexia · 2025-03-12T18:32:36Z

docs/source/user-guide/sql/explain.md


 <pre>
-EXPLAIN [ANALYZE] [VERBOSE] statement
+EXPLAIN [ANALYZE] [VERBOSE] [FORMAT format] statement


Maybe also state here that using VERBOSE and FORMAT together is not supported (yet?) like ANALYZE and FORMAT below

I will do so

I tried to clarify in 398cf1824

datafusion/sql/src/parser.rs

Co-authored-by: Ruihang Xia <[email protected]>

alamb · 2025-03-12T18:59:57Z

datafusion/physical-plan/src/display.rs

@@ -79,21 +78,6 @@ pub enum DisplayFormatType {
    TreeRender,
 }

-impl FromStr for DisplayFormatType {


This is superceded by the ExplainFormat

alamb · 2025-03-12T19:00:09Z

datafusion/sqllogictest/test_files/explain.slt

@@ -434,3 +434,135 @@ drop table t1;

 statement ok
 drop table t2;
+
+## Tests for explain format


Here are the tests showing how this works

alamb · 2025-03-12T20:08:36Z

FYI @irenjj

…o alamb/explan_format

irenjj · 2025-03-13T01:07:49Z

Thanks @alamb !
I did a quick local run. It looks like datafusion.explain.format and explain ... format [type] work together to determine the final output of EXPLAIN. datafusion.explain.format sets the default, while explain ... format [type] determines the final output. The explain ... format [type] option has a higher priority. Maybe we can mention this in explain.md.

irenjj · 2025-03-13T01:09:27Z

I'll take a look at others later today.❤️

irenjj · 2025-03-13T04:33:19Z

datafusion/core/src/physical_planner.rs

+                    displayable(optimized_plan.as_ref()).to_stringified(
+                        e.verbose,
+                        FinalPhysicalPlan,
+                        DisplayFormatType::TreeRender,
+                    ),


Maybe we can add a new function to handle tree explain like display_pg_json, then the third parameter is no longer needed as I found the values passed in below are all default values.

That is a good idea -- I will try it

I did it in a83ec4e and i think it looks much nicer ❤️

irenjj · 2025-03-13T04:37:28Z

Thanks @alamb , others LGTM.

alamb · 2025-03-13T20:10:43Z

Thanks @alamb ! I did a quick local run. It looks like datafusion.explain.format and explain ... format [type] work together to determine the final output of EXPLAIN. datafusion.explain.format sets the default, while explain ... format [type] determines the final output. The explain ... format [type] option has a higher priority. Maybe we can mention this in explain.md.

This is a good idea -- thank you. I did it in 31f402d

irenjj

LGTM, thanks again! @alamb and @waynexia

github-actions bot added sql SQL Planner logical-expr Logical plan and expressions core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Mar 11, 2025

alamb force-pushed the alamb/explan_format branch from d2e0d63 to aa53939 Compare March 12, 2025 15:54

alamb commented Mar 12, 2025

View reviewed changes

datafusion/sql/src/parser.rs Outdated Show resolved Hide resolved

alamb force-pushed the alamb/explan_format branch from aa53939 to b23e3cd Compare March 12, 2025 16:37

github-actions bot added the documentation Improvements or additions to documentation label Mar 12, 2025

alamb mentioned this pull request Mar 12, 2025

Minor: split EXPLAIN and ANALYZE planning into different functions #15188

Merged

alamb force-pushed the alamb/explan_format branch from b23e3cd to 187f38e Compare March 12, 2025 16:48

alamb mentioned this pull request Mar 12, 2025

Support different EXPLAIN formats via SQL #15021

Open

waynexia reviewed Mar 12, 2025

View reviewed changes

Support EXPLAIN FORMAT <format>

c965e8e

alamb force-pushed the alamb/explan_format branch from 187f38e to c965e8e Compare March 12, 2025 18:57

alamb marked this pull request as ready for review March 12, 2025 18:58

Update datafusion/sql/src/parser.rs

9a93ec1

Co-authored-by: Ruihang Xia <[email protected]>

alamb commented Mar 12, 2025

View reviewed changes

Improve documentation

398cf18

Merge branch 'alamb/explan_format' of github.com:alamb/datafusion int…

114bf4b

…o alamb/explan_format

irenjj reviewed Mar 13, 2025

View reviewed changes

alamb added 4 commits March 13, 2025 09:04

Remove to_stringified and simplify code

a83ec4e

mark deprecated rather than remove

f2cbf9d

Merge remote-tracking branch 'apache/main' into alamb/explan_format

fe08421

Add a note about explain format configuration

31f402d

irenjj approved these changes Mar 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support `EXPLAIN ... FORMAT <indent | tree | json | graphviz > ...` #15166

Support `EXPLAIN ... FORMAT <indent | tree | json | graphviz > ...` #15166

alamb commented Mar 11, 2025 •

edited

Loading

waynexia Mar 12, 2025

alamb Mar 12, 2025

waynexia Mar 12, 2025

alamb Mar 12, 2025

alamb Mar 12, 2025

alamb Mar 12, 2025

alamb Mar 12, 2025

alamb commented Mar 12, 2025

irenjj commented Mar 13, 2025

irenjj commented Mar 13, 2025

irenjj Mar 13, 2025

alamb Mar 13, 2025

alamb Mar 13, 2025

irenjj commented Mar 13, 2025

alamb commented Mar 13, 2025

irenjj left a comment

Support EXPLAIN ... FORMAT <indent | tree | json | graphviz > ... #15166

Are you sure you want to change the base?

Support EXPLAIN ... FORMAT <indent | tree | json | graphviz > ... #15166

Conversation

alamb commented Mar 11, 2025 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Mar 12, 2025

irenjj commented Mar 13, 2025

irenjj commented Mar 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

irenjj commented Mar 13, 2025

alamb commented Mar 13, 2025

irenjj left a comment

Choose a reason for hiding this comment

Support `EXPLAIN ... FORMAT <indent | tree | json | graphviz > ...` #15166

Support `EXPLAIN ... FORMAT <indent | tree | json | graphviz > ...` #15166

alamb commented Mar 11, 2025 •

edited

Loading