Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataFusion 18 docs #313

Merged
merged 1 commit into from
Feb 13, 2023
Merged

DataFusion 18 docs #313

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion datafusion/searchindex.js
View file
Open in desktop

Large diffs are not rendered by default.

36 changes: 28 additions & 8 deletions datafusion/user-guide/configs.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -410,42 +410,62 @@ <h1>Configuration Settings<a class="headerlink" href="#configuration-settings" t
<td><p>true</p></td>
<td><p>Should DataFusion repartition data using the aggregate keys to execute aggregates in parallel using the provided <code class="docutils literal notranslate"><span class="pre">target_partitions</span></code> level”</p></td>
</tr>
<tr class="row-odd"><td><p>datafusion.optimizer.repartition_joins</p></td>
<tr class="row-odd"><td><p>datafusion.optimizer.repartition_file_min_size</p></td>
<td><p>10485760</p></td>
<td><p>Minimum total files size in bytes to perform file scan repartitioning.</p></td>
</tr>
<tr class="row-even"><td><p>datafusion.optimizer.repartition_joins</p></td>
<td><p>true</p></td>
<td><p>Should DataFusion repartition data using the join keys to execute joins in parallel using the provided <code class="docutils literal notranslate"><span class="pre">target_partitions</span></code> level”</p></td>
</tr>
<tr class="row-odd"><td><p>datafusion.optimizer.repartition_file_scans</p></td>
<td><p>false</p></td>
<td><p>When set to true, file groups will be repartitioned to achieve maximum parallelism. Currently supported only for Parquet format in which case multiple row groups from the same file may be read concurrently. If false then each row group is read serially, though different files may be read in parallel.</p></td>
</tr>
<tr class="row-even"><td><p>datafusion.optimizer.repartition_windows</p></td>
<td><p>true</p></td>
<td><p>Should DataFusion repartition data using the partitions keys to execute window functions in parallel using the provided <code class="docutils literal notranslate"><span class="pre">target_partitions</span></code> level”</p></td>
</tr>
<tr class="row-odd"><td><p>datafusion.optimizer.skip_failed_rules</p></td>
<tr class="row-odd"><td><p>datafusion.optimizer.repartition_sorts</p></td>
<td><p>true</p></td>
<td><p>Should DataFusion execute sorts in a per-partition fashion and merge afterwards instead of coalescing first and sorting globally With this flag is enabled, plans in the form below “SortExec: [a&#64;0 ASC]”, ” CoalescePartitionsExec”, ” RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1”, would turn into the plan below which performs better in multithreaded environments “SortPreservingMergeExec: [a&#64;0 ASC]”, ” SortExec: [a&#64;0 ASC]”, ” RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1”,</p></td>
</tr>
<tr class="row-even"><td><p>datafusion.optimizer.skip_failed_rules</p></td>
<td><p>true</p></td>
<td><p>When set to true, the logical plan optimizer will produce warning messages if any optimization rules produce errors and then proceed to the next rule. When set to false, any rules that produce errors will cause the query to fail</p></td>
</tr>
<tr class="row-even"><td><p>datafusion.optimizer.max_passes</p></td>
<tr class="row-odd"><td><p>datafusion.optimizer.max_passes</p></td>
<td><p>3</p></td>
<td><p>Number of times that the optimizer will attempt to optimize the plan</p></td>
</tr>
<tr class="row-odd"><td><p>datafusion.optimizer.top_down_join_key_reordering</p></td>
<tr class="row-even"><td><p>datafusion.optimizer.top_down_join_key_reordering</p></td>
<td><p>true</p></td>
<td><p>When set to true, the physical plan optimizer will run a top down process to reorder the join keys</p></td>
</tr>
<tr class="row-even"><td><p>datafusion.optimizer.prefer_hash_join</p></td>
<tr class="row-odd"><td><p>datafusion.optimizer.prefer_hash_join</p></td>
<td><p>true</p></td>
<td><p>When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin. HashJoin can work more efficiently than SortMergeJoin but consumes more memory</p></td>
</tr>
<tr class="row-odd"><td><p>datafusion.optimizer.hash_join_single_partition_threshold</p></td>
<tr class="row-even"><td><p>datafusion.optimizer.hash_join_single_partition_threshold</p></td>
<td><p>1048576</p></td>
<td><p>The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition</p></td>
</tr>
<tr class="row-even"><td><p>datafusion.explain.logical_plan_only</p></td>
<tr class="row-odd"><td><p>datafusion.explain.logical_plan_only</p></td>
<td><p>false</p></td>
<td><p>When set to true, the explain statement will only print logical plans</p></td>
</tr>
<tr class="row-odd"><td><p>datafusion.explain.physical_plan_only</p></td>
<tr class="row-even"><td><p>datafusion.explain.physical_plan_only</p></td>
<td><p>false</p></td>
<td><p>When set to true, the explain statement will only print physical plans</p></td>
</tr>
<tr class="row-odd"><td><p>datafusion.sql_parser.parse_float_as_decimal</p></td>
<td><p>false</p></td>
<td><p>When set to true, sql parser will parse float as decimal type</p></td>
</tr>
<tr class="row-even"><td><p>datafusion.sql_parser.enable_ident_normalization</p></td>
<td><p>true</p></td>
<td><p>When set to true, sql parser will normalize ident(convert ident to lowercase when not quoted)</p></td>
</tr>
</tbody>
</table>
</section>
Expand Down
21 changes: 12 additions & 9 deletions datafusion/user-guide/dataframe.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,31 +380,34 @@ <h2>DataFrame Transformations<a class="headerlink" href="#dataframe-transformati
<tr class="row-odd"><td><p>join</p></td>
<td><p>Join this DataFrame with another DataFrame using the specified columns as join keys.</p></td>
</tr>
<tr class="row-even"><td><p>limit</p></td>
<tr class="row-even"><td><p>join_on</p></td>
<td><p>Join this DataFrame with another DataFrame using arbitrary expressions.</p></td>
</tr>
<tr class="row-odd"><td><p>limit</p></td>
<td><p>Limit the number of rows returned from this DataFrame.</p></td>
</tr>
<tr class="row-odd"><td><p>repartition</p></td>
<tr class="row-even"><td><p>repartition</p></td>
<td><p>Repartition a DataFrame based on a logical partitioning scheme.</p></td>
</tr>
<tr class="row-even"><td><p>sort</p></td>
<tr class="row-odd"><td><p>sort</p></td>
<td><p>Sort the DataFrame by the specified sorting expressions. Any expression can be turned into a sort expression by calling its <code class="docutils literal notranslate"><span class="pre">sort</span></code> method.</p></td>
</tr>
<tr class="row-odd"><td><p>select</p></td>
<tr class="row-even"><td><p>select</p></td>
<td><p>Create a projection based on arbitrary expressions. Example: <code class="docutils literal notranslate"><span class="pre">df..select(vec![col(&quot;c1&quot;),</span> <span class="pre">abs(col(&quot;c2&quot;))])?</span></code></p></td>
</tr>
<tr class="row-even"><td><p>select_columns</p></td>
<tr class="row-odd"><td><p>select_columns</p></td>
<td><p>Create a projection based on column names. Example: <code class="docutils literal notranslate"><span class="pre">df.select_columns(&amp;[&quot;id&quot;,</span> <span class="pre">&quot;name&quot;])?</span></code>.</p></td>
</tr>
<tr class="row-odd"><td><p>union</p></td>
<tr class="row-even"><td><p>union</p></td>
<td><p>Calculate the union of two DataFrames, preserving duplicate rows. The two DataFrames must have exactly the same schema.</p></td>
</tr>
<tr class="row-even"><td><p>union_distinct</p></td>
<tr class="row-odd"><td><p>union_distinct</p></td>
<td><p>Calculate the distinct union of two DataFrames. The two DataFrames must have exactly the same schema.</p></td>
</tr>
<tr class="row-odd"><td><p>with_column</p></td>
<tr class="row-even"><td><p>with_column</p></td>
<td><p>Add an additional column to the DataFrame.</p></td>
</tr>
<tr class="row-even"><td><p>with_column_renamed</p></td>
<tr class="row-odd"><td><p>with_column_renamed</p></td>
<td><p>Rename one column by applying a new projection.</p></td>
</tr>
</tbody>
Expand Down
4 changes: 2 additions & 2 deletions datafusion/user-guide/expressions.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -383,8 +383,8 @@ <h1>Expressions<a class="headerlink" href="#expressions" title="Permalink to thi
<p>DataFrame methods such as <code class="docutils literal notranslate"><span class="pre">select</span></code> and <code class="docutils literal notranslate"><span class="pre">filter</span></code> accept one or more logical expressions and there are many functions
available for creating logical expressions. These are documented below.</p>
<p>Expressions can be chained together using a fluent-style API:</p>
<div class="highlight-rust notranslate"><div class="highlight"><pre><span></span><span class="c1">// create the expression `(a &gt; 5) AND (b &lt; 7)`</span>
<span class="n">col</span><span class="p">(</span><span class="s">&quot;a&quot;</span><span class="p">).</span><span class="n">gt</span><span class="p">(</span><span class="n">lit</span><span class="p">(</span><span class="mi">5</span><span class="p">)).</span><span class="n">and</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s">&quot;b&quot;</span><span class="p">).</span><span class="n">lt</span><span class="p">(</span><span class="n">lit</span><span class="p">(</span><span class="mi">7</span><span class="p">)))</span><span class="w"></span>
<div class="highlight-rust notranslate"><div class="highlight"><pre><span></span><span class="c1">// create the expression `(a &gt; 6) AND (b &lt; 7)`</span>
<span class="n">col</span><span class="p">(</span><span class="s">&quot;a&quot;</span><span class="p">).</span><span class="n">gt</span><span class="p">(</span><span class="n">lit</span><span class="p">(</span><span class="mi">6</span><span class="p">)).</span><span class="n">and</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s">&quot;b&quot;</span><span class="p">).</span><span class="n">lt</span><span class="p">(</span><span class="n">lit</span><span class="p">(</span><span class="mi">7</span><span class="p">)))</span><span class="w"></span>
</pre></div>
</div>
<section id="identifiers">
Expand Down