From 355f4f7fadffdd3a8d4e9636c54db02a279a6307 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 30 Nov 2022 19:28:53 +0800 Subject: [PATCH 1/6] explain-index-merge, optimizer-hints: introduce intersection type IndexMerge --- explain-index-merge.md | 84 ++++++++++++++++++++++++++++++++++-------- optimizer-hints.md | 10 ++--- system-variables.md | 9 +++++ 3 files changed, 82 insertions(+), 21 deletions(-) diff --git a/explain-index-merge.md b/explain-index-merge.md index 7378ce21952f6..ccb200ea946bf 100644 --- a/explain-index-merge.md +++ b/explain-index-merge.md @@ -7,39 +7,91 @@ summary: Learn about the execution plan information returned by the `EXPLAIN` st `IndexMerge` is a method introduced in TiDB v4.0 to access tables. Using this method, the TiDB optimizer can use multiple indexes per table and merge the results returned by each index. In some scenarios, this method makes the query more efficient by avoiding full table scans. +## Enable `IndexMerge` + +In v5.4.0 or a later TiDB version, `IndexMerge` is enabled by default. In other situations, if `IndexMerge` is not enabled, you need to set the variable [`tidb_enable_index_merge`](/system-variables.md#tidb_enable_index_merge-new-in-v40) to `ON` to enable this feature. + +The `IndexMerge` in TiDB has two types: the intersection type and the union type. The former is suitable for the `AND` expression, while the latter is suitable for the `OR` expression. The union type `IndexMerge` is introduced in TiDB v4.0. The intersection type is introduced in TiDB v6.5.0, and can only be used when the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint is specified. + +```sql +SET session tidb_enable_index_merge = ON; +``` + ```sql -mysql> EXPLAIN SELECT * from t where a = 1 or b = 1; +CREATE TABLE t(a int, b int, c int, d int, INDEX idx_a(a), INDEX idx_b(b), INDEX idx_c(c), INDEX idx_d(d)); +``` + +```sql +EXPLAIN SELECT /*+ NO_INDEX_MERGE() */ * FROM t WHERE a = 1 OR b = 1; + +-------------------------+----------+-----------+---------------+--------------------------------------+ | id | estRows | task | access object | operator info | +-------------------------+----------+-----------+---------------+--------------------------------------+ -| TableReader_7 | 8000.00 | root | | data:Selection_6 | -| └─Selection_6 | 8000.00 | cop[tikv] | | or(eq(test.t.a, 1), eq(test.t.b, 1)) | +| TableReader_7 | 19.99 | root | | data:Selection_6 | +| └─Selection_6 | 19.99 | cop[tikv] | | or(eq(test.t.a, 1), eq(test.t.b, 1)) | | └─TableFullScan_5 | 10000.00 | cop[tikv] | table:t | keep order:false, stats:pseudo | +-------------------------+----------+-----------+---------------+--------------------------------------+ -mysql> set @@tidb_enable_index_merge = 1; -mysql> explain select * from t use index(idx_a, idx_b) where a > 1 or b > 1; -+--------------------------------+---------+-----------+-------------------------+------------------------------------------------+ -| id | estRows | task | access object | operator info | -+--------------------------------+---------+-----------+-------------------------+------------------------------------------------+ -| IndexMerge_16 | 6666.67 | root | | | -| ├─IndexRangeScan_13(Build) | 3333.33 | cop[tikv] | table:t, index:idx_a(a) | range:(1,+inf], keep order:false, stats:pseudo | -| ├─IndexRangeScan_14(Build) | 3333.33 | cop[tikv] | table:t, index:idx_b(b) | range:(1,+inf], keep order:false, stats:pseudo | -| └─TableRowIDScan_15(Probe) | 6666.67 | cop[tikv] | table:t | keep order:false, stats:pseudo | -+--------------------------------+---------+-----------+-------------------------+------------------------------------------------+ +EXPLAIN SELECT /*+ USE_INDEX_MERGE(t) */ * FROM t WHERE a > 1 OR b > 1; ++-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ +| id | estRows | task | access object | operator info | ++-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ +| IndexMerge_8 | 5555.56 | root | | type: union | +| ├─IndexRangeScan_5(Build) | 3333.33 | cop[tikv] | table:t, index:idx_a(a) | range:(1,+inf], keep order:false, stats:pseudo | +| ├─IndexRangeScan_6(Build) | 3333.33 | cop[tikv] | table:t, index:idx_b(b) | range:(1,+inf], keep order:false, stats:pseudo | +| └─TableRowIDScan_7(Probe) | 5555.56 | cop[tikv] | table:t | keep order:false, stats:pseudo | ++-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ ``` -In the above query, the filter condition is a `WHERE` clause that uses `OR` as the connector. Without `IndexMerge`, you can use only one index per table. `a = 1` cannot be pushed down to the index `a`; neither can `b = 1` be pushed down to the index `b`. The full table scan is inefficient when a huge volume of data exists in `t`. To handle such a scenario, `IndexMerge` is introduced in TiDB to access tables. +In the preceding query, the filter condition is a `WHERE` clause that uses `OR` as the connector. Without `IndexMerge`, you can use only one index per table. `a = 1` cannot be pushed down to the index `a`; neither can `b = 1` be pushed down to the index `b`. The full table scan is inefficient when a huge volume of data exists in `t`. To handle such a scenario, `IndexMerge` is introduced in TiDB to access tables. + +For the preceding query, the optimizer chooses the intersection type `IndexMerge` to access the table.`IndexMerge` allows the optimizer to use multiple indexes per table, and merge the results returned by each index to generate the execution plan of the latter `IndexMerge` in the figure above. -`IndexMerge` allows the optimizer to use multiple indexes per table, and merge the results returned by each index to generate the execution plan of the latter `IndexMerge` in the figure above. Here the `IndexMerge_16` operator has three child nodes, among which `IndexRangeScan_13` and `IndexRangeScan_14` get all the `RowID`s that meet the conditions based on the result of range scan, and then the `TableRowIDScan_15` operator accurately reads all the data that meets the conditions according to these `RowID`s. +At this time, the `type: union` information in `operator info` of the `IndexMerge_8` operator indicates that this operator is a union type `IndexMerge`. It has three child nodes. `IndexRangeScan_5` and `IndexRangeScan_6` scan the `RowID`s that meet the condition according to the range, and then the `TableRowIDScan_7` operator reads all the data that meets the condition according to these `RowID`s. For the scan operation that is performed on a specific range of data, such as `IndexRangeScan`/`TableRangeScan`, the `operator info` column in the result has additional information about the scan range compared with other scan operations like `IndexFullScan`/`TableFullScan`. In the above example, the `range:(1,+inf]` in the `IndexRangeScan_13` operator indicates that the operator scans the data from 1 to positive infinity. +```sql +EXPLAIN SELECT /*+ NO_INDEX_MERGE() */ * FROM t WHERE a > 1 AND b > 1 AND c = 1; -- Does not use IndexMerge + ++--------------------------------+---------+-----------+-------------------------+---------------------------------------------+ +| id | estRows | task | access object | operator info | ++--------------------------------+---------+-----------+-------------------------+---------------------------------------------+ +| IndexLookUp_19 | 1.11 | root | | | +| ├─IndexRangeScan_16(Build) | 10.00 | cop[tikv] | table:t, index:idx_c(c) | range:[1,1], keep order:false, stats:pseudo | +| └─Selection_18(Probe) | 1.11 | cop[tikv] | | gt(test.t.a, 1), gt(test.t.b, 1) | +| └─TableRowIDScan_17 | 10.00 | cop[tikv] | table:t | keep order:false, stats:pseudo | ++--------------------------------+---------+-----------+-------------------------+---------------------------------------------+ + +EXPLAIN SELECT /*+ USE_INDEX_MERGE(t, idx_a, idx_b, idx_c) */ * FROM t WHERE a > 1 AND b > 1 AND c = 1; -- Uses IndexMerge ++-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ +| id | estRows | task | access object | operator info | ++-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ +| IndexMerge_9 | 1.11 | root | | type: intersection | +| ├─IndexRangeScan_5(Build) | 3333.33 | cop[tikv] | table:t, index:idx_a(a) | range:(1,+inf], keep order:false, stats:pseudo | +| ├─IndexRangeScan_6(Build) | 3333.33 | cop[tikv] | table:t, index:idx_b(b) | range:(1,+inf], keep order:false, stats:pseudo | +| ├─IndexRangeScan_7(Build) | 10.00 | cop[tikv] | table:t, index:idx_c(c) | range:[1,1], keep order:false, stats:pseudo | +| └─TableRowIDScan_8(Probe) | 1.11 | cop[tikv] | table:t | keep order:false, stats:pseudo | ++-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ +``` + +From the preceding example, you can see that the filter condition is a `WHERE` clause that uses `AND` as the connector. Before `IndexMerge` is enabled, you can only choose one of the three indexes `idx_a`, `idx_b`, and `idx_c` to use. + +If one of the filter conditions has a very good filtering performance, you can directly select the corresponding index to achieve the ideal execution efficiency. However, if the data distribution meets all of the following three conditions, you can consider using the intersection type `IndexMerge`: + +- The data size of the full table is large, and directly reading the full table is inefficient. +- For each one of the three filter conditions, the respective filtering performance is not good enough, so the execution efficiency of `IndexLookUp` using a single index is not ideal. +- The overall filtering performance of the three filter conditions is good. + +When using the intersection type `IndexMerge` to access table, the optimizer can choose to use multiple indexes on a table, and merge the results returned by each index to generate the execution plan of the latter `IndexMerge` in the preceding example. The `type: intersection` in the `operator info` of the `IndexMerge_9` operator indicates that this operator is an intersection type `IndexMerge`. The other parts of the execution plan are similar to the preceding union type `IndexMerge` example. + > **Note:** > > - The Index Merge feature is enabled by default from v5.4.0. That is, [`tidb_enable_index_merge`](/system-variables.md#tidb_enable_index_merge-new-in-v40) is `ON`. > > - You can use the SQL hint [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) to force the optimizer to apply Index Merge, regardless of the setting of `tidb_enable_index_merge`. To enable Index Merge when the filtering conditions contain expressions that cannot be pushed down, you must use the SQL hint [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-). > -> - Index Merge supports only disjunctive normal form (expressions connected by `or`) and does not support conjunctive normal form (expressions connected by `and`). +> - If there is a single index scan method (other than full table scan) can be selected for a query plan, the optimizer will not automatically select `IndexMerge`. You can only use the optimizer hint to specify using `IndexMerge`. > > - Index Merge is not supported in [tempoaray tables](/temporary-tables.md) for now. +> +> - The intersection type `IndexMerge` will not automatically be selected by the optimizer. You must specify the **table name and index name** using the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint for it to be selected. diff --git a/optimizer-hints.md b/optimizer-hints.md index 483a01c7a269d..d9fe12cae1b56 100644 --- a/optimizer-hints.md +++ b/optimizer-hints.md @@ -368,7 +368,11 @@ select /*+ READ_FROM_STORAGE(TIFLASH[t1], TIKV[t2]) */ t1.a from t t1, t t2 wher ### USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...]) -The `USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...])` hint tells the optimizer to access a specific table with the index merge method. The given list of indexes are optional parameters. If you explicitly specify the list, TiDB selects indexes from the list to build index merge; if you do not give the list of indexes, TiDB selects indexes from all available indexes to build index merge. For example: +The `USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...])` hint tells the optimizer to access a specific table with the index merge method. Index merge has two types: intersection type and union type. For details, see [Explain Statements Using Index Merge](/explain-index-merge.md) + +For the union type index merge, the given list of indexes are optional parameters in the hint. If you explicitly specify the list, TiDB selects indexes from the list to build index merge; if you do not give the list of indexes, TiDB selects indexes from all available indexes to build index merge. See the following example. + +For the intersection type index merge, the given list of indexes are required parameters in the hint. {{< copyable "sql" >}} @@ -382,10 +386,6 @@ When multiple `USE_INDEX_MERGE` hints are made to the same table, the optimizer > > The parameters of `USE_INDEX_MERGE` refer to index names, rather than column names. The index name of the primary key is `primary`. -This hint takes effect on strict conditions, including: - -- If the query can select a single index scan in addition to full table scan, the optimizer does not select index merge. - ### LEADING(t1_name [, tl_name ...]) The `LEADING(t1_name [, tl_name ...])` hint reminds the optimizer that, when generating the execution plan, to determine the order of multi-table joins according to the order of table names specified in the hint. For example: diff --git a/system-variables.md b/system-variables.md index 9d34d1fa3c864..da0ef29490c27 100644 --- a/system-variables.md +++ b/system-variables.md @@ -2295,6 +2295,15 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - This variable is used to set the concurrency of the `index lookup join` algorithm. - A value of `-1` means that the value of `tidb_executor_concurrency` will be used instead. +### tidb_index_merge_intersection_concurrency New in v6.5.0 + +- Scope: SESSION | GLOBAL +- Persists to cluster: Yes +- Default value: `-1` +- Range: `[1, 256]` +- This variable sets the maximum concurrency of the intersection type IndexMerge operations. It is effective only when TiDB accesses partitioned tables in the dynamic pruning mode. The actual concurrency is the smaller value of `tidb_index_merge_intersection_concurrency` and the number of partitions of the partitioned table. +- The default value `-1` means that the value of `tidb_executor_concurrency` is used. + ### tidb_index_lookup_size - Scope: SESSION | GLOBAL From 0dbd0c4120eae4709b2945935714f8360119db3b Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Thu, 1 Dec 2022 14:29:11 +0800 Subject: [PATCH 2/6] refine --- explain-index-merge.md | 34 ++++++++++++++++++---------------- optimizer-hints.md | 6 +++--- system-variables.md | 2 +- 3 files changed, 22 insertions(+), 20 deletions(-) diff --git a/explain-index-merge.md b/explain-index-merge.md index ccb200ea946bf..22c7599464224 100644 --- a/explain-index-merge.md +++ b/explain-index-merge.md @@ -5,18 +5,20 @@ summary: Learn about the execution plan information returned by the `EXPLAIN` st # Explain Statements Using Index Merge -`IndexMerge` is a method introduced in TiDB v4.0 to access tables. Using this method, the TiDB optimizer can use multiple indexes per table and merge the results returned by each index. In some scenarios, this method makes the query more efficient by avoiding full table scans. +Index merge is a method introduced in TiDB v4.0 to access tables. Using this method, the TiDB optimizer can use multiple indexes per table and merge the results returned by each index. In some scenarios, this method makes the query more efficient by avoiding full table scans. -## Enable `IndexMerge` +Index merge in TiDB has two types: the intersection type and the union type. The former is suitable for the `AND` expression, while the latter is suitable for the `OR` expression. The union-type index merge is introduced in TiDB v4.0 as an experimental feature and has become GA in v5.4.0. The intersection type is introduced in TiDB v6.5.0, and can be used only when the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint is specified. -In v5.4.0 or a later TiDB version, `IndexMerge` is enabled by default. In other situations, if `IndexMerge` is not enabled, you need to set the variable [`tidb_enable_index_merge`](/system-variables.md#tidb_enable_index_merge-new-in-v40) to `ON` to enable this feature. +## Enable index merge -The `IndexMerge` in TiDB has two types: the intersection type and the union type. The former is suitable for the `AND` expression, while the latter is suitable for the `OR` expression. The union type `IndexMerge` is introduced in TiDB v4.0. The intersection type is introduced in TiDB v6.5.0, and can only be used when the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint is specified. +In v5.4.0 or a later TiDB version, index merge is enabled by default. In other situations, if index merge is not enabled, you need to set the variable [`tidb_enable_index_merge`](/system-variables.md#tidb_enable_index_merge-new-in-v40) to `ON` to enable this feature. ```sql SET session tidb_enable_index_merge = ON; ``` +## Examples + ```sql CREATE TABLE t(a int, b int, c int, d int, INDEX idx_a(a), INDEX idx_b(b), INDEX idx_c(c), INDEX idx_d(d)); ``` @@ -42,16 +44,16 @@ EXPLAIN SELECT /*+ USE_INDEX_MERGE(t) */ * FROM t WHERE a > 1 OR b > 1; +-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ ``` -In the preceding query, the filter condition is a `WHERE` clause that uses `OR` as the connector. Without `IndexMerge`, you can use only one index per table. `a = 1` cannot be pushed down to the index `a`; neither can `b = 1` be pushed down to the index `b`. The full table scan is inefficient when a huge volume of data exists in `t`. To handle such a scenario, `IndexMerge` is introduced in TiDB to access tables. +In the preceding query, the filter condition is a `WHERE` clause that uses `OR` as the connector. Without index merge, you can use only one index per table. `a = 1` cannot be pushed down to the index `a`; neither can `b = 1` be pushed down to the index `b`. The full table scan is inefficient when a huge volume of data exists in `t`. To handle such a scenario, index merge is introduced in TiDB to access tables. -For the preceding query, the optimizer chooses the intersection type `IndexMerge` to access the table.`IndexMerge` allows the optimizer to use multiple indexes per table, and merge the results returned by each index to generate the execution plan of the latter `IndexMerge` in the figure above. +For the preceding query, the optimizer chooses the intersection-type index merge to access the table. Index merge allows the optimizer to use multiple indexes per table, to merge the results returned by each index and to generate the latter execution plan in the preceding output. -At this time, the `type: union` information in `operator info` of the `IndexMerge_8` operator indicates that this operator is a union type `IndexMerge`. It has three child nodes. `IndexRangeScan_5` and `IndexRangeScan_6` scan the `RowID`s that meet the condition according to the range, and then the `TableRowIDScan_7` operator reads all the data that meets the condition according to these `RowID`s. +In the output, the `type: union` information in `operator info` of the `IndexMerge_8` operator indicates that this operator is a union-type index merge operator. It has three child nodes. `IndexRangeScan_5` and `IndexRangeScan_6` scan the `RowID`s that meet the condition according to the range, and then the `TableRowIDScan_7` operator accurately reads all the data that meets the condition according to these `RowID`s. For the scan operation that is performed on a specific range of data, such as `IndexRangeScan`/`TableRangeScan`, the `operator info` column in the result has additional information about the scan range compared with other scan operations like `IndexFullScan`/`TableFullScan`. In the above example, the `range:(1,+inf]` in the `IndexRangeScan_13` operator indicates that the operator scans the data from 1 to positive infinity. ```sql -EXPLAIN SELECT /*+ NO_INDEX_MERGE() */ * FROM t WHERE a > 1 AND b > 1 AND c = 1; -- Does not use IndexMerge +EXPLAIN SELECT /*+ NO_INDEX_MERGE() */ * FROM t WHERE a > 1 AND b > 1 AND c = 1; -- Does not use index merge +--------------------------------+---------+-----------+-------------------------+---------------------------------------------+ | id | estRows | task | access object | operator info | @@ -62,7 +64,7 @@ EXPLAIN SELECT /*+ NO_INDEX_MERGE() */ * FROM t WHERE a > 1 AND b > 1 AND c = 1; | └─TableRowIDScan_17 | 10.00 | cop[tikv] | table:t | keep order:false, stats:pseudo | +--------------------------------+---------+-----------+-------------------------+---------------------------------------------+ -EXPLAIN SELECT /*+ USE_INDEX_MERGE(t, idx_a, idx_b, idx_c) */ * FROM t WHERE a > 1 AND b > 1 AND c = 1; -- Uses IndexMerge +EXPLAIN SELECT /*+ USE_INDEX_MERGE(t, idx_a, idx_b, idx_c) */ * FROM t WHERE a > 1 AND b > 1 AND c = 1; -- Uses index merge +-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ | id | estRows | task | access object | operator info | +-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ @@ -74,15 +76,15 @@ EXPLAIN SELECT /*+ USE_INDEX_MERGE(t, idx_a, idx_b, idx_c) */ * FROM t WHERE a > +-------------------------------+---------+-----------+-------------------------+------------------------------------------------+ ``` -From the preceding example, you can see that the filter condition is a `WHERE` clause that uses `AND` as the connector. Before `IndexMerge` is enabled, you can only choose one of the three indexes `idx_a`, `idx_b`, and `idx_c` to use. +From the preceding example, you can see that the filter condition is a `WHERE` clause that uses `AND` as the connector. Before index merge is enabled, the optimizer can only choose one of the three indexes (`idx_a`, `idx_b`, or `idx_c`). -If one of the filter conditions has a very good filtering performance, you can directly select the corresponding index to achieve the ideal execution efficiency. However, if the data distribution meets all of the following three conditions, you can consider using the intersection type `IndexMerge`: +If one of the filter conditions has a good filtering performance, the optimizer directly chooses the corresponding index to achieve the ideal execution efficiency. However, if the data distribution meets all of the following three conditions, the optimizer considers using the intersection-type index merge: -- The data size of the full table is large, and directly reading the full table is inefficient. -- For each one of the three filter conditions, the respective filtering performance is not good enough, so the execution efficiency of `IndexLookUp` using a single index is not ideal. +- The data size of the whole table is large, and directly reading the whole table is inefficient. +- For each one of the three filter conditions, the respective filtering performance is not poor, so the execution efficiency of `IndexLookUp` using a single index is not ideal. - The overall filtering performance of the three filter conditions is good. -When using the intersection type `IndexMerge` to access table, the optimizer can choose to use multiple indexes on a table, and merge the results returned by each index to generate the execution plan of the latter `IndexMerge` in the preceding example. The `type: intersection` in the `operator info` of the `IndexMerge_9` operator indicates that this operator is an intersection type `IndexMerge`. The other parts of the execution plan are similar to the preceding union type `IndexMerge` example. +When using the intersection-type index merge to access table, the optimizer can choose to use multiple indexes on a table, and merge the results returned by each index to generate the execution plan of the latter `IndexMerge` in the preceding example output. The `type: intersection` information in the `operator info` of the `IndexMerge_9` operator indicates that this operator is an intersection-type index merge. The other parts of the execution plan are similar to the preceding union-type index merge example. > **Note:** > @@ -90,8 +92,8 @@ When using the intersection type `IndexMerge` to access table, the optimizer can > > - You can use the SQL hint [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) to force the optimizer to apply Index Merge, regardless of the setting of `tidb_enable_index_merge`. To enable Index Merge when the filtering conditions contain expressions that cannot be pushed down, you must use the SQL hint [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-). > -> - If there is a single index scan method (other than full table scan) can be selected for a query plan, the optimizer will not automatically select `IndexMerge`. You can only use the optimizer hint to specify using `IndexMerge`. +> - If the optimizer can choose the single index scan method (other than full table scan) for a query plan, the optimizer will not automatically select `IndexMerge`. For the optimizer to use `IndexMerge`, you need to use the optimizer hint. > > - Index Merge is not supported in [tempoaray tables](/temporary-tables.md) for now. > -> - The intersection type `IndexMerge` will not automatically be selected by the optimizer. You must specify the **table name and index name** using the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint for it to be selected. +> - The intersection-type index merge will not automatically be selected by the optimizer. You must specify the **table name and index name** using the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint for it to be selected. diff --git a/optimizer-hints.md b/optimizer-hints.md index d9fe12cae1b56..a203333c22a66 100644 --- a/optimizer-hints.md +++ b/optimizer-hints.md @@ -368,11 +368,11 @@ select /*+ READ_FROM_STORAGE(TIFLASH[t1], TIKV[t2]) */ t1.a from t t1, t t2 wher ### USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...]) -The `USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...])` hint tells the optimizer to access a specific table with the index merge method. Index merge has two types: intersection type and union type. For details, see [Explain Statements Using Index Merge](/explain-index-merge.md) +The `USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...])` hint tells the optimizer to access a specific table with the index merge method. Index merge has two types: intersection type and union type. For details, see [Explain Statements Using Index Merge](/explain-index-merge.md). -For the union type index merge, the given list of indexes are optional parameters in the hint. If you explicitly specify the list, TiDB selects indexes from the list to build index merge; if you do not give the list of indexes, TiDB selects indexes from all available indexes to build index merge. See the following example. +For the union-type index merge, the given list of indexes are optional parameters in the hint. If you explicitly specify the list, TiDB selects indexes from the list to build index merge; if you do not give the list of indexes, TiDB selects indexes from all available indexes to build index merge. See the following example. -For the intersection type index merge, the given list of indexes are required parameters in the hint. +For the intersection-type index merge, the given list of indexes are required parameters in the hint. {{< copyable "sql" >}} diff --git a/system-variables.md b/system-variables.md index da0ef29490c27..b8d0dae883527 100644 --- a/system-variables.md +++ b/system-variables.md @@ -2301,7 +2301,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Persists to cluster: Yes - Default value: `-1` - Range: `[1, 256]` -- This variable sets the maximum concurrency of the intersection type IndexMerge operations. It is effective only when TiDB accesses partitioned tables in the dynamic pruning mode. The actual concurrency is the smaller value of `tidb_index_merge_intersection_concurrency` and the number of partitions of the partitioned table. +- This variable sets the maximum concurrency for the intersection operations that `IndexMerge` performs. It is effective only when TiDB accesses partitioned tables in the dynamic pruning mode. The actual concurrency is the smaller value of `tidb_index_merge_intersection_concurrency` and the number of partitions of the partitioned table. - The default value `-1` means that the value of `tidb_executor_concurrency` is used. ### tidb_index_lookup_size From 9c95ace361d6b96cf266aaaa411d03580b957682 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 5 Dec 2022 17:12:55 +0800 Subject: [PATCH 3/6] Apply suggestions from code review Co-authored-by: Aolin Co-authored-by: Zhou Kunqin <25057648+time-and-fate@users.noreply.github.com> --- explain-index-merge.md | 16 ++++++++-------- system-variables.md | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/explain-index-merge.md b/explain-index-merge.md index 22c7599464224..fdff1e638a036 100644 --- a/explain-index-merge.md +++ b/explain-index-merge.md @@ -7,7 +7,7 @@ summary: Learn about the execution plan information returned by the `EXPLAIN` st Index merge is a method introduced in TiDB v4.0 to access tables. Using this method, the TiDB optimizer can use multiple indexes per table and merge the results returned by each index. In some scenarios, this method makes the query more efficient by avoiding full table scans. -Index merge in TiDB has two types: the intersection type and the union type. The former is suitable for the `AND` expression, while the latter is suitable for the `OR` expression. The union-type index merge is introduced in TiDB v4.0 as an experimental feature and has become GA in v5.4.0. The intersection type is introduced in TiDB v6.5.0, and can be used only when the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint is specified. +Index merge in TiDB has two types: the intersection type and the union type. The former is applied to the `AND` expression, while the latter is applied to the `OR` expression. The union-type index merge is introduced in TiDB v4.0 as an experimental feature and has become GA in v5.4.0. The intersection type is introduced in TiDB v6.5.0, and can be used only when the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint is specified. ## Enable index merge @@ -24,7 +24,7 @@ CREATE TABLE t(a int, b int, c int, d int, INDEX idx_a(a), INDEX idx_b(b), INDEX ``` ```sql -EXPLAIN SELECT /*+ NO_INDEX_MERGE() */ * FROM t WHERE a = 1 OR b = 1; +EXPLAIN SELECT /*+ NO_INDEX_MERGE() */ * FROM t WHERE a = 1 OR b = 1; +-------------------------+----------+-----------+---------------+--------------------------------------+ | id | estRows | task | access object | operator info | @@ -46,7 +46,7 @@ EXPLAIN SELECT /*+ USE_INDEX_MERGE(t) */ * FROM t WHERE a > 1 OR b > 1; In the preceding query, the filter condition is a `WHERE` clause that uses `OR` as the connector. Without index merge, you can use only one index per table. `a = 1` cannot be pushed down to the index `a`; neither can `b = 1` be pushed down to the index `b`. The full table scan is inefficient when a huge volume of data exists in `t`. To handle such a scenario, index merge is introduced in TiDB to access tables. -For the preceding query, the optimizer chooses the intersection-type index merge to access the table. Index merge allows the optimizer to use multiple indexes per table, to merge the results returned by each index and to generate the latter execution plan in the preceding output. +For the preceding query, the optimizer chooses the union-type index merge to access the table. Index merge allows the optimizer to use multiple indexes per table, to merge the results returned by each index, and to generate the latter execution plan in the preceding output. In the output, the `type: union` information in `operator info` of the `IndexMerge_8` operator indicates that this operator is a union-type index merge operator. It has three child nodes. `IndexRangeScan_5` and `IndexRangeScan_6` scan the `RowID`s that meet the condition according to the range, and then the `TableRowIDScan_7` operator accurately reads all the data that meets the condition according to these `RowID`s. @@ -78,13 +78,13 @@ EXPLAIN SELECT /*+ USE_INDEX_MERGE(t, idx_a, idx_b, idx_c) */ * FROM t WHERE a > From the preceding example, you can see that the filter condition is a `WHERE` clause that uses `AND` as the connector. Before index merge is enabled, the optimizer can only choose one of the three indexes (`idx_a`, `idx_b`, or `idx_c`). -If one of the filter conditions has a good filtering performance, the optimizer directly chooses the corresponding index to achieve the ideal execution efficiency. However, if the data distribution meets all of the following three conditions, the optimizer considers using the intersection-type index merge: +If one of the filter conditions has a low selectivity, the optimizer directly chooses the corresponding index to achieve the ideal execution efficiency. However, if the data distribution meets all of the following three conditions, you can consider using the intersection-type index merge: - The data size of the whole table is large, and directly reading the whole table is inefficient. -- For each one of the three filter conditions, the respective filtering performance is not poor, so the execution efficiency of `IndexLookUp` using a single index is not ideal. -- The overall filtering performance of the three filter conditions is good. +- For each one of the three filter conditions, the respective selectivity is very high, so the execution efficiency of `IndexLookUp` using a single index is not ideal. +- The overall selectivity of the three filter conditions is low. -When using the intersection-type index merge to access table, the optimizer can choose to use multiple indexes on a table, and merge the results returned by each index to generate the execution plan of the latter `IndexMerge` in the preceding example output. The `type: intersection` information in the `operator info` of the `IndexMerge_9` operator indicates that this operator is an intersection-type index merge. The other parts of the execution plan are similar to the preceding union-type index merge example. +When using the intersection-type index merge to access tables, the optimizer can choose to use multiple indexes on a table, and merge the results returned by each index to generate the execution plan of the latter `IndexMerge` in the preceding example output. The `type: intersection` information in the `operator info` of the `IndexMerge_9` operator indicates that this operator is an intersection-type index merge. The other parts of the execution plan are similar to the preceding union-type index merge example. > **Note:** > @@ -92,7 +92,7 @@ When using the intersection-type index merge to access table, the optimizer can > > - You can use the SQL hint [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) to force the optimizer to apply Index Merge, regardless of the setting of `tidb_enable_index_merge`. To enable Index Merge when the filtering conditions contain expressions that cannot be pushed down, you must use the SQL hint [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-). > -> - If the optimizer can choose the single index scan method (other than full table scan) for a query plan, the optimizer will not automatically select `IndexMerge`. For the optimizer to use `IndexMerge`, you need to use the optimizer hint. +> - If the optimizer can choose the single index scan method (other than full table scan) for a query plan, the optimizer will not automatically use index merge. For the optimizer to use index merge, you need to use the optimizer hint. > > - Index Merge is not supported in [tempoaray tables](/temporary-tables.md) for now. > diff --git a/system-variables.md b/system-variables.md index b8d0dae883527..331d069ca77fc 100644 --- a/system-variables.md +++ b/system-variables.md @@ -2301,7 +2301,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - Persists to cluster: Yes - Default value: `-1` - Range: `[1, 256]` -- This variable sets the maximum concurrency for the intersection operations that `IndexMerge` performs. It is effective only when TiDB accesses partitioned tables in the dynamic pruning mode. The actual concurrency is the smaller value of `tidb_index_merge_intersection_concurrency` and the number of partitions of the partitioned table. +- This variable sets the maximum concurrency for the intersection operations that index merge performs. It is effective only when TiDB accesses partitioned tables in the dynamic pruning mode. The actual concurrency is the smaller value of `tidb_index_merge_intersection_concurrency` and the number of partitions of the partitioned table. - The default value `-1` means that the value of `tidb_executor_concurrency` is used. ### tidb_index_lookup_size From d9116b9d58c7a4c2b7cb4b16505d5a10a6ac5a0b Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 5 Dec 2022 17:46:56 +0800 Subject: [PATCH 4/6] Update explain-index-merge.md Co-authored-by: Aolin --- explain-index-merge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/explain-index-merge.md b/explain-index-merge.md index fdff1e638a036..ca81f80a6972e 100644 --- a/explain-index-merge.md +++ b/explain-index-merge.md @@ -48,7 +48,7 @@ In the preceding query, the filter condition is a `WHERE` clause that uses `OR` For the preceding query, the optimizer chooses the union-type index merge to access the table. Index merge allows the optimizer to use multiple indexes per table, to merge the results returned by each index, and to generate the latter execution plan in the preceding output. -In the output, the `type: union` information in `operator info` of the `IndexMerge_8` operator indicates that this operator is a union-type index merge operator. It has three child nodes. `IndexRangeScan_5` and `IndexRangeScan_6` scan the `RowID`s that meet the condition according to the range, and then the `TableRowIDScan_7` operator accurately reads all the data that meets the condition according to these `RowID`s. +In the output, the `type: union` information in `operator info` of the `IndexMerge_8` operator indicates that this operator is a union-type index merge. It has three child nodes. `IndexRangeScan_5` and `IndexRangeScan_6` scan the `RowID`s that meet the condition according to the range, and then the `TableRowIDScan_7` operator accurately reads all the data that meets the condition according to these `RowID`s. For the scan operation that is performed on a specific range of data, such as `IndexRangeScan`/`TableRangeScan`, the `operator info` column in the result has additional information about the scan range compared with other scan operations like `IndexFullScan`/`TableFullScan`. In the above example, the `range:(1,+inf]` in the `IndexRangeScan_13` operator indicates that the operator scans the data from 1 to positive infinity. From 6b1bfceee9ba1ce8032f3c19fed95bf85ae5ce95 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 6 Dec 2022 11:37:27 +0800 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Zhou Kunqin <25057648+time-and-fate@users.noreply.github.com> --- explain-index-merge.md | 2 +- optimizer-hints.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/explain-index-merge.md b/explain-index-merge.md index ca81f80a6972e..6d2456868d8a3 100644 --- a/explain-index-merge.md +++ b/explain-index-merge.md @@ -7,7 +7,7 @@ summary: Learn about the execution plan information returned by the `EXPLAIN` st Index merge is a method introduced in TiDB v4.0 to access tables. Using this method, the TiDB optimizer can use multiple indexes per table and merge the results returned by each index. In some scenarios, this method makes the query more efficient by avoiding full table scans. -Index merge in TiDB has two types: the intersection type and the union type. The former is applied to the `AND` expression, while the latter is applied to the `OR` expression. The union-type index merge is introduced in TiDB v4.0 as an experimental feature and has become GA in v5.4.0. The intersection type is introduced in TiDB v6.5.0, and can be used only when the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint is specified. +Index merge in TiDB has two types: the intersection type and the union type. The former applies to the `AND` expression, while the latter applies to the `OR` expression. The union-type index merge is introduced in TiDB v4.0 as an experimental feature and has become GA in v5.4.0. The intersection type is introduced in TiDB v6.5.0, and can be used only when the [`USE_INDEX_MERGE`](/optimizer-hints.md#use_index_merget1_name-idx1_name--idx2_name-) hint is specified. ## Enable index merge diff --git a/optimizer-hints.md b/optimizer-hints.md index a203333c22a66..b44c2785cc541 100644 --- a/optimizer-hints.md +++ b/optimizer-hints.md @@ -370,9 +370,9 @@ select /*+ READ_FROM_STORAGE(TIFLASH[t1], TIKV[t2]) */ t1.a from t t1, t t2 wher The `USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...])` hint tells the optimizer to access a specific table with the index merge method. Index merge has two types: intersection type and union type. For details, see [Explain Statements Using Index Merge](/explain-index-merge.md). -For the union-type index merge, the given list of indexes are optional parameters in the hint. If you explicitly specify the list, TiDB selects indexes from the list to build index merge; if you do not give the list of indexes, TiDB selects indexes from all available indexes to build index merge. See the following example. +If you explicitly specify the list of indexes, TiDB selects indexes from the list to build index merge; if you do not specify the list of indexes, TiDB selects indexes from all available indexes to build index merge. -For the intersection-type index merge, the given list of indexes are required parameters in the hint. +For the intersection-type index merge, the given list of indexes is a required parameter in the hint. For the union-type index merge, the given list of indexes is an optional parameter in the hint. See the following example. {{< copyable "sql" >}} From f35f0db1c97fd9eea40b4685727c75458873c419 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 6 Dec 2022 11:59:19 +0800 Subject: [PATCH 6/6] Update optimizer-hints.md --- optimizer-hints.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/optimizer-hints.md b/optimizer-hints.md index b44c2785cc541..4b0adfa167139 100644 --- a/optimizer-hints.md +++ b/optimizer-hints.md @@ -368,8 +368,18 @@ select /*+ READ_FROM_STORAGE(TIFLASH[t1], TIKV[t2]) */ t1.a from t t1, t t2 wher ### USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...]) + + The `USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...])` hint tells the optimizer to access a specific table with the index merge method. Index merge has two types: intersection type and union type. For details, see [Explain Statements Using Index Merge](/explain-index-merge.md). + + + + +The `USE_INDEX_MERGE(t1_name, idx1_name [, idx2_name ...])` hint tells the optimizer to access a specific table with the index merge method. Index merge has two types: intersection type and union type. + + + If you explicitly specify the list of indexes, TiDB selects indexes from the list to build index merge; if you do not specify the list of indexes, TiDB selects indexes from all available indexes to build index merge. For the intersection-type index merge, the given list of indexes is a required parameter in the hint. For the union-type index merge, the given list of indexes is an optional parameter in the hint. See the following example.