From 63ac57168edffc2e8a306ac49f72f54355fa701f Mon Sep 17 00:00:00 2001 From: qiancai Date: Tue, 24 Aug 2021 23:52:05 +0800 Subject: [PATCH 01/29] Create release-5.2.0.md --- releases/release-5.2.0.md | 328 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 328 insertions(+) create mode 100644 releases/release-5.2.0.md diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md new file mode 100644 index 0000000000000..1f013e04280de --- /dev/null +++ b/releases/release-5.2.0.md @@ -0,0 +1,328 @@ +--- +title: TiDB 5.2 Release Notes +--- + +# Release date: August 27, 2021 + +TiDB version: 5.2.0 + +In v5.2, the key new features and improvements are as follows: + +- Support expression index to greatly improve query performance +- Improve the accuracy of optimizer estimation (cardinality estimation) to select optimal execution plans +- Announce the general availability (GA) for the Lock View feature to observe transaction locking events and troubleshoot deadlock problems +- Add the TiFlash I/O traffic limit feature to improve the stability of read and write for TiFlash +- Optimize TiKV reserving space management to improve the stability of storage +- TiKV introduces a new flow control mechanism to replace the previous RocksDB write stall mechanism to improve the stability of TiKV flow control +- Simplify the operation and maintenance of Data Migration (DM) to reduce the management cost. +- TiCDC supports HTTP protocol OpenAPI to manage TiCDC tasks. It provides a more user-friendly operation method for both Kubernetes and on-premises environments. (Experimental feature) + +## Compatibility changes + +> **Note:** +> +> When upgrading from an earlier TiDB version to v5.2, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Note](/releases/release-notes.md) for the corresponding version. + +### System variables + +| Variable name | Change type | Description | +| :---------- | :----------- | :----------- | +| [`default_authentication_plugin`](/system-variables.md#default_authentication_plugin) | Newly added | Sets the authentication method that the server advertises. The default value is `mysql_native_password`. | +| [`tidb_enable_auto_increment_in_generated`](/system-variables.md#tidb_enable_auto_increment_in_generated) | Newly added | Determines whether to include the `AUTO_INCREMENT` columns when creating a generated column or an expression index. The default value is `OFF`. | +| [`tidb_opt_enable_correlation_adjustment`](/system-variables.md#tidb_opt_enable_correlation_adjustment) | Newly added | Controls whether the optimizer estimates the number of rows based on column order correlation. The default value is `ON`. | +| [`tidb_opt_limit_push_down_threshold`](/system-variables.md#tidb_opt_limit_push_down_threshold) | Newly added | Sets the threshold that determines whether to push the Limit or TopN operator down to TiKV. The default value is `100`. | +| [`tidb_restricted_read_only`](/system-variables.md#tidb_restricted_read_only) | Newly added | Controls whether the whole cluster is read-only. The default value is `OFF`. | +| [`tidb_stmt_summary_max_stmt_count`](/system-variables.md#tidb_stmt_summary_max_stmt_count-new-in-v40) | Modified | Sets the maximum number of statements that the statement summary tables store in memory. The default value is changed from `200` to `3000`. | +| `tidb_enable_streaming` | Deprecated | The system variable `enable-streaming` is deprecated and it is not recommended to use it any more. | + +### Configuration file parameters + +| Configuration file | Configuration item | Change type | Description | +| :---------- | :----------- | :----------- | :----------- | +| TiDB configuration file | [`pessimistic-txn.deadlock-history-collect-retryable`](/tidb-configuration-file.md#deadlock-history-collect-retryable) | Newly added | Controls whether the [`INFORMATION\_SCHEMA.DEADLOCKS`](/information-schema/information-schema-deadlocks.md) table collects retryable deadlock error messages or not. | +| TiDB configuration file | [`security.auto-tls`](/tidb-configuration-file.md#auto-tls) | Newly added | Determines whether to automatically generate the TLS certificates on startup. The default value is `true`. | +| TiDB configuration file | [`stmt-summary.max-stmt-count`](/tidb-configuration-file.md#max-stmt-count) | Modified | Indicates the maximum number of SQL categories allowed to be saved in the statement summary tables. The default value is changed from `200` to `3000`. | +| TiDB configuration file | `experimental.allow-expression-index` | Deprecated | The `allow-expression-index` configuration in the TiDB configuration file is deprecated. | +| TiKV configuration file | [`raftstore.cmd-batch`](/tikv-configuration-file.md#cmd-batch) | Newly added | Controls whether to enable batch processing of the requests. When it is enabled, the write performance is significantly improved. The default value is `true`. | +| TiKV configuration file | [`raftstore.inspect-interval`](/tikv-configuration-file.md#inspect-interval) | Newly added | At a certain interval, TiKV inspects the latency of the Raftstore component. This configuration item specifies the interval of the inspection. The default value is `500ms`. | +| TiKV configuration file | [`raftstore.max-peer-down-duration`](/tikv-configuration-file.md#max-peer-down-duration) | Modified | Indicates the longest inactive duration allowed for a peer. A peer with timeout is marked as `down`, and PD tries to delete it later. The default value is changed from `5m` to `10m`. | +| TiKV configuration file | [`server.raft-client-queue-size`](/tikv-configuration-file.md#raft-client-queue-size) | Newly added | Specifies the queue size of the Raft messages in TiKV. The default value is `8192`. | +| TiKV configuration file | [`storage.flow-control.enable`](/tikv-configuration-file.md#enable) | Newly added | Determines whether to enable the flow control mechanism. The default value is `true`. | +| TiKV configuration file | [`storage.flow-control.memtables-threshold`](/tikv-configuration-file.md#memtables-threshold) | Newly added | When the number of kvDB memtables reaches this threshold, the flow control mechanism starts to work. The default value is `5`. | +| TiKV configuration file | [`storage.flow-control.l0-files-threshold`](/tikv-configuration-file.md#l0-files-threshold) | Newly added | When the number of kvDB L0 files reaches this threshold, the flow control mechanism starts to work. The default value is `9`. | +| TiKV configuration file | [`storage.flow-control.soft-pending-compaction-bytes-limit`](/tikv-configuration-file.md#soft-pending-compaction-bytes-limit) | Newly added | When the pending compaction bytes in KvDB reach this threshold, the flow control mechanism starts to reject some write requests and reports the `ServerIsBusy` error. The default value is "192GB". | +| TiKV configuration file | [`storage.flow-control.hard-pending-compaction-bytes-limit`](/tikv-configuration-file.md#hard-pending-compaction-bytes-limit) | Newly added | When the pending compaction bytes in KvDB reach this threshold, the flow control mechanism rejects all write requests and reports the `ServerIsBusy` error. The default value is "1024GB". | + +### Others + +- Before the upgrade, check whether the value of the [`tidb_evolve_plan_baselines`](/system-variables.md#tidb_evolve_plan_baselines-new-in-v40) system variable is `ON`. If the value is `ON`, set it to `OFF`; otherwise, the upgrade will fail. +- For TiDB clusters upgraded from v4.0 to v5.2, the default value of [`tidb_multi_statement_mode`](/system-variables.md#tidb_multi_statement_mode-new-in-v4011) changes from `WARN` to `OFF`. +- Before the upgrade, check the value of the TiDB configuration [`feedback-probability`](/tidb-configuration-file.md#feedback-probability). If the value is not `0`, the "panic in the recoverable goroutine" error will occur after the upgrade, but this error does not affect the upgrade. +- TiDB is now compatible with MySQL 5.7's noop variable `innodb_default_row_format`. Setting this variable has no effect. [#23541](https://github.com/pingcap/tidb/issues/23541) +- When upgraded to v5.2, TiKV will calculate `memory-usage-limit` based on `block-cache.capacity`, and the result of `memory-usage-limit` defaults to the value of "`block-cache.capacity` / 0.45 * 0.75". + + For example, when [storage.block-cache] capacity = "24GB", `memory-usage-limit` is 40GB (24GB/0.45*0.75 = 40GB). + +## New features + +### SQL + +- **Support expression index** + + The expression index is a type of special index that can be created on an expression. After an expression index is created, TiDB supports expression-based queries, which greatly improves query performance. + + [User document](/sql-statements/sql-statement-create-index.md), [#25150](https://github.com/pingcap/tidb/issues/25150) + +- **Support the `translate` function in Oracle** + + The `translate` function replaces all occurrences of characters by other characters in a string. In TiDB, this function does not treat empty strings as `NULL` as Oracle does. + + [User document](/functions-and-operators/string-functions.md) + +- **Support spilling HashAgg** + + Support spilling HashAgg into disk. When a SQL statement that includes an HashAgg operator causes out of memory (OOM), you can try to set the concurrency of this operator to `1` to trigger disk spill, which alleviates memory stress. + + [User document](/configure-memory-usage.md#other-memory-control-behaviors-of-tidb-server), [#25882](https://github.com/pingcap/tidb/issues/25882) + +- **Improve the accuracy of optimizer estimation (cardinality estimation)** + + - Improve the accuracy of TiDB's estimation of TopN/Limit. For example, for pagination queries on a large table that contain the `order by col limit x` condition, TiDB can more easily select the right index and reduce query response time. + - Improve the accuracy of out-of-scope estimation. For example, even if the statistics for a day have not been updated, TiDB can accurately select the corresponding index for a query that contains `where date=Now()`. + - Introduce the `tidb_opt_limit_push_down_threshold` variable to control the optimizer's behavior of pushing down Limit/TopN, which resolves the issue that Limit/TopN cannot be pushed down in some situations due to wrong estimation. + + [User document](/system-variables.md#tidb_opt_limit_push_down_threshold), [#26085](https://github.com/pingcap/tidb/issues/26085) + +- **Improve index selection of the optimizer** + + Add pruning rules for index selection. Before using the statistics for comparison, TiDB uses these rules to narrow down the scope of possible indexes to be selected, which reduces the possibility of selecting non-optimal indexes. + + [User document](/choose-index.md) + +### Transaction + +- **General availability (GA) for Lock View** + + The Lock View feature provides more information about lock conflicts and lock waits of pessimistic locks, which helps DBAs to observe transaction locking events and troubleshoot deadlock problems. + + In v5.2, the following enhancements are made to Lock View: + + - In addition to the SQL digest column in the Lock View-related tables, add a column to these tables that shows the corresponding normalized SQL text. You do not have to manually query the statement corresponding to a SQL digest. + - Add the `TIDB_DECODE_SQL_DIGESTS` function to query the normalized SQL statements (a form without formats and arguments) corresponding to a set of SQL digests in the cluster. This simplifies the operation of querying the statements that have been historically executed by a transaction. + - Add a column in the `DATA_LOCK_WAITS` and `DEADLOCKS` system tables to show the table name, row ID, index value, and other key information interpreted from a key. This simplifies the operations such as locating the table to which a key belongs and interpreting the key information. + - Support collecting the information of retryable deadlock errors in the `DEADLOCKS` table, which makes it easier to troubleshoot issues caused by such errors. The error collection is disabled by default and can be enabled using the `pessimistic-txn.deadlock-history-collect-retryable` configuration. + - Support distinguishing query-executing transactions from idle transactions on the `TIDB_TRX` system table. The `Normal` state is now divided into `Running` and `Idle` states. + + User documents: + + - View the pessimistic lock-waiting events that are occurring on all TiKV nodes in the cluster: [`DATA_LOCK_WAITS`](/information-schema/information-schema-data-lock-waits.md) + - View the deadlock errors recently occurred on a TiDB node: [`DEADLOCKS`](/information-schema/information-schema-deadlocks.md) + - View the executing transaction on a TiDB node: [`TIDB_TRX`](/information-schema/information-schema-tidb-trx.md) + + - Optimize the user scenarios of adding indexes on tables with the `AUTO_RANDOM` or `SHARD_ROW_ID_BITS` attribute. + +### Stability + +- **Add TiFlash I/O traffic limit** +This new feature is suitable for cloud storage with disk bandwidth of a small and specific size. It is disabled by default. + +TiFlash IO Rate Limiter provides a new mechanism to avoid excessive race for I/O resources between read and write tasks. It balances the responses to read and write tasks, and limits the rate automatically according to read/write workload. + +[User document](/tiflash/tiflash-configuration.md) + +- **Improve stability of TiKV flow control** + +TiKV introduces a new flow control mechanism to replacesupersede the previous RocksDB write stall mechanism. This mechanism controls flow at the scheduler layer, which avoids the issue of QPS drop caused by the stuck Raftstore or Apply threads when the write traffic is high. + +[User document](/tikv-configuration-file.md#storageflow-control), [#10137](https://github.com/tikv/tikv/issues/10137) + +- **Detect and recover automatically from impact caused by a single slow TiKV node in a cluster** + +TiKV introduces the slow node detection mechanism. This mechanism calculates a score by inspecting the rate of TiKV Raftstore, and then reports the score to PD through store heartbeats. Meanwhile, it adds the `evict-slow-store-scheduler` scheduler on PD to automatically evict the leader on a single slow TiKV node. In this way, the impact on the whole cluster is mitigated. At the same time, more alert items about slow nodes are introduced to help you quickly pinpoint and solve problems. + +[User document]( /tikv-configuration-file.md#inspect-interval), [#10539](https://github.com/tikv/tikv/issues/10539) + +### Data Migration + +- **Simplify operations of Data Migration (DM)** + +DM v2.0.6 can automatically identify the change event (failover or plan change) of the data source using VIP, and can automatically connect to a new data source instance, to reduce data replication latency and simplify operation procedures. + +- TiDB Lightning supports customized line terminators in the CSV data, and is compatible with the MySQL LOAD DATA CSV data formats. You can then use TiDB Lightning directly in your data flow architecture. +[#1297](https://github.com/pingcap/br/pull/1297) + +### TiDB data share subscription + +TiCDC supports using the HTTP protocol (OpenAPI) to manage TiCDC tasks, which is a more user-friendly operation method for both Kubernetes and on-premises environments. (Experimental feature) + +[#2411](https://github.com/pingcap/ticdc/issues/2411) + +### Deployment and operations + +Support running the `tiup playground` command on Mac computers with Apple M1 chips. + +### Telemetry + +Telemetry supports collecting usage data of specific features, such as the usage data of built-in functions. + +For detailed information of the collected data and how to disable data collection, refer to [Telemetry](/telemetry.md). + +## Feature Enhancements + ++ Tools + + + TiCDC + + - Add HTTP API. Support querying and modifying TiCDC clusters. [#2416](https://github.com/pingcap/ticdc/pull/2416) + - Add the binary MQ format designed for TiDB. It is more compact than the open protocols based on JSON [#1621](https://github.com/pingcap/ticdc/pull/1621) + - Remove support for file sorter [#2114](https://github.com/pingcap/ticdc/pull/2114) + - Support log rotation configurations [#2182](https://github.com/pingcap/ticdc/pull/2182) + + + TiDB Lightning + + - Support customized line terminators (except `\r` and `\n`) [#1297](https://github.com/pingcap/br/pull/1297) + - Support expression index and the index that depends on virtual generated columns [#1407](https://github.com/pingcap/br/pull/1407) + + + Dumpling + - Support backing up MySQL compatible databases but does not support `START TRANSACTION ... WITH CONSISTENT SNAPSHOT` or`SHOW CREATE TABLE` [#311](https://github.com/pingcap/dumpling/pull/311) + +## Improvements + ++ TiDB + + - Support pushing down the built-in function `json_unquote()` to TiKV [#24415](https://github.com/pingcap/tidb/issues/24415) + - Support removing the `union` branch from the dual table [#25614](https://github.com/pingcap/tidb/pull/25614) + - Optimize the aggregate operator's cost factor [#25241](https://github.com/pingcap/tidb/pull/25241) + - Allow the MPP outer join to choose the build table based on the table row count [#25142](https://github.com/pingcap/tidb/pull/25142) + - Support balancing the MPP query workload among different TiFlash nodes based on Regions [#24724](https://github.com/pingcap/tidb/pull/24724) + - Support invalidating stale Regions in the cache after the MPP query is executed [#24432](https://github.com/pingcap/tidb/pull/24432) + - Improve the MySQL compatibility of the built-in function`str_to_date` for the format specifiers `%b/%M/%r/%T` [#25767](https://github.com/pingcap/tidb/pull/25767) + - Fix the issue that inconsistent binding caches might be created in multiple TiDB after recreating different bindings for the same query. [#26015](https://github.com/pingcap/tidb/pull/26015) + - Fix the issue that the existing bindings cannot be loaded into cache after upgrade [#23295](https://github.com/pingcap/tidb/pull/23295) + - Support ordering the result of `SHOW BINDINGS` by (`original_sql`, `update_time`) [#26139](https://github.com/pingcap/tidb/pull/26139) + - Improve the logic of query optimization when bindings exist, and reduce optimization times of a query [#26141](https://github.com/pingcap/tidb/pull/26141) + - Support completing the garbage collection automatically for the bindings in the "deleted" status [#26206](https://github.com/pingcap/tidb/pull/26206) + - Support showing whether a binding is used for query optimization in the result of `EXPLAIN VERBOSE` [#26930](https://github.com/pingcap/tidb/pull/26930) + - Add a new status variation `last_plan_binding_update_time` status variation to view the timestamp corresponding to the binding cache in the current TiDB instance. [#26340](https://github.com/pingcap/tidb/pull/26340) + - Support reporting an error when starting binding evolution or running `admin evolve bindings` to ban the baseline evolution (currently disabled in the on-premises TiDB version because it is an experimental feature) affecting other features. [#26333](https://github.com/pingcap/tidb/pull/26333) + ++ TiKV + + - Limit memory consumption of the TiCDC sink[#10305](https://github.com/tikv/tikv/pull/10305) + - Add the memory-bounded upper limit for the TiCDC old value cache [#10313](https://github.com/tikv/tikv/pull/10313) + ++ PD + + - Add more QPS dimensions for hot Region scheduling, and support adjusting the priority of the scheduling [#3869](https://github.com/tikv/pd/issues/3869) + - Support hot Region balance scheduling for the write hotspot of TiFlash [#3900](https://github.com/tikv/pd/pull/3900) + ++ TiFlash + + - Add operators: `MOD / %`, `LIKE` + - Add string functions: `ASCII()`, `COALESCE()`, `LENGTH()`, `POSITION()`, `TRIM()` + - Add mathematical functions: `CONV()`, `CRC32()`, `DEGREES()`, `EXP()`, `LN()`, `LOG()`, `LOG10()`, `LOG2()`, `POW()`, `RADIANS()`, `ROUND(decimal)`, `SIN()`, `MOD()` + - Add date functions: `ADDDATE(string, real)`, `DATE_ADD(string, real)`, `DATE()` + - Add other functions: `INET_NTOA()`, `INET_ATON()`, `INET6_ATON`, `INET6_NTOA()` + - Support Shuffled Hash Join calculation and Shuffled Hash Aggregation calculation in the MPP mode when a new collation is enabled + - Optimize basic code to improve MPP performance + - Add `-1` as the positive infinity value for `tidb_broadcast_join_threshold_count` and `tidb_broadcast_join_threshold_size` + - Support casting the `STRING` type to the `DOUBLE` type + - Optimize the non-joined data in right outer join using multiple threads + - Support automatically invalidating stale Regions in MPP queries + ++ Tools + + + TiCDC + - Add the concurrency limit to the incremental scan of kv client [#1899](https://github.com/pingcap/ticdc/pull/1899) + - TiCDC can always pull the old value internally [#2271](https://github.com/pingcap/ticdc/pull/2271) + - TiCDC can fail and exit fast when unrecoverable DML errors occur [#1928](https://github.com/pingcap/ticdc/pull/1928) + - `resolve lock` cannot be run immediately after a Region is initialized [#2235](https://github.com/pingcap/ticdc/pull/2235) + - Optimize workerpool to reduce the number of goroutines under high concurrency [#2201](https://github.com/pingcap/ticdc/pull/2201) + + + Dumpling + - Support always splitting TiDB v3.x tables through `tidb_rowid` to save TiDB memory [#301](https://github.com/pingcap/dumpling/pull/301) + - Reduce access of Dumpling to the `information_schema` to improve stability [#305](https://github.com/pingcap/dumpling/pull/305) + +## Bug 修复 + ++ TiDB + + - Fix the issue that an incorrect result is returned when using merge join on the `SET` type column [#25669](https://github.com/pingcap/tidb/issues/25669) + - Fix the data corruption issue in the `IN` expression's arguments [#25591](https://github.com/pingcap/tidb/issues/25591) + - Avoid the sessions of GC being affected by global variables [#24976](https://github.com/pingcap/tidb/issues/24976) + - Fix the panic issue that occurs when using `limit` in the window function queries [#25344](https://github.com/pingcap/tidb/issues/25344) + - Fix the wrong value returned when querying a partitioned table using `Limit` [#24636](https://github.com/pingcap/tidb/issues/24636) + - Fix the issue that `IFNULL` does not correctly take effect on the `ENUM` or `SET` type column [#24944](https://github.com/pingcap/tidb/issues/24944) + - Fix the wrong results caused by changing the `count` in the join subqueries to `first_row` [#24865](https://github.com/pingcap/tidb/issues/24865) + - Fix the query hang issue that occurs when `ParallelApply` is used under the `TopN` operator [#24930](https://github.com/pingcap/tidb/issues/24930) + - Fix the issue that more results than expected are returned when executing SQL statements using multi-column prefix indexes [#24356](https://github.com/pingcap/tidb/issues/24356) + - Fix the issue that the `<=>` operator cannot correctly take effect [#24477](https://github.com/pingcap/tidb/issues/24477) + - Fix the data race issue of the parallel `Apply` operator [#23280](https://github.com/pingcap/tidb/issues/23280) + - Fix the issue that the `index out of range` error is reported when sorting the IndexMerge results of the PartitionUnion operator [#23919](https://github.com/pingcap/tidb/issues/23919) + - Fix the issue that setting the `tidb_snapshot` variable to an unexpectedly large value might damage the transaction isolation [#25680](https://github.com/pingcap/tidb/issues/25680) + - Fix the issue that the ODBC-styled constant (for example, `{d '2020-01-01'}`) cannot be used as the expression [#25531](https://github.com/pingcap/tidb/issues/25531) + - Fix the issue that `SELECT DISTINCT` converted to `Batch Get` causes incorrect results [#25320](https://github.com/pingcap/tidb/issues/25320) + - Fix the issue that backing off queries from TiFlash to TiKV cannot be triggered [#23665](https://github.com/pingcap/tidb/issues/23665) [#24421](https://github.com/pingcap/tidb/issues/24421) + - Fix the `index-out-of-range` error that occurs when checking `only_full_group_by` [#23839](https://github.com/pingcap/tidb/issues/23839)) + - Fix the issue that the result of index join in correlated subqueries is wrong [#25799](https://github.com/pingcap/tidb/issues/25799) + ++ TiKV + + - Fix the wrong `tikv_raftstore_hibernated_peer_state` metric [#10330](https://github.com/tikv/tikv/issues/10330) + - Fix the wrong arguments type of the `json_unquote()` function in the coprocessor [#10176](https://github.com/tikv/tikv/issues/10176) + - Skip clearing callback during graceful shutdown to avoid breaking ACID in some cases [#10353](https://github.com/tikv/tikv/issues/10353) [#10307](https://github.com/tikv/tikv/issues/10307) + - Fix a bug that the read index is shared for replica reads on a Leader [#10347](https://github.com/tikv/tikv/issues/10347) + - Fix the wrong function that casts `DOUBLE` to `DOUBLE` [#25200](https://github.com/pingcap/tidb/issues/25200) + ++ PD + + - Fix the issue that the expected scheduling cannot be generated due to scheduling conflicts among multiple schedulers + [#3807](https://github.com/tikv/pd/issues/3807) [#3778](https://github.com/tikv/pd/issues/3778) + ++ TiFlash + + - Fix the issue that TiFlash keeps restarting because of the split failure + - Fix the potential issue that TiFlash cannot delete the delta data + - Fix a bug that TiFlash adds wrong padding for non-binary characters in the `CAST` function + - Fix the issue of incorrect results when handling aggregation queries with complex `GROUP BY` columns + - Fix the TiFlash panic issue that occurs under heavy write pressure + - Fix the panic that occurs when the right jon key is not nullable and the left join key is nullable + - Fix the potential issue that the `read-index` requests take a long time + - Fix the panic issue that occurs when the read load is heavy + - Fix the panic issue that might occur when the `Date_Format` function is called with the `STRING` type argument and `NULL` values + ++ Tools + + + TiCDC + + - Fix a bug that TiCDC owner exits abnormally when refreshing the checkpoint [#1985](https://github.com/pingcap/ticdc/pull/1985) + - Fix a bug that changefeed fails immediately after its successful creation [#2115](https://github.com/pingcap/ticdc/pull/2115) + - Fix a bug that changefeed fails due to the invalid format of rules filter [#2117](https://github.com/pingcap/ticdc/pull/2117) + - Fix the potential DDL loss issue when the TiCDC owner panics [#2252](https://github.com/pingcap/ticdc/pull/2252) + - Fix the CLI compatibility issue with 4.0.x clusters on the default sort-engine option [#2385](https://github.com/pingcap/ticdc/pull/2385) + - Fix a bug that changefeed might be reset unexpectedly when TiCDC gets the `ErrSchemaStorageTableMiss` error [#2423](https://github.com/pingcap/ticdc/pull/2423) + - Fix a bug that changefeed cannot be removed when TiCDC gets the `ErrGCTTLExceeded` error [#2429](https://github.com/pingcap/ticdc/pull/2429) + - Fix a bug that TiCDC fails to synchronize large tables to cdclog [#2431](https://github.com/pingcap/ticdc/pull/2431) + - Fix a bug that multiple processors might write data to the same table when TiCDC is rescheduling the table [#2417](https://github.com/pingcap/ticdc/pull/2417) + + + Backup & Restore (BR) + + - Fix a bug that BR skips restoring all system tables during the restore [#1197](https://github.com/pingcap/br/issues/1197) [#1201](https://github.com/pingcap/br/issues/1201) + - Fix a bug that BR misses DDL operations when restoring cdclog [#870](https://github.com/pingcap/br/issues/870) + + + TiDB Lightning + + - Fix a bug that Lightning fails to parse the `DECIMAL` data type in Parquet file [#1272](https://github.com/pingcap/br/pull/1272) + - Fix a bug that Lightning reports the "Error 9007: Write conflict" error when restoring table schemas [#1290](https://github.com/pingcap/br/issues/1290) + - Fix a bug that Lightning fails to import data due to the overflow of int handle [#1291](https://github.com/pingcap/br/issues/1291) + - Fix a bug that Lightning might get a checksum mismatching error due to data loss in the local backend mode [#1413](https://github.com/pingcap/br/pull/1413) + - Fix the Lighting incompatibility issue with clustered index when Lightning is restoring table schemas [#1364](https://github.com/pingcap/br/pull/1364) + + + + Dumpling + + - Fix a bug that the data export fails because the Dumpling GC safepoint is set too late [#290](https://github.com/pingcap/dumpling/pull/290) + - Fix the Dumpling getting stuck issue when exporting table names from the upstream database in certain MySQL versions [#325](https://github.com/pingcap/dumpling/pull/325) + + + From 5db56485b9b30f4f560a3dff5621bbda6f470da7 Mon Sep 17 00:00:00 2001 From: qiancai Date: Tue, 24 Aug 2021 23:58:59 +0800 Subject: [PATCH 02/29] Update release-5.2.0.md --- releases/release-5.2.0.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 1f013e04280de..aff290e978f99 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -124,21 +124,22 @@ In v5.2, the key new features and improvements are as follows: ### Stability - **Add TiFlash I/O traffic limit** -This new feature is suitable for cloud storage with disk bandwidth of a small and specific size. It is disabled by default. - -TiFlash IO Rate Limiter provides a new mechanism to avoid excessive race for I/O resources between read and write tasks. It balances the responses to read and write tasks, and limits the rate automatically according to read/write workload. - -[User document](/tiflash/tiflash-configuration.md) + + This new feature is suitable for cloud storage with disk bandwidth of a small and specific size. It is disabled by default. + + TiFlash IO Rate Limiter provides a new mechanism to avoid excessive race for I/O resources between read and write tasks. It balances the responses to read and write tasks, and limits the rate automatically according to read/write workload. + + [User document](/tiflash/tiflash-configuration.md) - **Improve stability of TiKV flow control** -TiKV introduces a new flow control mechanism to replacesupersede the previous RocksDB write stall mechanism. This mechanism controls flow at the scheduler layer, which avoids the issue of QPS drop caused by the stuck Raftstore or Apply threads when the write traffic is high. + TiKV introduces a new flow control mechanism to replacesupersede the previous RocksDB write stall mechanism. This mechanism controls flow at the scheduler layer, which avoids the issue of QPS drop caused by the stuck Raftstore or Apply threads when the write traffic is high. [User document](/tikv-configuration-file.md#storageflow-control), [#10137](https://github.com/tikv/tikv/issues/10137) - **Detect and recover automatically from impact caused by a single slow TiKV node in a cluster** -TiKV introduces the slow node detection mechanism. This mechanism calculates a score by inspecting the rate of TiKV Raftstore, and then reports the score to PD through store heartbeats. Meanwhile, it adds the `evict-slow-store-scheduler` scheduler on PD to automatically evict the leader on a single slow TiKV node. In this way, the impact on the whole cluster is mitigated. At the same time, more alert items about slow nodes are introduced to help you quickly pinpoint and solve problems. + TiKV introduces the slow node detection mechanism. This mechanism calculates a score by inspecting the rate of TiKV Raftstore, and then reports the score to PD through store heartbeats. Meanwhile, it adds the `evict-slow-store-scheduler` scheduler on PD to automatically evict the leader on a single slow TiKV node. In this way, the impact on the whole cluster is mitigated. At the same time, more alert items about slow nodes are introduced to help you quickly pinpoint and solve problems. [User document]( /tikv-configuration-file.md#inspect-interval), [#10539](https://github.com/tikv/tikv/issues/10539) @@ -146,7 +147,7 @@ TiKV introduces the slow node detection mechanism. This mechanism calculates a s - **Simplify operations of Data Migration (DM)** -DM v2.0.6 can automatically identify the change event (failover or plan change) of the data source using VIP, and can automatically connect to a new data source instance, to reduce data replication latency and simplify operation procedures. + DM v2.0.6 can automatically identify the change event (failover or plan change) of the data source using VIP, and can automatically connect to a new data source instance, to reduce data replication latency and simplify operation procedures. - TiDB Lightning supports customized line terminators in the CSV data, and is compatible with the MySQL LOAD DATA CSV data formats. You can then use TiDB Lightning directly in your data flow architecture. [#1297](https://github.com/pingcap/br/pull/1297) From 08ee6dc016785c06f11e536a277190be3619e5c1 Mon Sep 17 00:00:00 2001 From: qiancai Date: Wed, 25 Aug 2021 09:35:27 +0800 Subject: [PATCH 03/29] Update release-5.2.0.md --- releases/release-5.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index aff290e978f99..139fb55f76b43 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -244,7 +244,7 @@ For detailed information of the collected data and how to disable data collectio - Support always splitting TiDB v3.x tables through `tidb_rowid` to save TiDB memory [#301](https://github.com/pingcap/dumpling/pull/301) - Reduce access of Dumpling to the `information_schema` to improve stability [#305](https://github.com/pingcap/dumpling/pull/305) -## Bug 修复 +## Bug Fixes + TiDB From 2997387c1531a0770b881cd2acc97a8a2f0f8e22 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 25 Aug 2021 11:07:37 +0800 Subject: [PATCH 04/29] Update releases/release-5.2.0.md --- releases/release-5.2.0.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 139fb55f76b43..149518d406cb5 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -133,7 +133,14 @@ In v5.2, the key new features and improvements are as follows: - **Improve stability of TiKV flow control** - TiKV introduces a new flow control mechanism to replacesupersede the previous RocksDB write stall mechanism. This mechanism controls flow at the scheduler layer, which avoids the issue of QPS drop caused by the stuck Raftstore or Apply threads when the write traffic is high. + TiKV introduces a new flow control mechanism to replace the previous RocksDB write stall mechanism. Compared with the write stall mechanism, this new mechanism reduces the impact on the stability of foreground write by the following: + + When RocksDB compaction stacks, flow control is performed on the TiKV scheduler layer instead of the RocksDB layer, to avoid the following issues: + + - Raftstore is stuck, which is caused by RocksDB write stall. + - Raft election times out, and the node leader is transferred as a result. + + This new mechanism improves the flow control algorithm to mitigate QPS decrease when the write traffic is high. [User document](/tikv-configuration-file.md#storageflow-control), [#10137](https://github.com/tikv/tikv/issues/10137) From 49022ce211d5c4ace6c95068f5d15158a648ed03 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 25 Aug 2021 11:08:42 +0800 Subject: [PATCH 05/29] Update releases/release-5.2.0.md --- releases/release-5.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 149518d406cb5..3c3e9c1772d92 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -142,7 +142,7 @@ In v5.2, the key new features and improvements are as follows: This new mechanism improves the flow control algorithm to mitigate QPS decrease when the write traffic is high. -[User document](/tikv-configuration-file.md#storageflow-control), [#10137](https://github.com/tikv/tikv/issues/10137) + [User document](/tikv-configuration-file.md#storageflow-control), [#10137](https://github.com/tikv/tikv/issues/10137) - **Detect and recover automatically from impact caused by a single slow TiKV node in a cluster** From 40b154233c8703293564c08a200130ba36308ec6 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 25 Aug 2021 14:37:39 +0800 Subject: [PATCH 06/29] Update releases/release-5.2.0.md Co-authored-by: Enwei --- releases/release-5.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 3c3e9c1772d92..390eb4b710a61 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -211,7 +211,7 @@ For detailed information of the collected data and how to disable data collectio - Improve the logic of query optimization when bindings exist, and reduce optimization times of a query [#26141](https://github.com/pingcap/tidb/pull/26141) - Support completing the garbage collection automatically for the bindings in the "deleted" status [#26206](https://github.com/pingcap/tidb/pull/26206) - Support showing whether a binding is used for query optimization in the result of `EXPLAIN VERBOSE` [#26930](https://github.com/pingcap/tidb/pull/26930) - - Add a new status variation `last_plan_binding_update_time` status variation to view the timestamp corresponding to the binding cache in the current TiDB instance. [#26340](https://github.com/pingcap/tidb/pull/26340) + - Add a new status variation `last_plan_binding_update_time` to view the timestamp corresponding to the binding cache in the current TiDB instance [#26340](https://github.com/pingcap/tidb/pull/26340) - Support reporting an error when starting binding evolution or running `admin evolve bindings` to ban the baseline evolution (currently disabled in the on-premises TiDB version because it is an experimental feature) affecting other features. [#26333](https://github.com/pingcap/tidb/pull/26333) + TiKV From 2da4b83fd957d9225669a9453ac669e696976788 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 25 Aug 2021 14:40:05 +0800 Subject: [PATCH 07/29] Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> Co-authored-by: Enwei --- releases/release-5.2.0.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 390eb4b710a61..5016e0333b1b7 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -4,6 +4,8 @@ title: TiDB 5.2 Release Notes # Release date: August 27, 2021 +Release date: August 27, 2021 + TiDB version: 5.2.0 In v5.2, the key new features and improvements are as follows: @@ -133,9 +135,9 @@ In v5.2, the key new features and improvements are as follows: - **Improve stability of TiKV flow control** - TiKV introduces a new flow control mechanism to replace the previous RocksDB write stall mechanism. Compared with the write stall mechanism, this new mechanism reduces the impact on the stability of foreground write by the following: + TiKV introduces a new flow control mechanism to replace the previous RocksDB write stall mechanism. Compared with the write stall mechanism, this new mechanism reduces the impact on the stability of foreground write. - When RocksDB compaction stacks, flow control is performed on the TiKV scheduler layer instead of the RocksDB layer, to avoid the following issues: + In specific, when the stress of RocksDB compaction accumulates, flow control is performed at the TiKV scheduler layer instead of the RocksDB layer, to avoid the following issues: - Raftstore is stuck, which is caused by RocksDB write stall. - Raft election times out, and the node leader is transferred as a result. @@ -205,7 +207,7 @@ For detailed information of the collected data and how to disable data collectio - Support balancing the MPP query workload among different TiFlash nodes based on Regions [#24724](https://github.com/pingcap/tidb/pull/24724) - Support invalidating stale Regions in the cache after the MPP query is executed [#24432](https://github.com/pingcap/tidb/pull/24432) - Improve the MySQL compatibility of the built-in function`str_to_date` for the format specifiers `%b/%M/%r/%T` [#25767](https://github.com/pingcap/tidb/pull/25767) - - Fix the issue that inconsistent binding caches might be created in multiple TiDB after recreating different bindings for the same query. [#26015](https://github.com/pingcap/tidb/pull/26015) + - Fix the issue that inconsistent binding caches might be created in multiple TiDB after recreating different bindings for the same query [#26015](https://github.com/pingcap/tidb/pull/26015) - Fix the issue that the existing bindings cannot be loaded into cache after upgrade [#23295](https://github.com/pingcap/tidb/pull/23295) - Support ordering the result of `SHOW BINDINGS` by (`original_sql`, `update_time`) [#26139](https://github.com/pingcap/tidb/pull/26139) - Improve the logic of query optimization when bindings exist, and reduce optimization times of a query [#26141](https://github.com/pingcap/tidb/pull/26141) @@ -216,7 +218,7 @@ For detailed information of the collected data and how to disable data collectio + TiKV - - Limit memory consumption of the TiCDC sink[#10305](https://github.com/tikv/tikv/pull/10305) + - Limit memory consumption of the TiCDC sink [#10305](https://github.com/tikv/tikv/pull/10305) - Add the memory-bounded upper limit for the TiCDC old value cache [#10313](https://github.com/tikv/tikv/pull/10313) + PD From c873442f9f65bdd4dc48fbebfe98abc9be1f9fa4 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 25 Aug 2021 14:40:18 +0800 Subject: [PATCH 08/29] Apply suggestions from code review Co-authored-by: Enwei --- releases/release-5.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 5016e0333b1b7..d2010fcc7df8d 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -214,7 +214,7 @@ For detailed information of the collected data and how to disable data collectio - Support completing the garbage collection automatically for the bindings in the "deleted" status [#26206](https://github.com/pingcap/tidb/pull/26206) - Support showing whether a binding is used for query optimization in the result of `EXPLAIN VERBOSE` [#26930](https://github.com/pingcap/tidb/pull/26930) - Add a new status variation `last_plan_binding_update_time` to view the timestamp corresponding to the binding cache in the current TiDB instance [#26340](https://github.com/pingcap/tidb/pull/26340) - - Support reporting an error when starting binding evolution or running `admin evolve bindings` to ban the baseline evolution (currently disabled in the on-premises TiDB version because it is an experimental feature) affecting other features. [#26333](https://github.com/pingcap/tidb/pull/26333) + - Support reporting an error when starting binding evolution or running `admin evolve bindings` to ban the baseline evolution (currently disabled in the on-premises TiDB version because it is an experimental feature) affecting other features [#26333](https://github.com/pingcap/tidb/pull/26333) + TiKV From 72153c09938e47b20318dab61e8b0f4eef3e9c24 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 25 Aug 2021 19:00:34 +0800 Subject: [PATCH 09/29] bug fix: pr --> issue --- releases/release-5.2.0.md | 125 ++++++++++++++++++-------------------- 1 file changed, 60 insertions(+), 65 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index d2010fcc7df8d..2cbbc1609ff36 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -23,37 +23,37 @@ In v5.2, the key new features and improvements are as follows: > **Note:** > -> When upgrading from an earlier TiDB version to v5.2, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Note](/releases/release-notes.md) for the corresponding version. +> When upgrading from an earlier TiDB version to v5.2, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Note](/releases/release-notes.md) for the corresponding version. ### System variables -| Variable name | Change type | Description | -| :---------- | :----------- | :----------- | -| [`default_authentication_plugin`](/system-variables.md#default_authentication_plugin) | Newly added | Sets the authentication method that the server advertises. The default value is `mysql_native_password`. | -| [`tidb_enable_auto_increment_in_generated`](/system-variables.md#tidb_enable_auto_increment_in_generated) | Newly added | Determines whether to include the `AUTO_INCREMENT` columns when creating a generated column or an expression index. The default value is `OFF`. | -| [`tidb_opt_enable_correlation_adjustment`](/system-variables.md#tidb_opt_enable_correlation_adjustment) | Newly added | Controls whether the optimizer estimates the number of rows based on column order correlation. The default value is `ON`. | -| [`tidb_opt_limit_push_down_threshold`](/system-variables.md#tidb_opt_limit_push_down_threshold) | Newly added | Sets the threshold that determines whether to push the Limit or TopN operator down to TiKV. The default value is `100`. | -| [`tidb_restricted_read_only`](/system-variables.md#tidb_restricted_read_only) | Newly added | Controls whether the whole cluster is read-only. The default value is `OFF`. | -| [`tidb_stmt_summary_max_stmt_count`](/system-variables.md#tidb_stmt_summary_max_stmt_count-new-in-v40) | Modified | Sets the maximum number of statements that the statement summary tables store in memory. The default value is changed from `200` to `3000`. | -| `tidb_enable_streaming` | Deprecated | The system variable `enable-streaming` is deprecated and it is not recommended to use it any more. | +| Variable name | Change type | Description | +| :---------- | :----------- | :----------- | +| [`default_authentication_plugin`](/system-variables.md#default_authentication_plugin) | Newly added | Sets the authentication method that the server advertises. The default value is `mysql_native_password`. | +| [`tidb_enable_auto_increment_in_generated`](/system-variables.md#tidb_enable_auto_increment_in_generated) | Newly added | Determines whether to include the `AUTO_INCREMENT` columns when creating a generated column or an expression index. The default value is `OFF`. | +| [`tidb_opt_enable_correlation_adjustment`](/system-variables.md#tidb_opt_enable_correlation_adjustment) | Newly added | Controls whether the optimizer estimates the number of rows based on column order correlation. The default value is `ON`. | +| [`tidb_opt_limit_push_down_threshold`](/system-variables.md#tidb_opt_limit_push_down_threshold) | Newly added | Sets the threshold that determines whether to push the Limit or TopN operator down to TiKV. The default value is `100`. | +| [`tidb_restricted_read_only`](/system-variables.md#tidb_restricted_read_only) | Newly added | Controls whether the whole cluster is read-only. The default value is `OFF`. | +| [`tidb_stmt_summary_max_stmt_count`](/system-variables.md#tidb_stmt_summary_max_stmt_count-new-in-v40) | Modified | Sets the maximum number of statements that the statement summary tables store in memory. The default value is changed from `200` to `3000`. | +| `tidb_enable_streaming` | Deprecated | The system variable `enable-streaming` is deprecated and it is not recommended to use it any more. | ### Configuration file parameters -| Configuration file | Configuration item | Change type | Description | -| :---------- | :----------- | :----------- | :----------- | -| TiDB configuration file | [`pessimistic-txn.deadlock-history-collect-retryable`](/tidb-configuration-file.md#deadlock-history-collect-retryable) | Newly added | Controls whether the [`INFORMATION\_SCHEMA.DEADLOCKS`](/information-schema/information-schema-deadlocks.md) table collects retryable deadlock error messages or not. | -| TiDB configuration file | [`security.auto-tls`](/tidb-configuration-file.md#auto-tls) | Newly added | Determines whether to automatically generate the TLS certificates on startup. The default value is `true`. | -| TiDB configuration file | [`stmt-summary.max-stmt-count`](/tidb-configuration-file.md#max-stmt-count) | Modified | Indicates the maximum number of SQL categories allowed to be saved in the statement summary tables. The default value is changed from `200` to `3000`. | -| TiDB configuration file | `experimental.allow-expression-index` | Deprecated | The `allow-expression-index` configuration in the TiDB configuration file is deprecated. | -| TiKV configuration file | [`raftstore.cmd-batch`](/tikv-configuration-file.md#cmd-batch) | Newly added | Controls whether to enable batch processing of the requests. When it is enabled, the write performance is significantly improved. The default value is `true`. | -| TiKV configuration file | [`raftstore.inspect-interval`](/tikv-configuration-file.md#inspect-interval) | Newly added | At a certain interval, TiKV inspects the latency of the Raftstore component. This configuration item specifies the interval of the inspection. The default value is `500ms`. | -| TiKV configuration file | [`raftstore.max-peer-down-duration`](/tikv-configuration-file.md#max-peer-down-duration) | Modified | Indicates the longest inactive duration allowed for a peer. A peer with timeout is marked as `down`, and PD tries to delete it later. The default value is changed from `5m` to `10m`. | -| TiKV configuration file | [`server.raft-client-queue-size`](/tikv-configuration-file.md#raft-client-queue-size) | Newly added | Specifies the queue size of the Raft messages in TiKV. The default value is `8192`. | -| TiKV configuration file | [`storage.flow-control.enable`](/tikv-configuration-file.md#enable) | Newly added | Determines whether to enable the flow control mechanism. The default value is `true`. | -| TiKV configuration file | [`storage.flow-control.memtables-threshold`](/tikv-configuration-file.md#memtables-threshold) | Newly added | When the number of kvDB memtables reaches this threshold, the flow control mechanism starts to work. The default value is `5`. | -| TiKV configuration file | [`storage.flow-control.l0-files-threshold`](/tikv-configuration-file.md#l0-files-threshold) | Newly added | When the number of kvDB L0 files reaches this threshold, the flow control mechanism starts to work. The default value is `9`. | -| TiKV configuration file | [`storage.flow-control.soft-pending-compaction-bytes-limit`](/tikv-configuration-file.md#soft-pending-compaction-bytes-limit) | Newly added | When the pending compaction bytes in KvDB reach this threshold, the flow control mechanism starts to reject some write requests and reports the `ServerIsBusy` error. The default value is "192GB". | -| TiKV configuration file | [`storage.flow-control.hard-pending-compaction-bytes-limit`](/tikv-configuration-file.md#hard-pending-compaction-bytes-limit) | Newly added | When the pending compaction bytes in KvDB reach this threshold, the flow control mechanism rejects all write requests and reports the `ServerIsBusy` error. The default value is "1024GB". | +| Configuration file | Configuration item | Change type | Description | +| :---------- | :----------- | :----------- | :----------- | +| TiDB configuration file | [`pessimistic-txn.deadlock-history-collect-retryable`](/tidb-configuration-file.md#deadlock-history-collect-retryable) | Newly added | Controls whether the [`INFORMATION\_SCHEMA.DEADLOCKS`](/information-schema/information-schema-deadlocks.md) table collects retryable deadlock error messages or not. | +| TiDB configuration file | [`security.auto-tls`](/tidb-configuration-file.md#auto-tls) | Newly added | Determines whether to automatically generate the TLS certificates on startup. The default value is `true`. | +| TiDB configuration file | [`stmt-summary.max-stmt-count`](/tidb-configuration-file.md#max-stmt-count) | Modified | Indicates the maximum number of SQL categories allowed to be saved in the statement summary tables. The default value is changed from `200` to `3000`. | +| TiDB configuration file | `experimental.allow-expression-index` | Deprecated | The `allow-expression-index` configuration in the TiDB configuration file is deprecated. | +| TiKV configuration file | [`raftstore.cmd-batch`](/tikv-configuration-file.md#cmd-batch) | Newly added | Controls whether to enable batch processing of the requests. When it is enabled, the write performance is significantly improved. The default value is `true`. | +| TiKV configuration file | [`raftstore.inspect-interval`](/tikv-configuration-file.md#inspect-interval) | Newly added | At a certain interval, TiKV inspects the latency of the Raftstore component. This configuration item specifies the interval of the inspection. The default value is `500ms`. | +| TiKV configuration file | [`raftstore.max-peer-down-duration`](/tikv-configuration-file.md#max-peer-down-duration) | Modified | Indicates the longest inactive duration allowed for a peer. A peer with timeout is marked as `down`, and PD tries to delete it later. The default value is changed from `5m` to `10m`. | +| TiKV configuration file | [`server.raft-client-queue-size`](/tikv-configuration-file.md#raft-client-queue-size) | Newly added | Specifies the queue size of the Raft messages in TiKV. The default value is `8192`. | +| TiKV configuration file | [`storage.flow-control.enable`](/tikv-configuration-file.md#enable) | Newly added | Determines whether to enable the flow control mechanism. The default value is `true`. | +| TiKV configuration file | [`storage.flow-control.memtables-threshold`](/tikv-configuration-file.md#memtables-threshold) | Newly added | When the number of kvDB memtables reaches this threshold, the flow control mechanism starts to work. The default value is `5`. | +| TiKV configuration file | [`storage.flow-control.l0-files-threshold`](/tikv-configuration-file.md#l0-files-threshold) | Newly added | When the number of kvDB L0 files reaches this threshold, the flow control mechanism starts to work. The default value is `9`. | +| TiKV configuration file | [`storage.flow-control.soft-pending-compaction-bytes-limit`](/tikv-configuration-file.md#soft-pending-compaction-bytes-limit) | Newly added | When the pending compaction bytes in KvDB reach this threshold, the flow control mechanism starts to reject some write requests and reports the `ServerIsBusy` error. The default value is "192GB". | +| TiKV configuration file | [`storage.flow-control.hard-pending-compaction-bytes-limit`](/tikv-configuration-file.md#hard-pending-compaction-bytes-limit) | Newly added | When the pending compaction bytes in KvDB reach this threshold, the flow control mechanism rejects all write requests and reports the `ServerIsBusy` error. The default value is "1024GB". | ### Others @@ -100,7 +100,7 @@ In v5.2, the key new features and improvements are as follows: Add pruning rules for index selection. Before using the statistics for comparison, TiDB uses these rules to narrow down the scope of possible indexes to be selected, which reduces the possibility of selecting non-optimal indexes. [User document](/choose-index.md) - + ### Transaction - **General availability (GA) for Lock View** @@ -128,61 +128,61 @@ In v5.2, the key new features and improvements are as follows: - **Add TiFlash I/O traffic limit** This new feature is suitable for cloud storage with disk bandwidth of a small and specific size. It is disabled by default. - + TiFlash IO Rate Limiter provides a new mechanism to avoid excessive race for I/O resources between read and write tasks. It balances the responses to read and write tasks, and limits the rate automatically according to read/write workload. - + [User document](/tiflash/tiflash-configuration.md) - **Improve stability of TiKV flow control** - + TiKV introduces a new flow control mechanism to replace the previous RocksDB write stall mechanism. Compared with the write stall mechanism, this new mechanism reduces the impact on the stability of foreground write. In specific, when the stress of RocksDB compaction accumulates, flow control is performed at the TiKV scheduler layer instead of the RocksDB layer, to avoid the following issues: - + - Raftstore is stuck, which is caused by RocksDB write stall. - - Raft election times out, and the node leader is transferred as a result. + - Raft election times out, and the node leader is transferred as a result. This new mechanism improves the flow control algorithm to mitigate QPS decrease when the write traffic is high. - + [User document](/tikv-configuration-file.md#storageflow-control), [#10137](https://github.com/tikv/tikv/issues/10137) - **Detect and recover automatically from impact caused by a single slow TiKV node in a cluster** - + TiKV introduces the slow node detection mechanism. This mechanism calculates a score by inspecting the rate of TiKV Raftstore, and then reports the score to PD through store heartbeats. Meanwhile, it adds the `evict-slow-store-scheduler` scheduler on PD to automatically evict the leader on a single slow TiKV node. In this way, the impact on the whole cluster is mitigated. At the same time, more alert items about slow nodes are introduced to help you quickly pinpoint and solve problems. - + [User document]( /tikv-configuration-file.md#inspect-interval), [#10539](https://github.com/tikv/tikv/issues/10539) ### Data Migration - + - **Simplify operations of Data Migration (DM)** - + DM v2.0.6 can automatically identify the change event (failover or plan change) of the data source using VIP, and can automatically connect to a new data source instance, to reduce data replication latency and simplify operation procedures. - + - TiDB Lightning supports customized line terminators in the CSV data, and is compatible with the MySQL LOAD DATA CSV data formats. You can then use TiDB Lightning directly in your data flow architecture. [#1297](https://github.com/pingcap/br/pull/1297) ### TiDB data share subscription - + TiCDC supports using the HTTP protocol (OpenAPI) to manage TiCDC tasks, which is a more user-friendly operation method for both Kubernetes and on-premises environments. (Experimental feature) - + [#2411](https://github.com/pingcap/ticdc/issues/2411) - + ### Deployment and operations - + Support running the `tiup playground` command on Mac computers with Apple M1 chips. ### Telemetry - + Telemetry supports collecting usage data of specific features, such as the usage data of built-in functions. - + For detailed information of the collected data and how to disable data collection, refer to [Telemetry](/telemetry.md). ## Feature Enhancements - + + Tools - + + TiCDC - + - Add HTTP API. Support querying and modifying TiCDC clusters. [#2416](https://github.com/pingcap/ticdc/pull/2416) - Add the binary MQ format designed for TiDB. It is more compact than the open protocols based on JSON [#1621](https://github.com/pingcap/ticdc/pull/1621) - Remove support for file sorter [#2114](https://github.com/pingcap/ticdc/pull/2114) @@ -235,7 +235,6 @@ For detailed information of the collected data and how to disable data collectio - Add other functions: `INET_NTOA()`, `INET_ATON()`, `INET6_ATON`, `INET6_NTOA()` - Support Shuffled Hash Join calculation and Shuffled Hash Aggregation calculation in the MPP mode when a new collation is enabled - Optimize basic code to improve MPP performance - - Add `-1` as the positive infinity value for `tidb_broadcast_join_threshold_count` and `tidb_broadcast_join_threshold_size` - Support casting the `STRING` type to the `DOUBLE` type - Optimize the non-joined data in right outer join using multiple threads - Support automatically invalidating stale Regions in MPP queries @@ -305,15 +304,15 @@ For detailed information of the collected data and how to disable data collectio + TiCDC - - Fix a bug that TiCDC owner exits abnormally when refreshing the checkpoint [#1985](https://github.com/pingcap/ticdc/pull/1985) - - Fix a bug that changefeed fails immediately after its successful creation [#2115](https://github.com/pingcap/ticdc/pull/2115) - - Fix a bug that changefeed fails due to the invalid format of rules filter [#2117](https://github.com/pingcap/ticdc/pull/2117) - - Fix the potential DDL loss issue when the TiCDC owner panics [#2252](https://github.com/pingcap/ticdc/pull/2252) + - Fix a bug that TiCDC owner exits abnormally when refreshing the checkpoint [#1902](https://github.com/pingcap/ticdc/issues/1902) + - Fix a bug that changefeed fails immediately after its successful creation [#2113](https://github.com/pingcap/ticdc/issues/2113) + - Fix a bug that changefeed fails due to the invalid format of rules filter [#1625](https://github.com/pingcap/ticdc/issues/1625) + - Fix the potential DDL loss issue when the TiCDC owner panics [#1260](https://github.com/pingcap/ticdc/issues/1260) - Fix the CLI compatibility issue with 4.0.x clusters on the default sort-engine option [#2385](https://github.com/pingcap/ticdc/pull/2385) - - Fix a bug that changefeed might be reset unexpectedly when TiCDC gets the `ErrSchemaStorageTableMiss` error [#2423](https://github.com/pingcap/ticdc/pull/2423) - - Fix a bug that changefeed cannot be removed when TiCDC gets the `ErrGCTTLExceeded` error [#2429](https://github.com/pingcap/ticdc/pull/2429) - - Fix a bug that TiCDC fails to synchronize large tables to cdclog [#2431](https://github.com/pingcap/ticdc/pull/2431) - - Fix a bug that multiple processors might write data to the same table when TiCDC is rescheduling the table [#2417](https://github.com/pingcap/ticdc/pull/2417) + - Fix a bug that changefeed might be reset unexpectedly when TiCDC gets the `ErrSchemaStorageTableMiss` error [#2422](https://github.com/pingcap/ticdc/issues/2422) + - Fix a bug that changefeed cannot be removed when TiCDC gets the `ErrGCTTLExceeded` error [#2391](https://github.com/pingcap/ticdc/issues/2391) + - Fix a bug that TiCDC fails to synchronize large tables to cdclog [#1259](https://github.com/pingcap/ticdc/issues/1259) [#2424](https://github.com/pingcap/ticdc/issues/2424) + - Fix a bug that multiple processors might write data to the same table when TiCDC is rescheduling the table [#2230](https://github.com/pingcap/ticdc/issues/2230) + Backup & Restore (BR) @@ -322,17 +321,13 @@ For detailed information of the collected data and how to disable data collectio + TiDB Lightning - - Fix a bug that Lightning fails to parse the `DECIMAL` data type in Parquet file [#1272](https://github.com/pingcap/br/pull/1272) - - Fix a bug that Lightning reports the "Error 9007: Write conflict" error when restoring table schemas [#1290](https://github.com/pingcap/br/issues/1290) - - Fix a bug that Lightning fails to import data due to the overflow of int handle [#1291](https://github.com/pingcap/br/issues/1291) - - Fix a bug that Lightning might get a checksum mismatching error due to data loss in the local backend mode [#1413](https://github.com/pingcap/br/pull/1413) - - Fix the Lighting incompatibility issue with clustered index when Lightning is restoring table schemas [#1364](https://github.com/pingcap/br/pull/1364) - + - Fix a bug that TiDB Lightning fails to parse the `DECIMAL` data type in Parquet file [#1272](https://github.com/pingcap/br/pull/1272) + - Fix a bug that TiDB Lightning reports the "Error 9007: Write conflict" error when restoring table schemas [#1290](https://github.com/pingcap/br/issues/1290) + - Fix a bug that TiDB Lightning fails to import data due to the overflow of int handle [#1291](https://github.com/pingcap/br/issues/1291) + - Fix a bug that TiDB Lightning might get a checksum mismatching error due to data loss in the local backend mode [#1403](https://github.com/pingcap/br/issues/1403) + - Fix the Lighting incompatibility issue with clustered index when TiDB Lightning is restoring table schemas [#1362](https://github.com/pingcap/br/issues/1362) + Dumpling - Fix a bug that the data export fails because the Dumpling GC safepoint is set too late [#290](https://github.com/pingcap/dumpling/pull/290) - - Fix the Dumpling getting stuck issue when exporting table names from the upstream database in certain MySQL versions [#325](https://github.com/pingcap/dumpling/pull/325) - - - + - Fix the Dumpling getting stuck issue when exporting table names from the upstream database in certain MySQL versions [#322](https://github.com/pingcap/dumpling/issues/322) From ee2dcfbfee515e60c33e4aa1caf17f2c97313272 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 25 Aug 2021 19:04:42 +0800 Subject: [PATCH 10/29] Update releases/release-5.2.0.md --- releases/release-5.2.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 2cbbc1609ff36..7055063d813b3 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -139,8 +139,8 @@ In v5.2, the key new features and improvements are as follows: In specific, when the stress of RocksDB compaction accumulates, flow control is performed at the TiKV scheduler layer instead of the RocksDB layer, to avoid the following issues: - - Raftstore is stuck, which is caused by RocksDB write stall. - - Raft election times out, and the node leader is transferred as a result. + - Raftstore is stuck, which is caused by RocksDB write stall. + - Raft election times out, and the node leader is transferred as a result. This new mechanism improves the flow control algorithm to mitigate QPS decrease when the write traffic is high. From 3decd54aea05cc6ff15afc54990afb5f97462cd7 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 25 Aug 2021 19:11:28 +0800 Subject: [PATCH 11/29] refine format --- releases/release-5.2.0.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 7055063d813b3..0f66fc71d4f3f 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -183,17 +183,18 @@ For detailed information of the collected data and how to disable data collectio + TiCDC - - Add HTTP API. Support querying and modifying TiCDC clusters. [#2416](https://github.com/pingcap/ticdc/pull/2416) - - Add the binary MQ format designed for TiDB. It is more compact than the open protocols based on JSON [#1621](https://github.com/pingcap/ticdc/pull/1621) - - Remove support for file sorter [#2114](https://github.com/pingcap/ticdc/pull/2114) - - Support log rotation configurations [#2182](https://github.com/pingcap/ticdc/pull/2182) + - Add HTTP API. Support querying and modifying TiCDC clusters. [#2416](https://github.com/pingcap/ticdc/pull/2416) + - Add the binary MQ format designed for TiDB. It is more compact than the open protocols based on JSON [#1621](https://github.com/pingcap/ticdc/pull/1621) + - Remove support for file sorter [#2114](https://github.com/pingcap/ticdc/pull/2114) + - Support log rotation configurations [#2182](https://github.com/pingcap/ticdc/pull/2182) + TiDB Lightning - - Support customized line terminators (except `\r` and `\n`) [#1297](https://github.com/pingcap/br/pull/1297) - - Support expression index and the index that depends on virtual generated columns [#1407](https://github.com/pingcap/br/pull/1407) + - Support customized line terminators (except `\r` and `\n`) [#1297](https://github.com/pingcap/br/pull/1297) + - Support expression index and the index that depends on virtual generated columns [#1407](https://github.com/pingcap/br/pull/1407) + Dumpling + - Support backing up MySQL compatible databases but does not support `START TRANSACTION ... WITH CONSISTENT SNAPSHOT` or`SHOW CREATE TABLE` [#311](https://github.com/pingcap/dumpling/pull/311) ## Improvements @@ -242,6 +243,7 @@ For detailed information of the collected data and how to disable data collectio + Tools + TiCDC + - Add the concurrency limit to the incremental scan of kv client [#1899](https://github.com/pingcap/ticdc/pull/1899) - TiCDC can always pull the old value internally [#2271](https://github.com/pingcap/ticdc/pull/2271) - TiCDC can fail and exit fast when unrecoverable DML errors occur [#1928](https://github.com/pingcap/ticdc/pull/1928) @@ -249,6 +251,7 @@ For detailed information of the collected data and how to disable data collectio - Optimize workerpool to reduce the number of goroutines under high concurrency [#2201](https://github.com/pingcap/ticdc/pull/2201) + Dumpling + - Support always splitting TiDB v3.x tables through `tidb_rowid` to save TiDB memory [#301](https://github.com/pingcap/dumpling/pull/301) - Reduce access of Dumpling to the `information_schema` to improve stability [#305](https://github.com/pingcap/dumpling/pull/305) From c4bdfd29c302a7f2308073b8ce23bff82c87aa9f Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 26 Aug 2021 19:59:36 +0800 Subject: [PATCH 12/29] Apply suggestions from code review Co-authored-by: Zhou Kunqin <25057648+time-and-fate@users.noreply.github.com> --- releases/release-5.2.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 0f66fc71d4f3f..cc5251e21df13 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -11,7 +11,7 @@ TiDB version: 5.2.0 In v5.2, the key new features and improvements are as follows: - Support expression index to greatly improve query performance -- Improve the accuracy of optimizer estimation (cardinality estimation) to select optimal execution plans +- Improve the accuracy of optimizer cardinality estimation to help to select optimal execution plans - Announce the general availability (GA) for the Lock View feature to observe transaction locking events and troubleshoot deadlock problems - Add the TiFlash I/O traffic limit feature to improve the stability of read and write for TiFlash - Optimize TiKV reserving space management to improve the stability of storage @@ -87,10 +87,10 @@ In v5.2, the key new features and improvements are as follows: [User document](/configure-memory-usage.md#other-memory-control-behaviors-of-tidb-server), [#25882](https://github.com/pingcap/tidb/issues/25882) -- **Improve the accuracy of optimizer estimation (cardinality estimation)** +- **Improve the accuracy of optimizer cardinality estimation** - Improve the accuracy of TiDB's estimation of TopN/Limit. For example, for pagination queries on a large table that contain the `order by col limit x` condition, TiDB can more easily select the right index and reduce query response time. - - Improve the accuracy of out-of-scope estimation. For example, even if the statistics for a day have not been updated, TiDB can accurately select the corresponding index for a query that contains `where date=Now()`. + - Improve the accuracy of out-of-range estimation. For example, even if the statistics for a day have not been updated, TiDB can accurately select the corresponding index for a query that contains `where date=Now()`. - Introduce the `tidb_opt_limit_push_down_threshold` variable to control the optimizer's behavior of pushing down Limit/TopN, which resolves the issue that Limit/TopN cannot be pushed down in some situations due to wrong estimation. [User document](/system-variables.md#tidb_opt_limit_push_down_threshold), [#26085](https://github.com/pingcap/tidb/issues/26085) From cf4b1a7c13705da23fde4b0389a96fce16e97ed3 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 26 Aug 2021 20:03:05 +0800 Subject: [PATCH 13/29] Update releases/release-5.2.0.md --- releases/release-5.2.0.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index cc5251e21df13..e14c652958f16 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -61,9 +61,6 @@ In v5.2, the key new features and improvements are as follows: - For TiDB clusters upgraded from v4.0 to v5.2, the default value of [`tidb_multi_statement_mode`](/system-variables.md#tidb_multi_statement_mode-new-in-v4011) changes from `WARN` to `OFF`. - Before the upgrade, check the value of the TiDB configuration [`feedback-probability`](/tidb-configuration-file.md#feedback-probability). If the value is not `0`, the "panic in the recoverable goroutine" error will occur after the upgrade, but this error does not affect the upgrade. - TiDB is now compatible with MySQL 5.7's noop variable `innodb_default_row_format`. Setting this variable has no effect. [#23541](https://github.com/pingcap/tidb/issues/23541) -- When upgraded to v5.2, TiKV will calculate `memory-usage-limit` based on `block-cache.capacity`, and the result of `memory-usage-limit` defaults to the value of "`block-cache.capacity` / 0.45 * 0.75". - - For example, when [storage.block-cache] capacity = "24GB", `memory-usage-limit` is 40GB (24GB/0.45*0.75 = 40GB). ## New features From dc8e75868e72884d2dbb3b7be432be28c09dbc2b Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 26 Aug 2021 21:52:21 +0800 Subject: [PATCH 14/29] Apply suggestions from code review Co-authored-by: Yujie Xia --- releases/release-5.2.0.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index e14c652958f16..cdc447ba73e7a 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -180,7 +180,6 @@ For detailed information of the collected data and how to disable data collectio + TiCDC - - Add HTTP API. Support querying and modifying TiCDC clusters. [#2416](https://github.com/pingcap/ticdc/pull/2416) - Add the binary MQ format designed for TiDB. It is more compact than the open protocols based on JSON [#1621](https://github.com/pingcap/ticdc/pull/1621) - Remove support for file sorter [#2114](https://github.com/pingcap/ticdc/pull/2114) - Support log rotation configurations [#2182](https://github.com/pingcap/ticdc/pull/2182) @@ -214,10 +213,6 @@ For detailed information of the collected data and how to disable data collectio - Add a new status variation `last_plan_binding_update_time` to view the timestamp corresponding to the binding cache in the current TiDB instance [#26340](https://github.com/pingcap/tidb/pull/26340) - Support reporting an error when starting binding evolution or running `admin evolve bindings` to ban the baseline evolution (currently disabled in the on-premises TiDB version because it is an experimental feature) affecting other features [#26333](https://github.com/pingcap/tidb/pull/26333) -+ TiKV - - - Limit memory consumption of the TiCDC sink [#10305](https://github.com/tikv/tikv/pull/10305) - - Add the memory-bounded upper limit for the TiCDC old value cache [#10313](https://github.com/tikv/tikv/pull/10313) + PD From 110c132438a95cf3811839e603fdd9a4e36cad4b Mon Sep 17 00:00:00 2001 From: Enwei Date: Thu, 26 Aug 2021 16:38:51 +0200 Subject: [PATCH 15/29] update missed lines --- releases/release-5.2.0.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index cdc447ba73e7a..9f5daa7a4ec0c 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -213,6 +213,10 @@ For detailed information of the collected data and how to disable data collectio - Add a new status variation `last_plan_binding_update_time` to view the timestamp corresponding to the binding cache in the current TiDB instance [#26340](https://github.com/pingcap/tidb/pull/26340) - Support reporting an error when starting binding evolution or running `admin evolve bindings` to ban the baseline evolution (currently disabled in the on-premises TiDB version because it is an experimental feature) affecting other features [#26333](https://github.com/pingcap/tidb/pull/26333) ++ TiKV + + - Limit memory consumption of the TiCDC sink [#10305](https://github.com/tikv/tikv/pull/10305) + - Add the memory-bounded upper limit for the TiCDC old value cache [#10313](https://github.com/tikv/tikv/pull/10313) + PD @@ -228,6 +232,7 @@ For detailed information of the collected data and how to disable data collectio - Add other functions: `INET_NTOA()`, `INET_ATON()`, `INET6_ATON`, `INET6_NTOA()` - Support Shuffled Hash Join calculation and Shuffled Hash Aggregation calculation in the MPP mode when a new collation is enabled - Optimize basic code to improve MPP performance + - Add `-1` as the positive infinity value for `tidb_broadcast_join_threshold_count` and `tidb_broadcast_join_threshold_size` - Support casting the `STRING` type to the `DOUBLE` type - Optimize the non-joined data in right outer join using multiple threads - Support automatically invalidating stale Regions in MPP queries From d6c585e2fd3af9ab8a093d321f7f0223d1e58604 Mon Sep 17 00:00:00 2001 From: qiancai Date: Fri, 27 Aug 2021 11:07:43 +0800 Subject: [PATCH 16/29] Update telemetry.md --- telemetry.md | 1 - 1 file changed, 1 deletion(-) diff --git a/telemetry.md b/telemetry.md index 6c6cc7941f7fc..800dad1987051 100644 --- a/telemetry.md +++ b/telemetry.md @@ -23,7 +23,6 @@ When the telemetry collection feature is enabled in TiDB, the TiDB cluster colle - Deployment characteristics, such as the size of hardware (CPU, memory, disk), TiDB components versions, OS name. - The status of query requests in the system, such as the number of query requests and the duration. - Component usage, for example, whether the Async Commit feature is in use or not. -- The usage information of built-in functions, including per-function statistics on how often the function gets called. The arguments for the functions are not included. To view the full content of the usage information shared to PingCAP, execute the following SQL statement: From 60f4afb5bc7a0bef0cf7dc60542feb249abe18ea Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 27 Aug 2021 11:09:29 +0800 Subject: [PATCH 17/29] Apply suggestions from code review --- releases/release-5.2.0.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 9f5daa7a4ec0c..9d49d841a3d1c 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -33,7 +33,6 @@ In v5.2, the key new features and improvements are as follows: | [`tidb_enable_auto_increment_in_generated`](/system-variables.md#tidb_enable_auto_increment_in_generated) | Newly added | Determines whether to include the `AUTO_INCREMENT` columns when creating a generated column or an expression index. The default value is `OFF`. | | [`tidb_opt_enable_correlation_adjustment`](/system-variables.md#tidb_opt_enable_correlation_adjustment) | Newly added | Controls whether the optimizer estimates the number of rows based on column order correlation. The default value is `ON`. | | [`tidb_opt_limit_push_down_threshold`](/system-variables.md#tidb_opt_limit_push_down_threshold) | Newly added | Sets the threshold that determines whether to push the Limit or TopN operator down to TiKV. The default value is `100`. | -| [`tidb_restricted_read_only`](/system-variables.md#tidb_restricted_read_only) | Newly added | Controls whether the whole cluster is read-only. The default value is `OFF`. | | [`tidb_stmt_summary_max_stmt_count`](/system-variables.md#tidb_stmt_summary_max_stmt_count-new-in-v40) | Modified | Sets the maximum number of statements that the statement summary tables store in memory. The default value is changed from `200` to `3000`. | | `tidb_enable_streaming` | Deprecated | The system variable `enable-streaming` is deprecated and it is not recommended to use it any more. | @@ -168,12 +167,6 @@ TiCDC supports using the HTTP protocol (OpenAPI) to manage TiCDC tasks, which is Support running the `tiup playground` command on Mac computers with Apple M1 chips. -### Telemetry - -Telemetry supports collecting usage data of specific features, such as the usage data of built-in functions. - -For detailed information of the collected data and how to disable data collection, refer to [Telemetry](/telemetry.md). - ## Feature Enhancements + Tools From 8e4b50cf3cd27084fd57de335a844ce6ef101076 Mon Sep 17 00:00:00 2001 From: qiancai Date: Fri, 27 Aug 2021 11:40:12 +0800 Subject: [PATCH 18/29] Update release-5.2.0.md --- releases/release-5.2.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 9d49d841a3d1c..03f8b5a964cee 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -155,7 +155,8 @@ In v5.2, the key new features and improvements are as follows: DM v2.0.6 can automatically identify the change event (failover or plan change) of the data source using VIP, and can automatically connect to a new data source instance, to reduce data replication latency and simplify operation procedures. - TiDB Lightning supports customized line terminators in the CSV data, and is compatible with the MySQL LOAD DATA CSV data formats. You can then use TiDB Lightning directly in your data flow architecture. -[#1297](https://github.com/pingcap/br/pull/1297) + + [#1297](https://github.com/pingcap/br/pull/1297) ### TiDB data share subscription @@ -278,8 +279,7 @@ Support running the `tiup playground` command on Mac computers with Apple M1 chi + PD - - Fix the issue that the expected scheduling cannot be generated due to scheduling conflicts among multiple schedulers - [#3807](https://github.com/tikv/pd/issues/3807) [#3778](https://github.com/tikv/pd/issues/3778) + - Fix the issue that the expected scheduling cannot be generated due to scheduling conflicts among multiple schedulers [#3807](https://github.com/tikv/pd/issues/3807) [#3778](https://github.com/tikv/pd/issues/3778) + TiFlash From f3a487f73025dfd595bcb63ead1f6e21e4fa40e4 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 27 Aug 2021 13:42:41 +0800 Subject: [PATCH 19/29] Update releases/release-5.2.0.md --- releases/release-5.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 03f8b5a964cee..023451166bf79 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -301,7 +301,7 @@ Support running the `tiup playground` command on Mac computers with Apple M1 chi - Fix a bug that changefeed fails immediately after its successful creation [#2113](https://github.com/pingcap/ticdc/issues/2113) - Fix a bug that changefeed fails due to the invalid format of rules filter [#1625](https://github.com/pingcap/ticdc/issues/1625) - Fix the potential DDL loss issue when the TiCDC owner panics [#1260](https://github.com/pingcap/ticdc/issues/1260) - - Fix the CLI compatibility issue with 4.0.x clusters on the default sort-engine option [#2385](https://github.com/pingcap/ticdc/pull/2385) + - Fix the CLI compatibility issue with 4.0.x clusters on the default sort-engine option [#2373](https://github.com/pingcap/ticdc/issues/2373) - Fix a bug that changefeed might be reset unexpectedly when TiCDC gets the `ErrSchemaStorageTableMiss` error [#2422](https://github.com/pingcap/ticdc/issues/2422) - Fix a bug that changefeed cannot be removed when TiCDC gets the `ErrGCTTLExceeded` error [#2391](https://github.com/pingcap/ticdc/issues/2391) - Fix a bug that TiCDC fails to synchronize large tables to cdclog [#1259](https://github.com/pingcap/ticdc/issues/1259) [#2424](https://github.com/pingcap/ticdc/issues/2424) From 17b4223eeb4376f713ccaebe769a09d8f362e36c Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 27 Aug 2021 13:44:46 +0800 Subject: [PATCH 20/29] Update releases/release-5.2.0.md Co-authored-by: Yujie Xia --- releases/release-5.2.0.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 023451166bf79..87533bf8b33c4 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -207,10 +207,6 @@ Support running the `tiup playground` command on Mac computers with Apple M1 chi - Add a new status variation `last_plan_binding_update_time` to view the timestamp corresponding to the binding cache in the current TiDB instance [#26340](https://github.com/pingcap/tidb/pull/26340) - Support reporting an error when starting binding evolution or running `admin evolve bindings` to ban the baseline evolution (currently disabled in the on-premises TiDB version because it is an experimental feature) affecting other features [#26333](https://github.com/pingcap/tidb/pull/26333) -+ TiKV - - - Limit memory consumption of the TiCDC sink [#10305](https://github.com/tikv/tikv/pull/10305) - - Add the memory-bounded upper limit for the TiCDC old value cache [#10313](https://github.com/tikv/tikv/pull/10313) + PD From f3a9346620d7548ff0257dada80949db1bc75911 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 27 Aug 2021 13:56:16 +0800 Subject: [PATCH 21/29] Update releases/release-5.2.0.md --- releases/release-5.2.0.md | 1 - 1 file changed, 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 87533bf8b33c4..02447cbde545b 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -222,7 +222,6 @@ Support running the `tiup playground` command on Mac computers with Apple M1 chi - Add other functions: `INET_NTOA()`, `INET_ATON()`, `INET6_ATON`, `INET6_NTOA()` - Support Shuffled Hash Join calculation and Shuffled Hash Aggregation calculation in the MPP mode when a new collation is enabled - Optimize basic code to improve MPP performance - - Add `-1` as the positive infinity value for `tidb_broadcast_join_threshold_count` and `tidb_broadcast_join_threshold_size` - Support casting the `STRING` type to the `DOUBLE` type - Optimize the non-joined data in right outer join using multiple threads - Support automatically invalidating stale Regions in MPP queries From b64fa7651329152dd00a25166c9208cadc201fd3 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 27 Aug 2021 15:54:33 +0800 Subject: [PATCH 22/29] Update releases/release-5.2.0.md Co-authored-by: Grace Cai --- releases/release-5.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 02447cbde545b..b5b69f2ceeb3b 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -125,7 +125,7 @@ In v5.2, the key new features and improvements are as follows: This new feature is suitable for cloud storage with disk bandwidth of a small and specific size. It is disabled by default. - TiFlash IO Rate Limiter provides a new mechanism to avoid excessive race for I/O resources between read and write tasks. It balances the responses to read and write tasks, and limits the rate automatically according to read/write workload. + TiFlash I/O Rate Limiter provides a new mechanism to avoid excessive race for I/O resources between read and write tasks. It balances the responses to read and write tasks, and limits the rate automatically according to read/write workload. [User document](/tiflash/tiflash-configuration.md) From ea19a822fc5267a1030d04f301595b3d60809518 Mon Sep 17 00:00:00 2001 From: qiancai Date: Fri, 27 Aug 2021 19:07:01 +0800 Subject: [PATCH 23/29] Update tidb-configuration-file.md --- tidb-configuration-file.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-configuration-file.md b/tidb-configuration-file.md index 40867686bccb0..0f23c8c3c39fc 100644 --- a/tidb-configuration-file.md +++ b/tidb-configuration-file.md @@ -328,7 +328,7 @@ Configuration items related to security. ### `auto-tls` - Determines whether to automatically generate the TLS certificates on startup. -- Default value: `true` +- Default value: `false` ## Performance From 46d981e7d9a653cd327678551548bc7d47ca25bf Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 27 Aug 2021 19:11:38 +0800 Subject: [PATCH 24/29] Update releases/release-5.2.0.md Co-authored-by: Grace Cai --- releases/release-5.2.0.md | 1 - 1 file changed, 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index b5b69f2ceeb3b..35f76e33018b3 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -207,7 +207,6 @@ Support running the `tiup playground` command on Mac computers with Apple M1 chi - Add a new status variation `last_plan_binding_update_time` to view the timestamp corresponding to the binding cache in the current TiDB instance [#26340](https://github.com/pingcap/tidb/pull/26340) - Support reporting an error when starting binding evolution or running `admin evolve bindings` to ban the baseline evolution (currently disabled in the on-premises TiDB version because it is an experimental feature) affecting other features [#26333](https://github.com/pingcap/tidb/pull/26333) - + PD - Add more QPS dimensions for hot Region scheduling, and support adjusting the priority of the scheduling [#3869](https://github.com/tikv/pd/issues/3869) From 8984c17a9e8ca2b5ec310bc34a0509f32972c955 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 27 Aug 2021 19:11:48 +0800 Subject: [PATCH 25/29] Update releases/release-5.2.0.md Co-authored-by: Grace Cai --- releases/release-5.2.0.md | 1 + 1 file changed, 1 insertion(+) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 35f76e33018b3..582010d6a0ac0 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -60,6 +60,7 @@ In v5.2, the key new features and improvements are as follows: - For TiDB clusters upgraded from v4.0 to v5.2, the default value of [`tidb_multi_statement_mode`](/system-variables.md#tidb_multi_statement_mode-new-in-v4011) changes from `WARN` to `OFF`. - Before the upgrade, check the value of the TiDB configuration [`feedback-probability`](/tidb-configuration-file.md#feedback-probability). If the value is not `0`, the "panic in the recoverable goroutine" error will occur after the upgrade, but this error does not affect the upgrade. - TiDB is now compatible with MySQL 5.7's noop variable `innodb_default_row_format`. Setting this variable has no effect. [#23541](https://github.com/pingcap/tidb/issues/23541) +Starting from TiDB 5.2, to improve system security, it is recommended (but not mandatory) to encrypt the transport layer for connections from clients. TiDB provides the Auto TLS feature to automatically configure and enable encryption in TiDB. To use the Auto TLS feature, before or after the TiDB upgrade, set [`security.auto-tls`](/tidb-configuration-file.md#auto-tls) in the TiDB configuration file to `true`. ## New features From aa9e30d5d1f6e61568307d1815249b7729544623 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 27 Aug 2021 19:11:58 +0800 Subject: [PATCH 26/29] Update releases/release-5.2.0.md --- releases/release-5.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 582010d6a0ac0..7b2468ed598bc 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -118,7 +118,7 @@ Starting from TiDB 5.2, to improve system security, it is recommended (but not m - View the deadlock errors recently occurred on a TiDB node: [`DEADLOCKS`](/information-schema/information-schema-deadlocks.md) - View the executing transaction on a TiDB node: [`TIDB_TRX`](/information-schema/information-schema-tidb-trx.md) - - Optimize the user scenarios of adding indexes on tables with the `AUTO_RANDOM` or `SHARD_ROW_ID_BITS` attribute. +- Optimize the user scenarios of adding indexes on tables with the `AUTO_RANDOM` or `SHARD_ROW_ID_BITS` attribute. ### Stability From 0d7bcdf5e1968d8acce2c60079ce693e6409b6d2 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 27 Aug 2021 19:20:25 +0800 Subject: [PATCH 27/29] Update releases/release-5.2.0.md Co-authored-by: Grace Cai --- releases/release-5.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index 7b2468ed598bc..bd6e237425f64 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -60,7 +60,7 @@ In v5.2, the key new features and improvements are as follows: - For TiDB clusters upgraded from v4.0 to v5.2, the default value of [`tidb_multi_statement_mode`](/system-variables.md#tidb_multi_statement_mode-new-in-v4011) changes from `WARN` to `OFF`. - Before the upgrade, check the value of the TiDB configuration [`feedback-probability`](/tidb-configuration-file.md#feedback-probability). If the value is not `0`, the "panic in the recoverable goroutine" error will occur after the upgrade, but this error does not affect the upgrade. - TiDB is now compatible with MySQL 5.7's noop variable `innodb_default_row_format`. Setting this variable has no effect. [#23541](https://github.com/pingcap/tidb/issues/23541) -Starting from TiDB 5.2, to improve system security, it is recommended (but not mandatory) to encrypt the transport layer for connections from clients. TiDB provides the Auto TLS feature to automatically configure and enable encryption in TiDB. To use the Auto TLS feature, before or after the TiDB upgrade, set [`security.auto-tls`](/tidb-configuration-file.md#auto-tls) in the TiDB configuration file to `true`. +- Starting from TiDB 5.2, to improve system security, it is recommended (but not mandatory) to encrypt the transport layer for connections from clients. TiDB provides the Auto TLS feature to automatically configure and enable encryption in TiDB. To use the Auto TLS feature, before the TiDB upgrade, set [`security.auto-tls`](/tidb-configuration-file.md#auto-tls) in the TiDB configuration file to `true`. ## New features From f09ed033adc3f763c8a98fd7f5adb905cc749b20 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 27 Aug 2021 19:20:50 +0800 Subject: [PATCH 28/29] Update releases/release-5.2.0.md --- releases/release-5.2.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.2.0.md b/releases/release-5.2.0.md index bd6e237425f64..136e409b2af4a 100644 --- a/releases/release-5.2.0.md +++ b/releases/release-5.2.0.md @@ -41,7 +41,7 @@ In v5.2, the key new features and improvements are as follows: | Configuration file | Configuration item | Change type | Description | | :---------- | :----------- | :----------- | :----------- | | TiDB configuration file | [`pessimistic-txn.deadlock-history-collect-retryable`](/tidb-configuration-file.md#deadlock-history-collect-retryable) | Newly added | Controls whether the [`INFORMATION\_SCHEMA.DEADLOCKS`](/information-schema/information-schema-deadlocks.md) table collects retryable deadlock error messages or not. | -| TiDB configuration file | [`security.auto-tls`](/tidb-configuration-file.md#auto-tls) | Newly added | Determines whether to automatically generate the TLS certificates on startup. The default value is `true`. | +| TiDB configuration file | [`security.auto-tls`](/tidb-configuration-file.md#auto-tls) | Newly added | Determines whether to automatically generate the TLS certificates on startup. The default value is `false`. | | TiDB configuration file | [`stmt-summary.max-stmt-count`](/tidb-configuration-file.md#max-stmt-count) | Modified | Indicates the maximum number of SQL categories allowed to be saved in the statement summary tables. The default value is changed from `200` to `3000`. | | TiDB configuration file | `experimental.allow-expression-index` | Deprecated | The `allow-expression-index` configuration in the TiDB configuration file is deprecated. | | TiKV configuration file | [`raftstore.cmd-batch`](/tikv-configuration-file.md#cmd-batch) | Newly added | Controls whether to enable batch processing of the requests. When it is enabled, the write performance is significantly improved. The default value is `true`. | From 3eb3bc51e9df8027d5ba9e44c38abcf6c820e376 Mon Sep 17 00:00:00 2001 From: qiancai Date: Fri, 27 Aug 2021 19:29:29 +0800 Subject: [PATCH 29/29] update --- TOC.md | 4 +++- _index.md | 2 +- releases/release-notes.md | 4 ++++ 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/TOC.md b/TOC.md index e35284e7a9c93..0a9675f591a44 100644 --- a/TOC.md +++ b/TOC.md @@ -7,7 +7,7 @@ + About TiDB + [TiDB Introduction](/overview.md) - + [TiDB 5.1 Release Notes](/releases/release-5.1.0.md) + + [TiDB 5.2 Release Notes](/releases/release-5.2.0.md) + [Basic Features](/basic-features.md) + [Experimental Features](/experimental-features.md) + Benchmarks @@ -552,6 +552,8 @@ + [Glossary](/glossary.md) + Release Notes + [All Releases](/releases/release-notes.md) + + v5.2 + + [5.2.0](/releases/release-5.2.0.md) + v5.1 + [5.1.1](/releases/release-5.1.1.md) + [5.1.0](/releases/release-5.1.0.md) diff --git a/_index.md b/_index.md index 595ddf554edf8..aa24270c21bca 100644 --- a/_index.md +++ b/_index.md @@ -15,7 +15,7 @@ Designed for the cloud, TiDB provides flexible scalability, reliability and secu - [TiDB Introduction](/overview.md) - [Basic Features](/basic-features.md) -- [TiDB 5.1 Release Notes](/releases/release-5.1.0.md) +- [TiDB 5.2 Release Notes](/releases/release-5.2.0.md) - [Compatibility with MySQL](/mysql-compatibility.md) - [Usage Limitations](/tidb-limitations.md) - [TiDB Adopters](/adopters.md) diff --git a/releases/release-notes.md b/releases/release-notes.md index a4adeba3aaee2..682382149c4bb 100644 --- a/releases/release-notes.md +++ b/releases/release-notes.md @@ -4,6 +4,10 @@ title: Release Notes # TiDB Release Notes +## 5.2 + +- [5.2.0](/releases/release-5.2.0.md) + ## 5.1 - [5.1.1](/releases/release-5.1.1.md)