diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index dae71379e682e..e402b0313569f 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -12,6 +12,12 @@ +**Tips for choosing the affected version(s):** + +By default, **CHOOSE MASTER ONLY** so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, **CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER**. + +For details, see [tips for choosing the affected versions](https://github.com/pingcap/docs/blob/master/CONTRIBUTING.md#guideline-for-choosing-the-affected-versions). + - [ ] master (the latest development version) - [ ] v5.1 (TiDB 5.1 versions) - [ ] v5.0 (TiDB 5.0 versions) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 476b6d6d559e2..6c8c464332407 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,12 +29,12 @@ Before you contribute, please take a quick look at some general information abou ### Learn about docs versions -Currently, we maintain six versions of TiDB documentation, each with a separate branch: +Currently, we maintain seven versions of TiDB documentation, each with a separate branch: | Docs branch name | Version description | | :--- | :--- | | `master` branch | the latest development version | -| `release-5.1` branch | the 5.1 version | +| `release-5.1` branch | the 5.1 stable version | | `release-5.0` branch | the 5.0 stable version | | `release-4.0` branch | the 4.0 stable version | | `release-3.1` branch | the 3.1 stable version | @@ -45,22 +45,24 @@ Currently, we maintain six versions of TiDB documentation, each with a separate > > Previously, we maintain all versions in the `master` branch, with directories like `dev` (the latest development version), `v3.0` and so on. Each docs version is updated very frequently and changes to one version often apply to another version or other versions as well. > -> Since February 21, 2020, to reduce manual editing and updating work among versions, we have started to maintain each version in a separate branch and introduce sre-bot to automatically file PRs to other versions as long as you add corresponding cherry-pick labels to your PR. +> Since February 21, 2020, to reduce manual editing and updating work among versions, we have started to maintain each version in a separate branch and introduced sre-bot (now ti-chi-bot) to automatically file PRs to other versions as long as you add corresponding cherry-pick labels to your PR. ### Use cherry-pick labels - If your changes apply to only one docs version, just submit a PR to the corresponding version branch. -- If your changes apply to multiple docs versions, you don't have to submit a PR to each branch. Instead, after you submit your PR, trigger the sre-bot to submit a PR to other version branches by adding one or several of the following labels as needed. Once the current PR is merged, sre-bot will start to work. - - `needs-cherry-pick-5.1` label: sre-bot will submit a PR to the `release-5.1` branch. - - `needs-cherry-pick-5.0` label: sre-bot will submit a PR to the `release-5.0` branch. - - `needs-cherry-pick-4.0` label: sre-bot will submit a PR to the `release-4.0` branch. - - `needs-cherry-pick-3.1` label: sre-bot will submit a PR to the `release-3.1` branch. - - `needs-cherry-pick-3.0` label: sre-bot will submit a PR to the `release-3.0` branch. - - `needs-cherry-pick-2.1` label: sre-bot will submit a PR to the `release-2.1` branch. - - `needs-cherry-pick-master` label: sre-bot will submit a PR to the `master` branch. +- If your changes apply to multiple docs versions, you don't have to submit a PR to each branch. Instead, after you submit your PR, trigger the ti-chi-bot to submit a PR to other version branches by adding one or several of the following labels as needed. Once the current PR is merged, ti-chi-bot will start to work. + - `needs-cherry-pick-5.1` label: ti-chi-bot will submit a PR to the `release-5.1` branch. + - `needs-cherry-pick-5.0` label: ti-chi-bot will submit a PR to the `release-5.0` branch. + - `needs-cherry-pick-4.0` label: ti-chi-bot will submit a PR to the `release-4.0` branch. + - `needs-cherry-pick-3.1` label: ti-chi-bot will submit a PR to the `release-3.1` branch. + - `needs-cherry-pick-3.0` label: ti-chi-bot will submit a PR to the `release-3.0` branch. + - `needs-cherry-pick-2.1` label: ti-chi-bot will submit a PR to the `release-2.1` branch. + - `needs-cherry-pick-master` label: ti-chi-bot will submit a PR to the `master` branch. -- If most of your changes apply to multiple docs versions but some differences exist among versions, you still can use cherry-pick labels to let sre-bot create PRs to other versions. After the PR to another version is successfully submitted by sre-bot, you can make changes to that PR. + For how to choose the docs versions, refer to [Guideline for choosing the affected version(s)](#guideline-for-choosing-the-affected-versions). + +- If most of your changes apply to multiple docs versions but some differences exist among versions, you still can use cherry-pick labels to let ti-chi-bot create PRs to other versions. After the PR to another version is successfully submitted by ti-chi-bot, you can make changes to that PR. ## How to contribute @@ -142,6 +144,23 @@ git push -u origin new-branch-name # "-u" is used to track the remote branch fro Now, your PR is successfully submitted! After this PR is merged, you will automatically become a contributor to TiDB documentation. +## Guideline for choosing the affected version(s) + +When you create a Pull Request, you need to choose the release version to which your document change applies in the description template on your Pull Request page. + +If your change fits one of the following situations, it is recommended to **CHOOSE THE MASTER BRANCH ONLY**. After the PR is merged, the change will be soon displayed on the [Dev page of the PingCAP documentation website](https://docs.pingcap.com/tidb/dev/). After the next major or minor version of TiDB is released, the change will also be displayed on the website page for the new version. + +- Relates to a documentation enhancement, such as supplementing missing or incomplete document contents. +- Fixes inaccurate or incorrect document contents, including values, descriptions, examples, or typos. +- Involves a documentation refactor in a specific topic module. + +If your change fits one of the following situations, **CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER**: + +- Involves a feature behavior change that relates to a specific version. +- Involves a compatibility change, including changing the default value of a configuration item or a system variable. +- Fixes format to resolve a display error +- Fixes broken links + ## Contact Join the Slack channel: [#sig-docs](https://slack.tidb.io/invite?team=tidb-community&channel=sig-docs&ref=pingcap-docs) diff --git a/TOC.md b/TOC.md index 316a3a506b19a..c9e5dff2e0a37 100644 --- a/TOC.md +++ b/TOC.md @@ -450,6 +450,7 @@ + [`METRICS_TABLES`](/information-schema/information-schema-metrics-tables.md) + [`PARTITIONS`](/information-schema/information-schema-partitions.md) + [`PROCESSLIST`](/information-schema/information-schema-processlist.md) + + [`REFERENTIAL_CONSTRAINTS`](/information-schema/information-schema-referential-constraints.md) + [`SCHEMATA`](/information-schema/information-schema-schemata.md) + [`SEQUENCES`](/information-schema/information-schema-sequences.md) + [`SESSION_VARIABLES`](/information-schema/information-schema-session-variables.md) diff --git a/br/use-br-command-line-tool.md b/br/use-br-command-line-tool.md index 25fc603e680d3..6921d73670a03 100644 --- a/br/use-br-command-line-tool.md +++ b/br/use-br-command-line-tool.md @@ -361,6 +361,10 @@ br restore db \ In the above command, `--db` specifies the name of the database to be restored. For descriptions of other options, see [Restore all backup data](#restore-all-the-backup-data)). +> **Note:** +> +> When you restore the backup data, the name of the database specified by `--db` must be the same as the one specified by `-- db` in the backup command. Otherwise, the restore fails. This is because the metafile of the backup data ( `backupmeta` file) records the database name, you can only restore data to the database with the same name. The recommended method is to restore the backup data to the database with the same name in another cluster. + ### Restore a table To restore a single table to the cluster, execute the `br restore table` command. To get help on this command, execute `br restore table -h` or `br restore table --help`. diff --git a/config-templates/complex-cdc.yaml b/config-templates/complex-cdc.yaml index 8c511e7a86ed5..9c4c48bcf56d5 100644 --- a/config-templates/complex-cdc.yaml +++ b/config-templates/complex-cdc.yaml @@ -98,16 +98,19 @@ cdc_servers: deploy_dir: "/tidb-deploy/cdc-8300" data_dir: "/tidb-data/cdc-8300" log_dir: "/tidb-deploy/cdc-8300/log" + # gc-ttl: 86400 - host: 10.0.1.2 port: 8300 deploy_dir: "/tidb-deploy/cdc-8300" data_dir: "/tidb-data/cdc-8300" log_dir: "/tidb-deploy/cdc-8300/log" + # gc-ttl: 86400 - host: 10.0.1.3 port: 8300 deploy_dir: "/tidb-deploy/cdc-8300" data_dir: "/tidb-data/cdc-8300" log_dir: "/tidb-deploy/cdc-8300/log" + # gc-ttl: 86400 monitoring_servers: - host: 10.0.1.10 diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 251a31f12d3be..554100b9f3a12 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -20,7 +20,7 @@ The current version of TiKV encryption has the following drawbacks. Be aware of * TiFlash supports encryption at rest since v4.0.5. For details, refer to [Encryption at Rest for TiFlash](#encryption-at-rest-for-tiflash-new-in-v405). When deploying TiKV with TiFlash earlier than v4.0.5, data stored in TiFlash is not encrypted. * TiKV currently does not exclude encryption keys and user data from core dumps. It is advised to disable core dumps for the TiKV process when using encryption at rest. This is not currently handled by TiKV itself. * TiKV tracks encrypted data files using the absolute path of the files. As a result, once encryption is turned on for a TiKV node, the user should not change data file paths configuration such as `storage.data-dir`, `raftstore.raftdb-path`, `rocksdb.wal-dir` and `raftdb.wal-dir`. -* TiKV info log contains user data for debugging purposes. The info log and this data in it are not encrypted. +* TiKV, TiDB, and PD info logs might contain user data for debugging purposes. The info log and this data in it are not encrypted. It is recommended to enable [log redaction](/log-redaction.md). ## TiKV encryption at rest diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 60b75c3a358d2..c9d2539fb8a58 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -6,64 +6,65 @@ aliases: ['/docs/dev/functions-and-operators/string-functions/','/docs/dev/refer # String Functions -TiDB supports most of the [string functions](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html) available in MySQL 5.7. +TiDB supports most of the [string functions](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html) available in MySQL 5.7 and some of the [functions](https://docs.oracle.com/en/database/oracle/oracle-database/21/sqlqr/SQL-Functions.html#GUID-93EC62F8-415D-4A7E-B050-5D5B2C127009) available in Oracle 21. ## Supported functions -| Name | Description | -|:------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------| -| [`ASCII()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_ascii) | Return numeric value of left-most character | -| [`BIN()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_bin) | Return a string containing binary representation of a number | -| [`BIT_LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_bit-length) | Return length of argument in bits | -| [`CHAR()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_char) | Return the character for each integer passed | -| [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_char-length) | Return number of characters in argument | -| [`CHARACTER_LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_character-length) | Synonym for `CHAR_LENGTH()` | -| [`CONCAT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat) | Return concatenated string | -| [`CONCAT_WS()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat-ws) | Return concatenate with separator | -| [`ELT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_elt) | Return string at index number | -| [`EXPORT_SET()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_export-set) | Return a string such that for every bit set in the value bits, you get an on string and for every unset bit, you get an off string | -| [`FIELD()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_field) | Return the index (position) of the first argument in the subsequent arguments | -| [`FIND_IN_SET()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_find-in-set) | Return the index position of the first argument within the second argument | -| [`FORMAT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_format) | Return a number formatted to specified number of decimal places | -| [`FROM_BASE64()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_from-base64) | Decode to a base-64 string and return result | -| [`HEX()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_hex) | Return a hexadecimal representation of a decimal or string value | -| [`INSERT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_insert) | Insert a substring at the specified position up to the specified number of characters | -| [`INSTR()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_instr) | Return the index of the first occurrence of substring | -| [`LCASE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_lcase) | Synonym for `LOWER()` | -| [`LEFT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_left) | Return the leftmost number of characters as specified | -| [`LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_length) | Return the length of a string in bytes | -| [`LIKE`](https://dev.mysql.com/doc/refman/5.7/en/string-comparison-functions.html#operator_like) | Simple pattern matching | -| [`LOCATE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_locate) | Return the position of the first occurrence of substring | -| [`LOWER()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_lower) | Return the argument in lowercase | -| [`LPAD()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_lpad) | Return the string argument, left-padded with the specified string | -| [`LTRIM()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_ltrim) | Remove leading spaces | -| [`MAKE_SET()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_make-set) | Return a set of comma-separated strings that have the corresponding bit in bits set | -| [`MID()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_mid) | Return a substring starting from the specified position | -| [`NOT LIKE`](https://dev.mysql.com/doc/refman/5.7/en/string-comparison-functions.html#operator_not-like) | Negation of simple pattern matching | -| [`NOT REGEXP`](https://dev.mysql.com/doc/refman/5.7/en/regexp.html#operator_not-regexp) | Negation of `REGEXP` | -| [`OCT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_oct) | Return a string containing octal representation of a number | -| [`OCTET_LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_octet-length) | Synonym for `LENGTH()` | -| [`ORD()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_ord) | Return character code for leftmost character of the argument | -| [`POSITION()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_position) | Synonym for `LOCATE()` | -| [`QUOTE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_quote) | Escape the argument for use in an SQL statement | -| [`REGEXP`](https://dev.mysql.com/doc/refman/5.7/en/regexp.html#operator_regexp) | Pattern matching using regular expressions | -| [`REPEAT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_repeat) | Repeat a string the specified number of times | -| [`REPLACE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_replace) | Replace occurrences of a specified string | -| [`REVERSE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_reverse) | Reverse the characters in a string | -| [`RIGHT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_right) | Return the specified rightmost number of characters | -| [`RLIKE`](https://dev.mysql.com/doc/refman/5.7/en/regexp.html#operator_regexp) | Synonym for `REGEXP` | -| [`RPAD()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_rpad) | Append string the specified number of times | -| [`RTRIM()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_rtrim) | Remove trailing spaces | -| [`SPACE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_space) | Return a string of the specified number of spaces | -| [`STRCMP()`](https://dev.mysql.com/doc/refman/5.7/en/string-comparison-functions.html#function_strcmp) | Compare two strings | -| [`SUBSTR()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_substr) | Return the substring as specified | -| [`SUBSTRING()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_substring) | Return the substring as specified | -| [`SUBSTRING_INDEX()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_substring-index) | Return a substring from a string before the specified number of occurrences of the delimiter | -| [`TO_BASE64()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_to-base64) | Return the argument converted to a base-64 string | -| [`TRIM()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_trim) | Remove leading and trailing spaces | -| [`UCASE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_ucase) | Synonym for `UPPER()` | -| [`UNHEX()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_unhex) | Return a string containing hex representation of a number | -| [`UPPER()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_upper) | Convert to uppercase | +| Name | Description | +|:----------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------| +| [`ASCII()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_ascii) | Return numeric value of left-most character | +| [`BIN()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_bin) | Return a string containing binary representation of a number | +| [`BIT_LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_bit-length) | Return length of argument in bits | +| [`CHAR()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_char) | Return the character for each integer passed | +| [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_char-length) | Return number of characters in argument | +| [`CHARACTER_LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_character-length) | Synonym for `CHAR_LENGTH()` | +| [`CONCAT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat) | Return concatenated string | +| [`CONCAT_WS()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat-ws) | Return concatenate with separator | +| [`ELT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_elt) | Return string at index number | +| [`EXPORT_SET()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_export-set) | Return a string such that for every bit set in the value bits, you get an on string and for every unset bit, you get an off string | +| [`FIELD()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_field) | Return the index (position) of the first argument in the subsequent arguments | +| [`FIND_IN_SET()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_find-in-set) | Return the index position of the first argument within the second argument | +| [`FORMAT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_format) | Return a number formatted to specified number of decimal places | +| [`FROM_BASE64()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_from-base64) | Decode to a base-64 string and return result | +| [`HEX()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_hex) | Return a hexadecimal representation of a decimal or string value | +| [`INSERT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_insert) | Insert a substring at the specified position up to the specified number of characters | +| [`INSTR()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_instr) | Return the index of the first occurrence of substring | +| [`LCASE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_lcase) | Synonym for `LOWER()` | +| [`LEFT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_left) | Return the leftmost number of characters as specified | +| [`LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_length) | Return the length of a string in bytes | +| [`LIKE`](https://dev.mysql.com/doc/refman/5.7/en/string-comparison-functions.html#operator_like) | Simple pattern matching | +| [`LOCATE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_locate) | Return the position of the first occurrence of substring | +| [`LOWER()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_lower) | Return the argument in lowercase | +| [`LPAD()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_lpad) | Return the string argument, left-padded with the specified string | +| [`LTRIM()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_ltrim) | Remove leading spaces | +| [`MAKE_SET()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_make-set) | Return a set of comma-separated strings that have the corresponding bit in bits set | +| [`MID()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_mid) | Return a substring starting from the specified position | +| [`NOT LIKE`](https://dev.mysql.com/doc/refman/5.7/en/string-comparison-functions.html#operator_not-like) | Negation of simple pattern matching | +| [`NOT REGEXP`](https://dev.mysql.com/doc/refman/5.7/en/regexp.html#operator_not-regexp) | Negation of `REGEXP` | +| [`OCT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_oct) | Return a string containing octal representation of a number | +| [`OCTET_LENGTH()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_octet-length) | Synonym for `LENGTH()` | +| [`ORD()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_ord) | Return character code for leftmost character of the argument | +| [`POSITION()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_position) | Synonym for `LOCATE()` | +| [`QUOTE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_quote) | Escape the argument for use in an SQL statement | +| [`REGEXP`](https://dev.mysql.com/doc/refman/5.7/en/regexp.html#operator_regexp) | Pattern matching using regular expressions | +| [`REPEAT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_repeat) | Repeat a string the specified number of times | +| [`REPLACE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_replace) | Replace occurrences of a specified string | +| [`REVERSE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_reverse) | Reverse the characters in a string | +| [`RIGHT()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_right) | Return the specified rightmost number of characters | +| [`RLIKE`](https://dev.mysql.com/doc/refman/5.7/en/regexp.html#operator_regexp) | Synonym for `REGEXP` | +| [`RPAD()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_rpad) | Append string the specified number of times | +| [`RTRIM()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_rtrim) | Remove trailing spaces | +| [`SPACE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_space) | Return a string of the specified number of spaces | +| [`STRCMP()`](https://dev.mysql.com/doc/refman/5.7/en/string-comparison-functions.html#function_strcmp) | Compare two strings | +| [`SUBSTR()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_substr) | Return the substring as specified | +| [`SUBSTRING()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_substring) | Return the substring as specified | +| [`SUBSTRING_INDEX()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_substring-index) | Return a substring from a string before the specified number of occurrences of the delimiter | +| [`TO_BASE64()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_to-base64) | Return the argument converted to a base-64 string | +| [`TRANSLATE()`](https://docs.oracle.com/en/database/oracle/oracle-database/21/sqlrf/TRANSLATE.html#GUID-80F85ACB-092C-4CC7-91F6-B3A585E3A690) | Replace all occurrences of characters by other characters in a string. It does not treat empty strings as `NULL` as Oracle does. | +| [`TRIM()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_trim) | Remove leading and trailing spaces | +| [`UCASE()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_ucase) | Synonym for `UPPER()` | +| [`UNHEX()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_unhex) | Return a string containing hex representation of a number | +| [`UPPER()`](https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_upper) | Convert to uppercase | ## Unsupported functions diff --git a/grafana-tidb-dashboard.md b/grafana-tidb-dashboard.md index 52e04b900ce64..e7b1e086514c5 100644 --- a/grafana-tidb-dashboard.md +++ b/grafana-tidb-dashboard.md @@ -165,6 +165,5 @@ To understand the key metrics displayed on the TiDB dashboard, check the followi - Batch Client - Pending Request Count by TiKV: the number of Batch messages that are pending processing - - Wait Duration 95: the waiting time of Batch messages that are pending processing - Batch Client Unavailable Duration 95: the unavailable time of the Batch client - No Available Connection Counter: the number of times the Batch client cannot find an available link diff --git a/information-schema/information-schema-referential-constraints.md b/information-schema/information-schema-referential-constraints.md new file mode 100644 index 0000000000000..18bb3b9d3d4cd --- /dev/null +++ b/information-schema/information-schema-referential-constraints.md @@ -0,0 +1,69 @@ +--- +title: REFERENTIAL_CONSTRAINTS +summary: Learn the `REFERENTIAL_CONSTRAINTS` information_schema table. +--- + +# REFERENTIAL_CONSTRAINTS + +The `REFERENTIAL_CONSTRAINTS` table provides information about `FOREIGN KEY` relationships between tables. Note that TiDB currently does not enforce `FOREIGN KEY` constraints, or perform actions such as `ON DELETE CASCADE`. + +{{< copyable "sql" >}} + +```sql +USE information_schema; +DESC referential_constraints; +``` + +```sql ++---------------------------+--------------+------+------+---------+-------+ +| Field | Type | Null | Key | Default | Extra | ++---------------------------+--------------+------+------+---------+-------+ +| CONSTRAINT_CATALOG | varchar(512) | NO | | NULL | | +| CONSTRAINT_SCHEMA | varchar(64) | NO | | NULL | | +| CONSTRAINT_NAME | varchar(64) | NO | | NULL | | +| UNIQUE_CONSTRAINT_CATALOG | varchar(512) | NO | | NULL | | +| UNIQUE_CONSTRAINT_SCHEMA | varchar(64) | NO | | NULL | | +| UNIQUE_CONSTRAINT_NAME | varchar(64) | YES | | NULL | | +| MATCH_OPTION | varchar(64) | NO | | NULL | | +| UPDATE_RULE | varchar(64) | NO | | NULL | | +| DELETE_RULE | varchar(64) | NO | | NULL | | +| TABLE_NAME | varchar(64) | NO | | NULL | | +| REFERENCED_TABLE_NAME | varchar(64) | NO | | NULL | | ++---------------------------+--------------+------+------+---------+-------+ +11 rows in set (0.00 sec) +``` + +{{< copyable "sql" >}} + +```sql +CREATE TABLE test.parent ( + id INT NOT NULL AUTO_INCREMENT, + PRIMARY KEY (id) +); + +CREATE TABLE test.child ( + id INT NOT NULL AUTO_INCREMENT, + name varchar(255) NOT NULL, + parent_id INT DEFAULT NULL, + PRIMARY KEY (id), + CONSTRAINT fk_parent FOREIGN KEY (parent_id) REFERENCES parent (id) ON UPDATE CASCADE ON DELETE RESTRICT +); + +SELECT * FROM referential_constraints\G +``` + +``` +*************************** 1. row *************************** + CONSTRAINT_CATALOG: def + CONSTRAINT_SCHEMA: test + CONSTRAINT_NAME: fk_parent +UNIQUE_CONSTRAINT_CATALOG: def + UNIQUE_CONSTRAINT_SCHEMA: test + UNIQUE_CONSTRAINT_NAME: PRIMARY + MATCH_OPTION: NONE + UPDATE_RULE: CASCADE + DELETE_RULE: RESTRICT + TABLE_NAME: child + REFERENCED_TABLE_NAME: parent +1 row in set (0.00 sec) +``` \ No newline at end of file diff --git a/information-schema/information-schema.md b/information-schema/information-schema.md index d2d763dffe501..c06b656376e69 100644 --- a/information-schema/information-schema.md +++ b/information-schema/information-schema.md @@ -32,7 +32,7 @@ Many `INFORMATION_SCHEMA` tables have a corresponding `SHOW` command. The benefi | `PLUGINS` | Not implemented by TiDB. Returns zero rows. | | [`PROCESSLIST`](/information-schema/information-schema-processlist.md) | Provides similar information to the command `SHOW PROCESSLIST`. | | `PROFILING` | Not implemented by TiDB. Returns zero rows. | -| `REFERENTIAL_CONSTRAINTS` | Not implemented by TiDB. Returns zero rows. | +| `REFERENTIAL_CONSTRAINTS` | Provides information on `FOREIGN KEY` constraints. | | `ROUTINES` | Not implemented by TiDB. Returns zero rows. | | [`SCHEMATA`](/information-schema/information-schema-schemata.md) | Provides similar information to `SHOW DATABASES`. | | `SCHEMA_PRIVILEGES` | Not implemented by TiDB. Returns zero rows. | diff --git a/maintain-tidb-using-tiup.md b/maintain-tidb-using-tiup.md index 03f50ee1662da..874a9a202cdd7 100644 --- a/maintain-tidb-using-tiup.md +++ b/maintain-tidb-using-tiup.md @@ -109,7 +109,7 @@ When the cluster is in operation, if you need to modify the parameters of a comp log.slow-threshold: 300 ``` - For the parameter format, see the [TiUP parameter template](https://github.com/pingcap/tiup/blob/master/embed/templates/examples/topology.example.yaml). + For the parameter format, see the [TiUP parameter template](https://github.com/pingcap/tiup/blob/master/embed/examples/cluster/topology.example.yaml). **Use `.` to represent the hierarchy of the configuration items**. diff --git a/production-deployment-using-tiup.md b/production-deployment-using-tiup.md index 73bc55696759f..9657f5809cad6 100644 --- a/production-deployment-using-tiup.md +++ b/production-deployment-using-tiup.md @@ -228,7 +228,7 @@ The following examples cover six common scenarios. You need to modify the config > > - For parameters that should be globally effective, configure these parameters of corresponding components in the `server_configs` section of the configuration file. > - For parameters that should be effective on a specific node, configure these parameters in the `config` of this node. -> - Use `.` to indicate the subcategory of the configuration, such as `log.slow-threshold`. For more formats, see [TiUP configuration template](https://github.com/pingcap/tiup/blob/master/embed/templates/examples/topology.example.yaml). +> - Use `.` to indicate the subcategory of the configuration, such as `log.slow-threshold`. For more formats, see [TiUP configuration template](https://github.com/pingcap/tiup/blob/master/embed/examples/cluster/topology.example.yaml). > - For more parameter description, see [TiDB `config.toml.example`](https://github.com/pingcap/tidb/blob/master/config/config.toml.example), [TiKV `config.toml.example`](https://github.com/tikv/tikv/blob/master/etc/config-template.toml), [PD `config.toml.example`](https://github.com/pingcap/pd/blob/master/conf/config.toml), and [TiFlash configuration](/tiflash/tiflash-configuration.md). ## Step 4: Execute the deployment command @@ -270,7 +270,7 @@ In the above command: - The initialization configuration file is `topology.yaml`. - `--user root`: Log in to the target machine through the `root` key to complete the cluster deployment, or you can use other users with `ssh` and `sudo` privileges to complete the deployment. - `[-i]` and `[-p]`: optional. If you have configured login to the target machine without password, these parameters are not required. If not, choose one of the two parameters. `[-i]` is the private key of the `root` user (or other users specified by `--user`) that has access to the target machine. `[-p]` is used to input the user password interactively. -- If you need to specify the user group name to be created on the target machine, see [this example](https://github.com/pingcap/tiup/blob/master/embed/templates/examples/topology.example.yaml#L7). +- If you need to specify the user group name to be created on the target machine, see [this example](https://github.com/pingcap/tiup/blob/master/embed/examples/cluster/topology.example.yaml#L7). At the end of the output log, you will see ```Deployed cluster `tidb-test` successfully```. This indicates that the deployment is successful. diff --git a/releases/release-5.1.0.md b/releases/release-5.1.0.md index fd50b983d733c..045a10b9451f3 100644 --- a/releases/release-5.1.0.md +++ b/releases/release-5.1.0.md @@ -174,7 +174,7 @@ In v5.1, the key new features or improvements are as follows: TiDB adds the running status of TiDB cluster requests in telemetry, including execution status, failure status, etc. -To learn more about the information and how to disable this behavior, refer to [Telemetry](https://docs.pingcap.com/zh/tidb/stable/telemetry). +To learn more about the information and how to disable this behavior, refer to [Telemetry](/telemetry.md). ## Improvements diff --git a/scripts/verify-link-anchors.sh b/scripts/verify-link-anchors.sh index 445159d537f15..77faf1fdcd30c 100755 --- a/scripts/verify-link-anchors.sh +++ b/scripts/verify-link-anchors.sh @@ -7,7 +7,7 @@ ROOT=$(unset CDPATH && cd $(dirname "${BASH_SOURCE[0]}")/.. && pwd) cd $ROOT -npm install -g remark-cli remark-lint breeswish/remark-lint-pingcap-docs-anchor +npm install -g remark-cli@9.0.0 remark-lint@8.0.0 breeswish/remark-lint-pingcap-docs-anchor echo "info: checking links anchors under $ROOT directory..." diff --git a/sql-statements/sql-statement-select.md b/sql-statements/sql-statement-select.md index fc2811a18b604..64afe8dbdca1b 100644 --- a/sql-statements/sql-statement-select.md +++ b/sql-statements/sql-statement-select.md @@ -98,7 +98,7 @@ AsOfClause ::= |`ORDER BY` | The `ORDER BY` clause is used to sort the data in ascending or descending order, based on columns, expressions or items in the `select_expr` list.| |`LIMIT` | The `LIMIT` clause can be used to constrain the number of rows. `LIMIT` takes one or two numeric arguments. With one argument, the argument specifies the maximum number of rows to return, the first row to return is the first row of the table by default; with two arguments, the first argument specifies the offset of the first row to return, and the second specifies the maximum number of rows to return. TiDB also supports the `FETCH FIRST/NEXT n ROW/ROWS ONLY` syntax, which has the same effect as `LIMIT n`. You can omit `n` in this syntax and its effect is the same as `LIMIT 1`. | |`Window window_definition`| This is the syntax for window function, which is usually used to do some analytical computation. For more information, refer to [Window Function](/functions-and-operators/window-functions.md). | -| `FOR UPDATE` | The `SELECT FOR UPDATE` clause locks all the data in the result sets to detect concurrent updates from other transactions. Data that match the query conditions but do not exist in the result sets are not read-locked, such as the row data written by other transactions after the current transaction is started. TiDB uses the [Optimistic Transaction Model](/optimistic-transaction.md). The transaction conflicts are not detected in the statement execution phase. Therefore, the current transaction does not block other transactions from executing `UPDATE`, `DELETE` or `SELECT FOR UPDATE` like other databases such as PostgreSQL. In the committing phase, the rows read by `SELECT FOR UPDATE` are committed in two phases, which means they can also join the conflict detection. If write conflicts occur, the commit fails for all transactions that include the `SELECT FOR UPDATE` clause. If no conflict is detected, the commit succeeds. And a new version is generated for the locked rows, so that write conflicts can be detected when other uncommitted transactions are being committed later. When using pessimistic transaction model, the behavior is basically the same as other databases. Refer to [Difference with MySQL InnoDB](/pessimistic-transaction.md#difference-with-mysql-innodb) to see the details. | +| `FOR UPDATE` | The `SELECT FOR UPDATE` clause locks all the data in the result sets to detect concurrent updates from other transactions. Data that match the query conditions but do not exist in the result sets are not read-locked, such as the row data written by other transactions after the current transaction is started. TiDB uses the [Optimistic Transaction Model](/optimistic-transaction.md). The transaction conflicts are not detected in the statement execution phase. Therefore, the current transaction does not block other transactions from executing `UPDATE`, `DELETE` or `SELECT FOR UPDATE` like other databases such as PostgreSQL. In the committing phase, the rows read by `SELECT FOR UPDATE` are committed in two phases, which means they can also join the conflict detection. If write conflicts occur, the commit fails for all transactions that include the `SELECT FOR UPDATE` clause. If no conflict is detected, the commit succeeds. And a new version is generated for the locked rows, so that write conflicts can be detected when other uncommitted transactions are being committed later. When using pessimistic transaction model, the behavior is basically the same as other databases. Refer to [Difference with MySQL InnoDB](/pessimistic-transaction.md#difference-with-mysql-innodb) to see the details. TiDB supports the `NOWAIT` modifier for `FOR UPDATE`. See [TiDB Pessimistic Transaction Model](/pessimistic-transaction.md) for details. | |`LOCK IN SHARE MODE` | To guarantee compatibility, TiDB parses these three modifiers, but will ignore them. | ## Examples diff --git a/ticdc/manage-ticdc.md b/ticdc/manage-ticdc.md index b529daa446f6a..ec6b32168f4c9 100644 --- a/ticdc/manage-ticdc.md +++ b/ticdc/manage-ticdc.md @@ -873,6 +873,6 @@ In the output of the above command, if the value of `sort-engine` is "unified", > **Note:** > > + If your servers use mechanical hard drives or other storage devices that have high latency or limited bandwidth, use the unified sorter with caution. -> + The total free capacity of hard drives must be greater than or equal to 128G. If you need to replicate a large amount of historical data, make sure that the free capacity on each node is greater than or equal to the size of the incremental data that needs to be replicated. +> + The total free capacity of hard drives must be greater than or equal to 500G. If you need to replicate a large amount of historical data, make sure that the free capacity on each node is greater than or equal to the size of the incremental data that needs to be replicated. > + Unified sorter is enabled by default. If your servers do not match the above requirements and you want to disable the unified sorter, you need to manually set `sort-engine` to `memory` for the changefeed. > + To enable Unified Sorter on an existing changefeed, see the methods provided in [How do I handle the OOM that occurs after TiCDC is restarted after a task interruption?](/ticdc/troubleshoot-ticdc.md#how-do-i-handle-the-oom-that-occurs-after-ticdc-is-restarted-after-a-task-interruption). diff --git a/ticdc/ticdc-open-protocol.md b/ticdc/ticdc-open-protocol.md index 22c5d9e6aa530..3e514ce565db1 100644 --- a/ticdc/ticdc-open-protocol.md +++ b/ticdc/ticdc-open-protocol.md @@ -274,7 +274,7 @@ COMMIT; Currently, TiCDC does not provide the standard parsing library for TiCDC Open Protocol, but the Golang version and Java version of parsing demonstrations are provided. You can refer to the data format provided in this document and the following demonstrations to implement the protocol parsing for consumers. -- [Golang demo](https://github.com/pingcap/ticdc/tree/master/kafka_consumer) +- [Golang demo](https://github.com/pingcap/ticdc/tree/master/cmd/kafka-consumer) - [Java demo](https://github.com/pingcap/ticdc/tree/master/demo/java) ## Column type code diff --git a/ticdc/ticdc-overview.md b/ticdc/ticdc-overview.md index da926141a9341..2ead1b20abdea 100644 --- a/ticdc/ticdc-overview.md +++ b/ticdc/ticdc-overview.md @@ -129,3 +129,13 @@ For details, refer to [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md). ## TiCDC Open Protocol TiCDC Open Protocol is a row-level data change notification protocol that provides data sources for monitoring, caching, full-text indexing, analysis engines, and primary-secondary replication between different databases. TiCDC complies with TiCDC Open Protocol and replicates data changes of TiDB to third-party data medium such as MQ (Message Queue). For more information, see [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md). + +## Compatibility notes for `sort-dir` and `data-dir` + +The `sort-dir` configuration is used to specify the temporary file directory for the TiCDC sorter. Its functionalities might vary in different versions. The following table lists `sort-dir`'s compatibility changes across versions. + +| Version | `sort-engine` functionality | Note | Recommendation | +| :--- | :--- | :-- | :-- | +| v4.0.11 or an earlier v4.0 version, v5.0.0-rc | It is a changefeed configuration item and specifies temporary file directory for the `file` sorter and `unified` sorter. | In these versions, `file` sorter and `unified` sorter are **experimental features** and **NOT** recommended for the production environment.

If multiple changefeeds use the `unified` sorter as its `sort-engine`, the actual temporary file directory might be the `sort-dir` configuration of any changefeed, and the directory used for each TiCDC node might be different. | It is not recommended to use `unified` sorter in the production environment. | +| v4.0.12, v4.0.13, v5.0.0, and v5.0.1 | It is a configuration item of changefeed or of `cdc server`. | By default, the `sort-dir` configuration of a changefeed does not take effect, and the `sort-dir` configuration of `cdc server` defaults to `/tmp/cdc_sort`. It is recommended to only configure `cdc server` in the production environment.

If you use TiUP to deploy TiCDC, it is recommended to use the latest TiUP version and set `sorter.sort-dir` in the TiCDC server configuration.

The `unified` sorter is enabled by default in v4.0.13, v5.0.0, and v5.0.1. If you want to upgrade your cluster to these versions, make sure that you have correctly configured `sorter.sort-dir` in the TiCDC server configuration. | You need to configure `sort-dir` using the `cdc server` command-line parameter (or TiUP). | +| v4.0.14 and later v4.0 versions, v5.0.2 and later v5.0 versions, later TiDB versions | `sort-dir` is deprecated. It is recommended to configure `data-dir`. | You can configure `data-dir` using the latest version of TiUP. In these TiDB versions, `unified` sorter is enabled by default. Make sure that `data-dir` has been configured correctly when you upgrade your cluster. Otherwise, `/tmp/cdc_data` will be used by default as the temporary file directory.

If the storage capacity of the device where the directory is located is insufficient, the problem of insufficient hard disk space might occur. In this situation, the previous `sort-dir` configuration of changefeed will become invalid.| You need to configure `data-dir` using the `cdc server` command-line parameter (or TiUP). | diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index 40921ce072a4e..be47a53d9caf9 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -399,6 +399,30 @@ cdc cli changefeed update -c test-cf --pd=http://10.0.10.25:2379 --start-ts 4152 cdc cli changefeed resume -c test-cf --pd=http://10.0.10.25:2379 ``` +## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? + +Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: + +{{< copyable "sql" >}} + +```sql +mysql root@127.0.0.1:test> show create table test; ++-------+----------------------------------------------------------------------------------+ +| Table | Create Table | ++-------+----------------------------------------------------------------------------------+ +| test | CREATE TABLE `test` ( | +| | `id` int(11) NOT NULL, | +| | `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, | +| | PRIMARY KEY (`id`) | +| | ) ENGINE=InnoDB DEFAULT CHARSET=latin1 | ++-------+----------------------------------------------------------------------------------+ +1 row in set +``` + +From the result, you can see that the table schema before and after the replication is inconsistent. This is because the default value of `explicit_defaults_for_timestamp` in TiDB is different from that in MySQL. See [MySQL Compatibility](/mysql-compatibility.md#default-differences) for details. + +Since v5.0.1 or v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1 or v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. + ## When the sink of the replication downstream is TiDB or MySQL, what permissions do users of the downstream database need? When the sink is TiDB or MySQL, the users of the downstream database need the following permissions: diff --git a/tidb-configuration-file.md b/tidb-configuration-file.md index 49fb30aa87985..c856bb976f7b1 100644 --- a/tidb-configuration-file.md +++ b/tidb-configuration-file.md @@ -274,6 +274,11 @@ Configuration items related to log files. Configuration items related to security. +### `require-secure-transport` + +- Determines whether to require the client to use the secure mode for data transport. +- Default value: `false` + ### `enable-sem` - Enables the Security Enhanced Mode (SEM). @@ -321,6 +326,11 @@ Configuration items related to security. + Default value: `"plaintext"`, which disables encryption. + Optional values: `"plaintext"` and `"aes128-ctr"` +### `auto-tls` + +- Determines whether to automatically generate the TLS certificates on startup. +- Default value: `true` + ## Performance Configuration items related to performance. diff --git a/tidb-lightning/tidb-lightning-backends.md b/tidb-lightning/tidb-lightning-backends.md index 64991a0c6773f..eb0aab23a2f5d 100644 --- a/tidb-lightning/tidb-lightning-backends.md +++ b/tidb-lightning/tidb-lightning-backends.md @@ -10,14 +10,14 @@ The backend determines how TiDB Lightning imports data into the target cluster. TiDB Lightning supports the following [backends](/tidb-lightning/tidb-lightning-glossary.md#back-end): -+ [Importer-backend](#tidb-lightning-importer-backend) (default) + [Local-backend](#tidb-lightning-local-backend) ++ [Importer-backend](#tidb-lightning-importer-backend) + [TiDB-backend](#tidb-lightning-tidb-backend) -The **Importer-backend** (default): `tidb-lightning` first encodes the SQL or CSV data into KV pairs, and relies on the external `tikv-importer` program to sort these KV pairs and ingest directly into the TiKV nodes. - The **Local-backend**: `tidb-lightning` first encodes data into key-value pairs, sorts and stores them in a local temporary directory, and *upload* these key-value pairs to each TiKV node *as SST files*. Then, TiKV ingests these *SST files* into the cluster. The implementation of Local-backend is the same with that of Importer-backend but does not rely on the external `tikv-importer` component. +The **Importer-backend**: `tidb-lightning` first encodes the SQL or CSV data into KV pairs, and relies on the external `tikv-importer` program to sort these KV pairs and ingest directly into the TiKV nodes. + The **TiDB-backend**: `tidb-lightning` first encodes these data into SQL `INSERT` statements, and has these statements executed directly on the TiDB node. | Backend | Local-backend | Importer-backend | TiDB-backend | diff --git a/tidb-lightning/tidb-lightning-configuration.md b/tidb-lightning/tidb-lightning-configuration.md index 47975da4d7a1f..d2432d520ae2f 100644 --- a/tidb-lightning/tidb-lightning-configuration.md +++ b/tidb-lightning/tidb-lightning-configuration.md @@ -98,8 +98,8 @@ driver = "file" #keep-after-success = false [tikv-importer] -# Delivery backend, can be "importer", "local", or "tidb". -# backend = "importer" +# Delivery backend, can be "local", "importer" or "tidb". +# backend = "local" # The listening address of tikv-importer when backend is "importer". Change it to the actual address. addr = "172.16.31.10:8287" # Action to do when trying to insert a duplicated entry in the "tidb" backend. @@ -362,7 +362,7 @@ min-available-ratio = 0.05 | -d *directory* | Directory or [external storage URL](/br/backup-and-restore-storages.md) of the data dump to read from | `mydumper.data-source-dir` | | -L *level* | Log level: debug, info, warn, error, fatal (default = info) | `lightning.log-level` | | -f *rule* | [Table filter rules](/table-filter.md) (can be specified multiple times) | `mydumper.filter` | -| --backend *backend* | [Delivery backend](/tidb-lightning/tidb-lightning-backends.md) (`importer`, `local`, or `tidb`) | `tikv-importer.backend` | +| --backend *backend* | [Delivery backend](/tidb-lightning/tidb-lightning-backends.md) (`local`, `importer`, or `tidb`) | `tikv-importer.backend` | | --log-file *file* | Log file path (default = a temporary file in `/tmp`) | `lightning.log-file` | | --status-addr *ip:port* | Listening address of the TiDB Lightning server | `lightning.status-port` | | --importer *host:port* | Address of TiKV Importer | `tikv-importer.addr` | diff --git a/tidb-lightning/tidb-lightning-faq.md b/tidb-lightning/tidb-lightning-faq.md index 1033900cf1276..d24584f24ae0b 100644 --- a/tidb-lightning/tidb-lightning-faq.md +++ b/tidb-lightning/tidb-lightning-faq.md @@ -348,3 +348,19 @@ This error occurs usually because the CSV data file does not contain a header (t [mydumper.csv] header = false ``` + +## How to get the runtime goroutine information of TiDB Lightning + +1. If [`status-port`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-configuration) has been specified in the configuration file of TiDB Lightning, skip this step. Otherwise, you need to send the USR1 signal to TiDB Lightning to enable `status-port`. + + Get the process ID (PID) of TiDB Lightning using commands like `ps`, and then run the following command: + + {{< copyable "shell-regular" >}} + + ```sh + kill -USR1 + ``` + + Check the log of TiDB Lightning. The log of `starting HTTP server` / `start HTTP server` / `started HTTP server` shows the newly enabled `status-port`. + +2. Access `http://:/debug/pprof/goroutine?debug=2` to get the goroutine information. diff --git a/tidb-monitoring-api.md b/tidb-monitoring-api.md index f5e6313b60b29..13dc89b2f47d7 100644 --- a/tidb-monitoring-api.md +++ b/tidb-monitoring-api.md @@ -8,7 +8,8 @@ aliases: ['/docs/dev/tidb-monitoring-api/'] You can use the following two types of interfaces to monitor the TiDB cluster state: -- [The state interface](#use-the-state-interface): this interface uses the HTTP interface to get the component information. +- [The state interface](#running-status): this interface uses the HTTP interface to get the component information. +- [Storage information](#storage-information): this interface uses the HTTP interface to get the storage information of data tables. - [The metrics interface](#use-the-metrics-interface): this interface uses Prometheus to record the detailed information of the various operations in components and views these metrics using Grafana. ## Use the state interface @@ -20,7 +21,9 @@ The state interface monitors the basic information of a specific component in th - TiDB API address: `http://${host}:${port}` - Default port: `10080` -The following example uses `http://${host}:${port}/status` to get the current state of the TiDB server and to determine whether the server is alive. The result is returned in JSON format. +### Running status + +The following example uses `http://${host}:${port}/status` to get the current state of the TiDB server and to determine whether the server is alive. The result is returned in **JSON** format. ```bash curl http://127.0.0.1:10080/status @@ -31,6 +34,48 @@ curl http://127.0.0.1:10080/status } ``` +#### Storage information + +The following example uses `http://${host}:${port}/schema_storage/${db}/${table}` to get the storage information of the specific data table. The result is returned in **JSON** format. + +{{< copyable "shell-regular" >}} + +```bash +curl http://127.0.0.1:10080/schema_storage/mysql/stats_histograms +``` + +``` +{ + "table_schema": "mysql", + "table_name": "stats_histograms", + "table_rows": 0, + "avg_row_length": 0, + "data_length": 0, + "max_data_length": 0, + "index_length": 0, + "data_free": 0 +} +``` + +```bash +curl http://127.0.0.1:10080/schema_storage/test +``` + +``` +[ + { + "table_schema": "test", + "table_name": "test", + "table_rows": 0, + "avg_row_length": 0, + "data_length": 0, + "max_data_length": 0, + "index_length": 0, + "data_free": 0 + } +] +``` + ### PD server - PD API address: `http://${host}:${port}/pd/api/v1/${api_name}` diff --git a/tidb-scheduling.md b/tidb-scheduling.md index a011bdbb30306..f8820ed0275ec 100644 --- a/tidb-scheduling.md +++ b/tidb-scheduling.md @@ -45,7 +45,7 @@ The above situations can be classified into two types: * Storage capacity of all TiKV peers are balanced; * Hot spots are balanced; * Speed of load balancing for the Regions needs to be limited to ensure that online services are stable; - * Maintainers are able to to take peers online/offline manually. + * Maintainers are able to take peers online/offline manually. After the first type of requirements is satisfied, the system will be failure tolerable. After the second type of requirements is satisfied, resources will be utilized more efficiently and the system will have better scalability. diff --git a/tiflash/tune-tiflash-performance.md b/tiflash/tune-tiflash-performance.md index 9bbe13d0dcc56..be517d1805d09 100644 --- a/tiflash/tune-tiflash-performance.md +++ b/tiflash/tune-tiflash-performance.md @@ -52,14 +52,4 @@ If you want to save machine resources and have no requirement on isolation, you ```sql set @@tidb_opt_distinct_agg_push_down = 1; - ``` - -5. If the `JOIN` operator does not choose the MPP mode, you can modify the value of `tidb_opt_network_factor` to make the`JOIN` operator choose the MPP mode: - - The variable `tidb_opt_network_factor` is used to set the ratio of network overhead that the optimizer takes into account when calculating the cost. The smaller the variable value is, the smaller the estimated cost for a large amount of network transmissions is, and the more TiDB inclined to choose the MPP operator. - - {{< copyable "sql" >}} - - ```sql - set @@tidb_opt_network_factor = 0.001; - ``` + ``` \ No newline at end of file diff --git a/tikv-control.md b/tikv-control.md index fd70acaf1b163..d5cb2f52325b7 100644 --- a/tikv-control.md +++ b/tikv-control.md @@ -518,3 +518,40 @@ Type "I consent" to continue, anything else to exit: I consent > **Note** > > The command will expose data encryption keys as plaintext. In production, DO NOT redirect the output to a file. Even deleting the output file afterward may not cleanly wipe out the content from disk. + +### Print information related to damaged SST files + +Damaged SST files in TiKV might cause the TiKV process to panic. To clean up the damaged SST files, you will need the information of these files. To get the information, you can execute the `bad-ssts` command in TiKV Control. The needed information is shown in the output. The following is an example command and output. + +> **Note:** +> +> Before running this command, stop the running TiKV instance. + +```bash +$ tikv-ctl bad-ssts --db --pd +``` + +```bash +-------------------------------------------------------- +corruption info: +data/tikv-21107/db/000014.sst: Corruption: Bad table magic number: expected 9863518390377041911, found 759105309091689679 in data/tikv-21107/db/000014.sst + +sst meta: +14:552997[1 .. 5520]['0101' seq:1, type:1 .. '7A7480000000000000FF0F5F728000000000FF0002160000000000FAFA13AB33020BFFFA' seq:2032, type:1] at level 0 for Column family "default" (ID 0) +it isn't easy to handle local data, start key:0101 + +overlap region: +RegionInfo { region: id: 4 end_key: 7480000000000000FF0500000000000000F8 region_epoch { conf_ver: 1 version: 2 } peers { id: 5 store_id: 1 }, leader: Some(id: 5 store_id: 1) } + +suggested operations: +tikv-ctl ldb --db=data/tikv-21107/db unsafe_remove_sst_file "data/tikv-21107/db/000014.sst" +tikv-ctl --db=data/tikv-21107/db tombstone -r 4 --pd +-------------------------------------------------------- +corruption analysis has completed +``` + +From the output above, you can see that the information of the damaged SST file is printed first and then the meta-information is printed. + ++ In the `sst meta` part, `14` means the SST file number; `552997` means the file size, followed by the smallest and largest sequence numbers and other meta-information. ++ The `overlap region` part shows the information of the Region involved. This information is obtained through the PD server. ++ The `suggested operations` part provides you suggestion to clean up the damaged SST file. You can take the suggestion to clean up files and restart the TiKV instance. diff --git a/tiup/tiup-cluster-topology-reference.md b/tiup/tiup-cluster-topology-reference.md index 795dbe68d1e1d..bd620f2f4d2c6 100644 --- a/tiup/tiup-cluster-topology-reference.md +++ b/tiup/tiup-cluster-topology-reference.md @@ -4,7 +4,7 @@ title: Topology Configuration File for TiDB Deployment Using TiUP # Topology Configuration File for TiDB Deployment Using TiUP -To deploy or scale TiDB using TiUP, you need to provide a topology file ([sample](https://github.com/pingcap/tiup/blob/master/embed/templates/examples/topology.example.yaml)) to describe the cluster topology. +To deploy or scale TiDB using TiUP, you need to provide a topology file ([sample](https://github.com/pingcap/tiup/blob/master/embed/examples/cluster/topology.example.yaml)) to describe the cluster topology. Similarly, to modify the cluster topology, you need to modify the topology file. The difference is that, after the cluster is deployed, you can only modify a part of the fields in the topology file. This document introduces each section of the topology file and each field in each section. diff --git a/tiup/tiup-cluster.md b/tiup/tiup-cluster.md index e5c685d32d246..dd6c219f3ed4e 100644 --- a/tiup/tiup-cluster.md +++ b/tiup/tiup-cluster.md @@ -63,7 +63,7 @@ tiup cluster deploy [flags] This command requires you to provide the cluster name, the TiDB cluster version, and a topology file of the cluster. -To write a topology file, refer to [the example](https://github.com/pingcap/tiup/blob/master/embed/templates/examples/topology.example.yaml). The following file is an example of the simplest topology: +To write a topology file, refer to [the example](https://github.com/pingcap/tiup/blob/master/embed/examples/cluster/topology.example.yaml). The following file is an example of the simplest topology: > **Note:** > diff --git a/tiup/tiup-dm-topology-reference.md b/tiup/tiup-dm-topology-reference.md index a135e4583acd7..b87d23b04d24d 100644 --- a/tiup/tiup-dm-topology-reference.md +++ b/tiup/tiup-dm-topology-reference.md @@ -4,7 +4,7 @@ title: Topology Configuration File for DM Cluster Deployment Using TiUP # Topology Configuration File for DM Cluster Deployment Using TiUP -To deploy or scale a TiDB Data Migration (DM) cluster, you need to provide a topology file ([sample](https://github.com/pingcap/tiup/blob/master/embed/templates/examples/dm/topology.example.yaml)) to describe the cluster topology. +To deploy or scale a TiDB Data Migration (DM) cluster, you need to provide a topology file ([sample](https://github.com/pingcap/tiup/blob/master/embed/examples/dm/topology.example.yaml)) to describe the cluster topology. Similarly, to modify the cluster topology, you need to modify the topology file. The difference is that, after the cluster is deployed, you can only modify a part of the fields in the topology file. This document introduces each section of the topology file and each field in each section. diff --git a/upgrade-tidb-using-tiup.md b/upgrade-tidb-using-tiup.md index 0c49c9d5b49c2..32e8ddb728cad 100644 --- a/upgrade-tidb-using-tiup.md +++ b/upgrade-tidb-using-tiup.md @@ -106,7 +106,7 @@ Now, the offline mirror has been upgraded successfully. If an error occurs durin tiup cluster edit-config ``` -2. Refer to the format of [topology](https://github.com/pingcap/tiup/blob/release-1.4/embed/templates/examples/topology.example.yaml) configuration template and fill the parameters you want to modify in the `server_configs` section of the topology file. +2. Refer to the format of [topology](https://github.com/pingcap/tiup/blob/master/embed/examples/cluster/topology.example.yaml) configuration template and fill the parameters you want to modify in the `server_configs` section of the topology file. 3. After the modification, enter : + w + q to save the change and exit the editing mode. Enter Y to confirm the change.