BR: Add a new doc about the batch create table

TOC.md

@@ -72,6 +72,7 @@
         - [Back up and Restore Data on Azure Blob Storage](/br/backup-and-restore-azblob.md)
       - BR Features
         - [Auto Tune](/br/br-auto-tune.md)
+        - [Batch Create Table](/br/br-batch-create-table.md)
       - [BR FAQ](/br/backup-and-restore-faq.md)
   - [Configure Time Zone](/configure-time-zone.md)
   - [Daily Checklist](/daily-check.md)
@@ -203,6 +204,7 @@
     - [External Storages](/br/backup-and-restore-storages.md)
     - BR Features
       - [Auto Tune](/br/br-auto-tune.md)
+      - [Batch Create Table](/br/br-batch-create-table.md)
     - [BR FAQ](/br/backup-and-restore-faq.md)
   - TiDB Binlog
     - [Overview](/tidb-binlog/tidb-binlog-overview.md)

br/backup-and-restore-faq.md

@@ -164,6 +164,12 @@ You can use [`filter.rules`](https://github.com/pingcap/tiflow/blob/7c3c2336f981
 
 Yes. BR backs up the [`SHARD_ROW_ID_BITS` and `PRE_SPLIT_REGIONS`](/sql-statements/sql-statement-split-region.md#pre_split_regions) information of a table. The data of the restored table is also split into multiple Regions.
 
+## What should I do if the restore fails with the error message `the entry too large, the max entry size is 6291456, the size of data is 7690800`?
+
+Try reducing the size of the concurrent batch table creation by setting `--ddl-batch-size` to `128` or a smaller value.
+
+When using BR to restore the back up data with the value of [`--ddl-batch-size`](/br/br-batch-create-table.md#how to use) is greater than `1`, TiDB writes a DDL job to the DDL jobs queue that is maintained by TiKV. At this time, the total size of all tables schema sent by TiDB at a time should not exceed 6 MB, because the maximum value of job messages is `6 MB` by default (you are **not recommended** to modify this value; for details, see [`txn-entry-size-limit`](/tidb-configuration-file.md#txn-entry-size-limit-new-in-v50) and [`raft-entry-max-size`](/tikv-configuration-file.md#raft-entry-max-size). Therefore, if you set the too large value to `--ddl-batch-size`, the schema size of the batch table sent by TiDB at a time exceeds the specified value, resulting in BR reporting `entry too large, the max entry size is 6291456, the size of data is 7690800` error.
+
 ## Why is the `region is unavailable` error reported for a SQL query after I use BR to restore the backup data?
 
 If the cluster backed up using BR has TiFlash, `TableInfo` stores the TiFlash information when BR restores the backup data. If the cluster to be restored does not have TiFlash, the `region is unavailable` error is reported.

br/br-batch-create-table.md

@@ -0,0 +1,65 @@
+---
+title: BR Batch Create Table
+summary: Learn how to use BR batch create table feature. When restoring data, BR can use batch create table feature to speed up the restoration.
+---
+
+# BR Batch Create Table
+
+When restoring data using Backup & Restore (BR), BR creates the databases and tables in the target TiDB first, then starts restoring table data. In the versions earlier than TiDB v6.0.0, BR uses the [serial execution](#implementation-principles) scheme to create tables in the restoration process. However, when restoring data with a large number (nearly 50000) of tables, this scheme takes much time to create tables.
+
+To speed up the table creation process, and thereby reduce the time for restoring data, the BR batch create table feature is introduced in TiDB v6.0.0. This feature is enabled by default.
+
+> **Note:**
+>
+> - To use the BR batch create table feature, both TiDB and BR should be in v6.0.0 or later. If either TiDB or BR is in the version lower than v6.0.0, BR uses the serial execution scheme.
+> - Suppose that you use a cluster management tool (for example, TiUP), and your TiDB and BR are in 6.0.0 or later versions. In this case, BR enables batch create table feature by default without additional configuration.
+
+## User scenario
+
+When you need to restore data with massive tables, for example, 50000 tables, you can use BR batch create table feature to speed up the restoration process.
+
+For the detailed effect, see [Test batch create table feature](#test-batch-create-table).
+
+## Use batch create table
+
+BR enables batch create table feature and configures `--ddl-batch-size=128` by default in v6.0.0 or later. Therefore, you do not need to configure this parameter additionally. `--ddl-batch-size=128` means that BR creates tables in multiple batches, and each batch has 128 tables.
+
+To disable this feature, you can set `--ddl-batch-size` to `0` by the following command:
+
+{{< copyable "shell-regular" >}}
+
+```shell
+br restore full -s local:///br_data/ --pd 172.16.5.198:2379 --log-file restore.log --ddl-batch-size=0
+```
+
+After disabling the feature, BR uses the [serial execution scheme](#implementation-principles) instead.
+
+## Implementation principles
+
+- Serial execution scheme before v6.0.0:
+
+    In the versions earlier than 6.0.0, BR uses the serial execution scheme. When restoring data, BR creates the databases and tables in the target TiDB first, then starts restoring data. To create tables, after calling TiDB API, BR uses the SQL statement `Create Table`. TiDB DDL owner creates tables sequentially. Once the DDL owner creates a table, the schema version changes correspondingly, and each version change synchronizes to other BRs and other TiDB DDL workers. Hence, when restoring a large number of tables, the serial execution scheme takes too much time.
+
+- Batch create table scheme since v6.0.0:
+
+    The batch create table feature uses the concurrent batch table creation scheme. From v6.0.0, by default, BR creates tables in multiple batches, and each batch has 128 tables. Using this scheme, when BR creates one batch of tables, TiDB schema version only changes once. This scheme significantly increases the speed of table creation.
+
+## Test batch create table
+
+This section describes the information of testing batch create table feature. The test environment is as follows:
+
+- Cluster configurations:
+
+    - 15 TiKV instances. Each TiKV instance has 16 CPU cores, 80 GB memory, and 16 threads to process RPC requests ([`import.num-threads`](/tikv-configuration-file.md#num-threads) = 16).
+    - 3 TiDB instances. Each TiDB instance is equipped with 16 CPU cores, 32 GB memory.
+    - 3 PD instances. Each PD instance is equipped with 16 CPU cores, 32 GB memory.
+
+- Data to be restored: 16.16 TB
+
+The test result is as follows:
+
+```
+‘[2022/03/12 22:37:49.060 +08:00] [INFO] [collector.go:67] ["Full restore success summary"] [total-ranges=751760] [ranges-succeed=751760] [ranges-failed=0] [split-region=1h33m18.078448449s] [restore-ranges=542693] [total-take=1h41m35.471476438s] [restore-data-size(after-compressed)=8.337TB] [Size=8336694965072] [BackupTS=431773933856882690] [total-kv=148015861383] [total-kv-size=16.16TB] [average-speed=2.661GB/s]’
+```
+
+In the result, you can find that the average speed of restoring one TiKV instance is as high as 181.65 MB/s （`average-speed(GB/s)`/`tikv_count` = `181.65(MB/s)`）.