From 641533df37bc02e4c1f2b0a2225ad1c8e5f5e4e1 Mon Sep 17 00:00:00 2001 From: hillium Date: Tue, 4 Mar 2025 13:50:41 +0800 Subject: [PATCH 1/4] refined br-log-arch --- br/br-log-architecture.md | 58 +++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 26 deletions(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index c75f3593dc75c..a50b340e3620c 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -108,18 +108,24 @@ Log backup generates the following types of files: . ├── v1 │   ├── backupmeta -│   │   ├── {min_restored_ts}-{uuid}.meta -│   │   ├── {checkpoint}-{uuid}.meta +│   │   ├── ... +│   │   └── {resolved_ts}-{uuid}.meta │   ├── global_checkpoint -│   │   ├── {store_id}.ts -│   ├── {date} -│   │   ├── {hour} -│   │   │   ├── {store_id} -│   │   │   │   ├── {min_ts}-{uuid}.log -│   │   │   │   ├── {min_ts}-{uuid}.log -├── v1_stream_truncate_safepoint.txt +│   │   └── {store_id}.ts +│   └── {date} +│      └── {hour} +│         └── {store_id} +│            ├── ... +│            └── {min_ts}-{uuid}.log +└── v1_stream_truncate_safepoint.txt ``` +Among them: + +- `backupmeta` contains the metadata of the backup. The `resolved_ts` in the file name indicates the progress of the backup: data before this TSO has been fully backed up. However, please note that this data only reflects the progress of some shards, so this number is not very meaningful to users. +- `global_checkpoint` stores the global progress of the backup: it records the latest point in time that can be restored to using `br restore point`. +- The `{date}/{hour}` directory stores the backup data for the corresponding date. Please note that when cleaning up storage, you should use `br log truncate` instead of manually deleting the data, because the metadata points to this data though deleted manually, and manually deleting it will cause problems. + The following is an example: ``` @@ -129,24 +135,24 @@ The following is an example: │   │   ├── ... │   │   ├── 435213818858112001-e2569bda-a75a-4411-88de-f469b49d6256.meta │   │   ├── 435214043785779202-1780f291-3b8a-455e-a31d-8a1302c43ead.meta -│   │   ├── 435214443785779202-224f1408-fff5-445f-8e41-ca4fcfbd2a67.meta +│   │   └── 435214443785779202-224f1408-fff5-445f-8e41-ca4fcfbd2a67.meta │   ├── global_checkpoint │   │   ├── 1.ts │   │   ├── 2.ts -│   │   ├── 3.ts -│   ├── 20220811 -│   │   ├── 03 -│   │   │   ├── 1 -│   │   │   │   ├── ... -│   │   │   │   ├── 435213866703257604-60fcbdb6-8f55-4098-b3e7-2ce604dafe54.log -│   │   │   │   ├── 435214023989657606-72ce65ff-1fa8-4705-9fd9-cb4a1e803a56.log -│   │   │   ├── 2 -│   │   │   │   ├── ... -│   │   │   │   ├── 435214102632857605-11deba64-beff-4414-bc9c-7a161b6fb22c.log -│   │   │   │   ├── 435214417205657604-e6980303-cbaa-4629-a863-1e745d7b8aed.log -│   │   │   ├── 3 -│   │   │   │   ├── ... -│   │   │   │   ├── 435214495848857605-7bf65e92-8c43-427e-b81e-f0050bd40be0.log -│   │   │   │   ├── 435214574492057604-80d3b15e-3d9f-4b0c-b133-87ed3f6b2697.log -├── v1_stream_truncate_safepoint.txt +│   │   └── 3.ts +│   └── 20220811 +│      └── 03 +│         ├── 1 +│         │   ├── ... +│         │   ├── 435213866703257604-60fcbdb6-8f55-4098-b3e7-2ce604dafe54.log +│         │   └── 435214023989657606-72ce65ff-1fa8-4705-9fd9-cb4a1e803a56.log +│         ├── 2 +│         │   ├── ... +│         │   ├── 435214102632857605-11deba64-beff-4414-bc9c-7a161b6fb22c.log +│         │   └── 435214417205657604-e6980303-cbaa-4629-a863-1e745d7b8aed.log +│         └── 3 +│            ├── ... +│            ├── 435214495848857605-7bf65e92-8c43-427e-b81e-f0050bd40be0.log +│            └── 435214574492057604-80d3b15e-3d9f-4b0c-b133-87ed3f6b2697.log +└── v1_stream_truncate_safepoint.txt ``` From 1d253966c219bc2f6747e0924b6074beb7fc60db Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=B1=B1=E5=B2=9A?= <36239017+YuJuncen@users.noreply.github.com> Date: Tue, 4 Mar 2025 17:17:42 +0800 Subject: [PATCH 2/4] Apply suggestions from code review Co-authored-by: Lilian Lee --- br/br-log-architecture.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index a50b340e3620c..9f79f183c2cba 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -120,11 +120,11 @@ Log backup generates the following types of files: └── v1_stream_truncate_safepoint.txt ``` -Among them: +Explanation of the backup file directory structure: -- `backupmeta` contains the metadata of the backup. The `resolved_ts` in the file name indicates the progress of the backup: data before this TSO has been fully backed up. However, please note that this data only reflects the progress of some shards, so this number is not very meaningful to users. -- `global_checkpoint` stores the global progress of the backup: it records the latest point in time that can be restored to using `br restore point`. -- The `{date}/{hour}` directory stores the backup data for the corresponding date. Please note that when cleaning up storage, you should use `br log truncate` instead of manually deleting the data, because the metadata points to this data though deleted manually, and manually deleting it will cause problems. +- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards, so its significance is limited. +- `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`. +- `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore. The following is an example: From 0cc4da2e0943bb766a4e85f1fdb8cec52742d2ed Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 4 Mar 2025 17:52:19 +0800 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: BornChanger <97348524+BornChanger@users.noreply.github.com> --- br/br-log-architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index 9f79f183c2cba..08bc0466c1d89 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -122,7 +122,7 @@ Log backup generates the following types of files: Explanation of the backup file directory structure: -- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards, so its significance is limited. +- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards. - `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`. - `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore. From 7859ad1a237eeeb9a7c46665736bf401aa0d6c89 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 4 Mar 2025 18:16:19 +0800 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Grace Cai --- br/br-log-architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index 08bc0466c1d89..fedd2c98fc305 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -122,7 +122,7 @@ Log backup generates the following types of files: Explanation of the backup file directory structure: -- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards. +- `backupmeta`: stores backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards. - `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`. - `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore.