Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

br: add more info about the br log structure #20443

Merged
merged 4 commits into from
Mar 4, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 32 additions & 26 deletions br/br-log-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,18 +108,24 @@ Log backup generates the following types of files:
.
├── v1
│   ├── backupmeta
│   │   ├── {min_restored_ts}-{uuid}.meta
│   │   ── {checkpoint}-{uuid}.meta
│   │   ├── ...
│   │   ── {resolved_ts}-{uuid}.meta
│   ├── global_checkpoint
│   │   ── {store_id}.ts
│   ── {date}
│      ── {hour}
│         ── {store_id}
│            ├── {min_ts}-{uuid}.log
│            ── {min_ts}-{uuid}.log
── v1_stream_truncate_safepoint.txt
│   │   ── {store_id}.ts
│   ── {date}
│      ── {hour}
│         ── {store_id}
│            ├── ...
│            ── {min_ts}-{uuid}.log
── v1_stream_truncate_safepoint.txt
```

Explanation of the backup file directory structure:

- `backupmeta`: stores backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards.
- `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`.
- `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore.

The following is an example:

```
Expand All @@ -129,24 +135,24 @@ The following is an example:
│   │   ├── ...
│   │   ├── 435213818858112001-e2569bda-a75a-4411-88de-f469b49d6256.meta
│   │   ├── 435214043785779202-1780f291-3b8a-455e-a31d-8a1302c43ead.meta
│   │   ── 435214443785779202-224f1408-fff5-445f-8e41-ca4fcfbd2a67.meta
│   │   ── 435214443785779202-224f1408-fff5-445f-8e41-ca4fcfbd2a67.meta
│   ├── global_checkpoint
│   │   ├── 1.ts
│   │   ├── 2.ts
│   │   ── 3.ts
│   ── 20220811
│      ── 03
│         ├── 1
│         │   ├── ...
│         │   ├── 435213866703257604-60fcbdb6-8f55-4098-b3e7-2ce604dafe54.log
│         │   ── 435214023989657606-72ce65ff-1fa8-4705-9fd9-cb4a1e803a56.log
│         ├── 2
│         │   ├── ...
│         │   ├── 435214102632857605-11deba64-beff-4414-bc9c-7a161b6fb22c.log
│         │   ── 435214417205657604-e6980303-cbaa-4629-a863-1e745d7b8aed.log
│         ── 3
│            ├── ...
│            ├── 435214495848857605-7bf65e92-8c43-427e-b81e-f0050bd40be0.log
│            ── 435214574492057604-80d3b15e-3d9f-4b0c-b133-87ed3f6b2697.log
── v1_stream_truncate_safepoint.txt
│   │   ── 3.ts
│   ── 20220811
│      ── 03
│         ├── 1
│         │   ├── ...
│         │   ├── 435213866703257604-60fcbdb6-8f55-4098-b3e7-2ce604dafe54.log
│         │   ── 435214023989657606-72ce65ff-1fa8-4705-9fd9-cb4a1e803a56.log
│         ├── 2
│         │   ├── ...
│         │   ├── 435214102632857605-11deba64-beff-4414-bc9c-7a161b6fb22c.log
│         │   ── 435214417205657604-e6980303-cbaa-4629-a863-1e745d7b8aed.log
│         ── 3
│            ├── ...
│            ├── 435214495848857605-7bf65e92-8c43-427e-b81e-f0050bd40be0.log
│            ── 435214574492057604-80d3b15e-3d9f-4b0c-b133-87ed3f6b2697.log
── v1_stream_truncate_safepoint.txt
```