Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add estimate_pending_compaction_bytes metric to analyze fragmentation rate #1736

Merged
merged 1 commit into from
Jul 17, 2023
Merged

feat: add estimate_pending_compaction_bytes metric to analyze fragmentation rate #1736

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/storage/src/redis.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,9 @@ void Redis::GetRocksDBInfo(std::string &info, const char *prefix) {
write_stream_key_value(rocksdb::DB::Properties::kTotalSstFilesSize, "total_sst_files_size");
write_stream_key_value(rocksdb::DB::Properties::kLiveSstFilesSize, "live_sst_files_size");

// pending compaction bytes
write_stream_key_value(rocksdb::DB::Properties::kEstimatePendingCompactionBytes, "estimate_pending_compaction_bytes");

// block cache
write_stream_key_value(rocksdb::DB::Properties::kBlockCacheCapacity, "block_cache_capacity");
write_stream_key_value(rocksdb::DB::Properties::kBlockCacheUsage, "block_cache_usage");
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the code patch provided, here are some observations and suggestions:

Observations:

  • The code is adding a new line to write the "estimate_pending_compaction_bytes" property to the info stream.
  • It appears that this code is part of a larger function called GetRocksDBInfo.
  • No other specific details about the overall function or the context in which this code is used are provided.

Suggestions:

  1. Confirm Dependency: Ensure that you have verified that the rocksdb::DB::Properties::kEstimatePendingCompactionBytes property is available and works as expected in your version of RocksDB.

  2. Error Handling: It's important to handle any potential errors that may occur during the process of retrieving and writing the properties. Make sure appropriate error handling mechanisms (e.g., try-catch) are implemented.

  3. Code Documentation: Consider adding comments to describe the purpose of this function and the significance of each property being written to the info stream.

  4. Review Other Properties: Since this code patch only includes the addition of the "estimate_pending_compaction_bytes" property, it's worth considering if there are any other relevant properties that should be included in the info stream.

  5. Unit Testing: Test the functionality of this code patch thoroughly to ensure it produces the desired results and doesn't introduce any unexpected issues. Verify that the info stream contains all the expected properties and their corresponding values.

Remember that without additional context and a complete understanding of your codebase and requirements, it's difficult to provide a comprehensive review.

Expand Down
55 changes: 28 additions & 27 deletions tools/pika_exporter/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,33 +142,34 @@ prometheus --config.file=./grafana/prometheus.yml

### Rocksdb Metrics

| Serial Number | Metric | Meaning |
| ------------- | ------ | ------- |
| 11 | rocksdb.num-immutable-mem-table | Number of immutable memtables not yet flushed. |
| 12 | rocksdb.num-immutable-mem-table-flushed | Number of immutable memtables that have been flushed. |
| 13 | rocksdb.mem-table-flush-pending | Returns 1 if there is a pending memtable flush; otherwise returns 0. |
| 14 | rocksdb.num-running-flushes | Number of currently running flush operations. |
| 15 | rocksdb.compaction-pending | Returns 1 if at least one compaction operation is pending; otherwise returns 0. |
| 16 | rocksdb.num-running-compactions | Number of running compactions. |
| 17 | rocksdb.background-errors | Total number of background errors. |
| 18 | rocksdb.cur-size-active-mem-table | Approximate size, in bytes, of the active memtable. |
| 19 | rocksdb.cur-size-all-mem-tables | Total size in bytes of memtables not yet flushed, including the current active memtable and the unflushed immutable memtables. |
| 20 | rocksdb.size-all-mem-tables | Total size in bytes of all memtables, including the active memtable, unflushed immutable memtables, and pinned immutable memtables. |
| 25 | rocksdb.estimate-num-keys | Estimated number of keys in active memtable, unflushed immutable memtables, and flushed SST files. |
| 26 | rocksdb.estimate-table-readers-mem | Estimated memory size used for reading SST files, excluding block cache (such as filter and index blocks). |
| 28 | rocksdb.num-snapshots | Number of unreleased snapshots in the database. |
| 31 | rocksdb.num-live-versions | Number of current versions. More current versions usually indicate more SST files being used by iterators or incomplete compactions. |
| 32 | rocksdb.current-super-version-number | Current number of the LSM version. It is a uint64_t integer that increments after any changes in the LSM tree. This number is not preserved after restarting the database and starts from 0 after a database restart. |
| 33 | rocksdb.estimate-live-data-size | Estimated size of the activity data in bytes.For BlobDB, it also includes the actual live bytes in the version's blob file. |
| 36 | rocksdb.total-sst-files-size | Total size (in bytes) of all SST files.Note: If there are too many files, it may slow down the online query. |
| 37 | rocksdb.live-sst-files-size | Total size in bytes of all SST files belonging to the latest LSM tree. |
| 47 | rocksdb.block-cache-capacity | The capacity of the block cache. |
| 48 | rocksdb.block-cache-usage | Memory size occupied by entries in the block cache. |
| 49 | rocksdb.block-cache-pinned-usage | Memory size occupied by pinned entries in the block cache. |
| 51 | rocksdb.num-blob-files | The number of blob files in the current version. |
| 52 | rocksdb.blob-stats | The total and size of all blob files, and the total amount of garbage (in bytes) in blob files in the current version. |
| 53 | rocksdb.total-blob-file-size | The total size of all blob files across all versions. |
| 54 | rocksdb.live-blob-file-size | The total size of all blob files in the current version. |
| Serial Number | Metric | Meaning |
|---------------| ------ |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 11 | rocksdb.num-immutable-mem-table | Number of immutable memtables not yet flushed. |
| 12 | rocksdb.num-immutable-mem-table-flushed | Number of immutable memtables that have been flushed. |
| 13 | rocksdb.mem-table-flush-pending | Returns 1 if there is a pending memtable flush; otherwise returns 0. |
| 14 | rocksdb.num-running-flushes | Number of currently running flush operations. |
| 15 | rocksdb.compaction-pending | Returns 1 if at least one compaction operation is pending; otherwise returns 0. |
| 16 | rocksdb.num-running-compactions | Number of running compactions. |
| 17 | rocksdb.background-errors | Total number of background errors. |
| 18 | rocksdb.cur-size-active-mem-table | Approximate size, in bytes, of the active memtable. |
| 19 | rocksdb.cur-size-all-mem-tables | Total size in bytes of memtables not yet flushed, including the current active memtable and the unflushed immutable memtables. |
| 20 | rocksdb.size-all-mem-tables | Total size in bytes of all memtables, including the active memtable, unflushed immutable memtables, and pinned immutable memtables. |
| 25 | rocksdb.estimate-num-keys | Estimated number of keys in active memtable, unflushed immutable memtables, and flushed SST files. |
| 26 | rocksdb.estimate-table-readers-mem | Estimated memory size used for reading SST files, excluding block cache (such as filter and index blocks). |
| 28 | rocksdb.num-snapshots | Number of unreleased snapshots in the database. |
| 31 | rocksdb.num-live-versions | Number of current versions. More current versions usually indicate more SST files being used by iterators or incomplete compactions. |
| 32 | rocksdb.current-super-version-number | Current number of the LSM version. It is a uint64_t integer that increments after any changes in the LSM tree. This number is not preserved after restarting the database and starts from 0 after a database restart. |
| 33 | rocksdb.estimate-live-data-size | Estimated size of the activity data in bytes.For BlobDB, it also includes the actual live bytes in the version's blob file. |
| 36 | rocksdb.total-sst-files-size | Total size (in bytes) of all SST files.Note: If there are too many files, it may slow down the online query. |
| 37 | rocksdb.live-sst-files-size | Total size in bytes of all SST files belonging to the latest LSM tree. |
| 40 | rocksdb.estimate-pending-compaction-bytes | Estimated total number of bytes that compression needs to rewrite to bring all levels down below the target size. Has no effect on compression other than level-based compression. |
| 47 | rocksdb.block-cache-capacity | The capacity of the block cache. |
| 48 | rocksdb.block-cache-usage | Memory size occupied by entries in the block cache. |
| 49 | rocksdb.block-cache-pinned-usage | Memory size occupied by pinned entries in the block cache. |
| 51 | rocksdb.num-blob-files | The number of blob files in the current version. |
| 52 | rocksdb.blob-stats | The total and size of all blob files, and the total amount of garbage (in bytes) in blob files in the current version. |
| 53 | rocksdb.total-blob-file-size | The total size of all blob files across all versions. |
| 54 | rocksdb.live-blob-file-size | The total size of all blob files in the current version. |

## Grafana Dashboard ##
See [here](./grafana/grafana_prometheus_pika_dashboard.json)
Expand Down
16 changes: 16 additions & 0 deletions tools/pika_exporter/exporter/metrics/rocksdb.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,22 @@ var collectRocksDBMetrics = map[string]MetricConfig{
},
},

// pending compaction bytes
"estimate_pending_compaction_bytes": {
Parser: &regexParser{
name: "estimate_pending_compaction_bytes",
reg: regexp.MustCompile(`(?P<data_type>\w+)_.*?estimate_pending_compaction_bytes:(?P<estimate_pending_compaction_bytes>\d+)`),
Parser: &normalParser{},
},
MetricMeta: &MetaData{
Name: "estimate_pending_compaction_bytes",
Help: "Estimated total number of bytes that compression needs to rewrite to bring all levels down below the target size. Has no effect on compression other than level-based compression.",
Type: metricTypeGauge,
Labels: []string{LabelNameAddr, LabelNameAlias, "data_type"},
ValueName: "estimate_pending_compaction_bytes",
},
},

// snapshots
"num_snapshots": {
Parser: &regexParser{
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch you provided adds a new metric configuration for "estimate_pending_compaction_bytes" to the existing collectRocksDBMetrics map. Here's a brief code review:

  1. Syntax and Formatting: The code appears to be formatted correctly, with indentation and line breaks.

  2. Regular Expression: The regular expression used in the reg field of the regexParser struct seems fine. However, it's always good to thoroughly test a regular expression to ensure it matches the intended patterns.

  3. MetaData Configuration: The MetricMeta struct defines metadata for the new metric. It includes the metric name, help description, type (presumably "gauge"), labels, and value name. As long as these fields are accurate and appropriate for your use case, the configuration seems fine.

Overall, from the given code patch, there don't appear to be any obvious bugs or risks. However, it's essential to thoroughly test the code to ensure proper functionality. Additionally, it's challenging to provide further improvement suggestions without more context about the codebase and its requirements.

Expand Down