Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve] [pip] PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. #22889

Merged
merged 4 commits into from
Jun 17, 2024
Merged

[improve] [pip] PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. #22889

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1be1b37
add pip-358.
thetumbled Jun 11, 2024
112a814
add discussion thread.
thetumbled Jun 11, 2024
58fa56f
update doc.
thetumbled Jun 13, 2024
54a5c83
add vote thread.
thetumbled Jun 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions pip/pip-358.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@

# PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl.

# Background knowledge

Initially, we introduce `loadBalancerCPUResourceWeight`, `loadBalancerBandwidthInResourceWeight`, `loadBalancerBandwidthOutResourceWeight`,
`loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight` in `ThresholdShedder` to control the resource weight for
different resources when calculating the load of the broker.
Then we let it work for `LeastResourceUsageWithWeight` for better bundle placement policy.

But https://github.com/apache/pulsar/pull/19559 and https://github.com/apache/pulsar/pull/21168 have pointed out that the actual load
of the broker is not related to the memory usage and direct memory usage, thus we have changed the default value of
`loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight` to 0.0.

There are still some places where memory usage and direct memory usage are used to calculate the load of the broker, such as
`OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl`. We should let the resource weight work for these places
so that we can set the resource weight to 0.0 to avoid the impact of memory usage and direct memory usage on the load of the broker.

# Motivation

The actual load of the broker is not related to the memory usage and direct memory usage, thus we should let the resource weight work for
`OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl` so that we can set the resource weight to 0.0 to avoid the impact of
memory usage and direct memory usage on the load of the broker.


# Detailed Design

Let resource weight work for `OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl`.
- For `OverloadShedder`, `LeastLongTermMessageRate`, we replace `getMaxResourceUsage()` with `getMaxResourceUsageWithWeight()` in the calculation of the load of the broker.
- For `ModularLoadManagerImpl`, we replace `getMaxResourceUsage()` with `getMaxResourceUsageWithWeight()` when checking if the broker is overloaded and decide whether to update the broker data to metadata store.

# Backward & Forward Compatibility


# Links

* Mailing List discussion thread: https://lists.apache.org/thread/lj34s3vmjbzlwmy8d66d0bsb25vnq9ky
* Mailing List voting thread: https://lists.apache.org/thread/b7dzm0yz6l40pkxmxhto5mro7brmz57r