rates_mode not configurable on management agent only nodes.

[databus23]

Since we upgraded our 3 node cluster to rabbitmq 3.5.1 we are seeing these messages in the management ui:

The management statistics database currently has a queue of X events to process. If this number keeps increasing, so will the memory used by the management plugin. You may find it useful to set the rates_mode config item to none.

So as suggested we tried to set rates_mode to none on all three cluster members but only the node where the rabbitmq_management plugin is running picks up the configuration setting from the config file.
On the two other nodes we only run the rabbitmq_management_agentplugin and the management ui show these nodes as running in basic rates_mode even with

 {rabbitmq_management, [
   {rates_mode, none}
 ]}

in the rabbitmq config file.

So it seems only the rabbitmq_management plugin actually loads the {rabbitmq_management, ...} stanza from the config file and nodes only running the agent plugin will default to basic mode regardless of what is specified in the config file.

[michaelklishin]

Yes, the problem is that loading configuration for an app that wasn't started from another app is tricky.

Not that many people run just the agent alone (or even know what it is). @dumbbell do we want to fix this or just ask people to run "the entire" management plugin?

[dumbbell]

I didn't look at the rabbitmq_management and rabbitmq_management_agent applications yet, but could it make sense to move this paramter to the latter?

[michaelklishin]

We can attempt loading rabbitmq_management but I'm not sure if there can be edge cases when only the agent one is enabled.

No, we can't just change the parameter — it would be breaking change. We can only introduce one more setting for people who know what they're doing and only run the agent. Or just ask everybody to run both.

[dumbbell]

Yes, it would be a change to make in 3.6.0. Meanwhile, we could document the fact rabbitmq_management must be enabled.

About application loading, it's worth a try.

[databus23]

I'm not really familiar with the inner workings of the management plugin but we moved it to a dedicated node because of memory issues on our main/productive node. I was under the assumption that this is some kind of "best practice". Is there no memory penalty involved in running the full management plugin on every node?

[michaelklishin]

There is only one management database node in a cluster at any time, even if you have management plugin enabled on every node.

[michaelklishin]

I don't think breaking this even for 3.6.0 is a good idea. We'll try pre-loading and if it works, so be it. If not, then enabling the entire plugin will be required.

[databus23]

@michaelklishin Ok, is there a way of controlling which node holds the management database if the management_plugin is running on every node of a cluster?

[michaelklishin]

@databus23 there's no dedicated command for that. Perhaps we should consider adding it. Please file another issue if you feel it's important.

[michaelklishin]

I've tried loading rabbitmq_management. Unfortunately, this won't work: we only "expand" (unpack) plugins that were enabled, so rabbitmq_management is not on the code path.

I'll add a rabbitmq_management_agent option to fall back to because your use case seems reasonable. rabbitmq_management will still take precedence.

[databus23]

Adding the rabbitmq_management_agent option is perfect. Thanks.

[michaelklishin]

We haven't merged it yet ;)

[michaelklishin]

@databus23 this will be in 3.5.2. You'll be able to use rabbitmq_management_agent instead of rabbitmq_management in the config but only when rabbitmq_management is not enabled.

[databus23]

@michaelklishin That's awesome! Thanks a lot for the quick response on this.