Handling of Bulk Messages and Rate Limits

[Linuturk]

We recently implemented a workaround that solves a possible rate limiting issue for Slack by collecting a bunch of rapid fire small messages and pushing them as one large message.

hubot-archive/hubot-circleci#12

It is unclear if we actually hit a rate limit, or if something else is wonky, but we wanted to get your input on this.

• Do you perform error handling when a rate limit error message is returned via the API?
• How would we catch that exception in the example I linked above?
• Do you have any buffer or queue implemented that would gather up these many small messages, and resend them as a bulk message if the rate limit is hit?

[evansolomon]

How many messages are you sending when this error happens (order of magnitude answer here would be fine)? Do you happen to have any kind of error message/stack trace/etc that might give more info on exactly what happened?

[Linuturk]

We saw the errors when we sent more than 40+ messages within a second or two. Unfortunately, hubot crashed and didn't give us any meaningful error message. You can see the errors returned here: hubot-archive/hubot-circleci#11

[Linuturk]

@evansolomon any word on this?

[Linuturk]

@evansolomon ?

[technicalpickles]

Slack has been experimenting with rate limiting for bot users lately. As far as I can tell, these are logged:

[Tue Sep 15 2015 16:08:07 GMT-0700 (PDT)] WARNING Received rate limiting error {"code":-1,"msg":"slow down, too many messages..."}
[Tue Sep 15 2015 16:08:07 GMT-0700 (PDT)] WARNING Received rate limiting error {"code":-1,"msg":"slow down, too many messages..."}
[Tue Sep 15 2015 16:08:07 GMT-0700 (PDT)] WARNING Received rate limiting error {"code":-1,"msg":"slow down, too many messages..."}
[Tue Sep 15 2015 16:08:07 GMT-0700 (PDT)] WARNING Received rate limiting error {"code":-1,"msg":"slow down, too many messages..."}
[Tue Sep 15 2015 16:08:07 GMT-0700 (PDT)] WARNING Received rate limiting error {"code":-1,"msg":"slow down, too many messages..."}

I'm not sure yet if all the messages are going through or not. I've been able to reproduce this pretty easily with our .help command because of the sheer number of commands. If you send a message large enough to trigger splitting it up into multiple messages, that should be enough to reproduce. For testing, I generated a bunch of bacon ipsum, and had a script like:

  robot.respond /bacon/i, (msg) ->
    msg.send """
<PASTE BACON HERE>
"""

It does seem that if you hit this too aggressively, you'll get disconnected. hubot will reconnect, but then you run in #210

[silarsis]

I'm also running into rate limits - we're using hubot as a central tool for a bunch of things our people are doing and as the number of people and tasks scales up, the number of responses hubot is sending is breaking the rate limit. Looks to me like we might need to implement something in the adaptor that slows down responses...

[xalakox]

I was running with this issue on hubot 2.17.0; and the bot wouldn't reconnect it just stalled; had to do the HUBOT_SLACK_EXIT_ON_DISCONNECT="true" env variable .. nasty but the reconnection never happened.

[b0bbywan]

I'm experiencing this issue too. From the documentation https://api.slack.com/docs/rate-limits, I was thinking it would be possible to modify the adapter to check the slack callback answer for HTTP 429 Too Many Requests, and if so, falling back to the sendBulkMessage until the Retry-After header is reached and we can send the bulked message. What do you think ? I'd try it but I'm a coffee and js newbie.

[UncannyBingo]

Agree that this is something that should be dealt with. The RTM API has its own rate limit (approx 1 message / second), and it returns an event in case you hit the rate limiter (rather than a 429 with the web APIs), that should be heeded.