Updates to Hystrix timeout setting are not applied

[cb372]

1. Register a slow endpoint that takes 1.5 seconds to respond. Give it a Hystrix timeout of 1 second.
2. Make one or more requests to the endpoint. They will time out.
3. Edit the endpoint in the admin UI. Change the Hystrix timeout to 2 seconds.

Expected result: Hystrix timeout changes to 2 seconds, so requests no longer timeout

Actual result: Hystix timeout is still 1 second

[lloydmeta]

Maybe related to #18

[mauhiz]

It looks like a new handler (and so a new HystrixExecutor) is made on each processWithConfig. There must be some failure to clear the cache on admin screen update.

[lloydmeta]

I looked into this a bit and it seems like the configs caches are being cleared properly and subsequent retrievals from cache show the updated timeout values.

Will look into this further to see if this is something on the Hystrix-level that we didn't know about/understand.

[cb372]

As @lloydmeta guessed, the old timeout value is being cached by Hystrix. Specifically the HystrixPropertiesFactory class. We are correctly passing the updated timeout value to the HystrixCommand constructor, but it is being ignored.

Hystrix does not provide any methods to clear/manually update this cache, so the options are:

• hack around the private modifier using reflection (NO!!!)
• change the Hystrix command key whenever the timeout setting is changed. This is the key that is used to cache the settings (tricky, as the command key is currently user-configurable)
• use a Hystrix plugin to customise how the properties cache key is generated (also tricky - would need to store the cache key as a DB field)
• send a PR to Hystrix and pray for their blessing
• close as WONTFIX and add a note to the docs

[lloydmeta]

change the Hystrix command key whenever the timeout setting is changed. This is the key that is used to cache the settings (tricky, as the command key is currently user-configurable)

Another issue with this is that it will make a mess of the Hystrix dashboard because you'll get a new "circuit" widget whenever you use a new command key.

It seems like ones with the least number of disadvantages are the last 3 (plugin, PR, or wontfix).

[mauhiz]

Voting for the reflection trick (at least while preparing a PR). We are not
building a piece if art, and it looks like an acceptable workaround for
such a corner case.

[lloydmeta]

It looks like we can simply implement a Hystrix plugin that has getCommandPropertiesCacheKey return null and leave everything else the same as the default. Sure, we lose caching of the properties for a given command key, but it's another alternative.

[mauhiz]

Fixed by #36