Feature/distributed state registry #312
Conversation
|
Just a question. Wouldn't it be better to have a state registry connector instead and use existing services, e.g. etcd, zookeeper or consul? |
|
We feel that it's important for deepstream going forward to be a truly distributed system without master/slave relationships or a centralized state registry (if we needed that, we could have just written to the cache) |
|
@WolframHempel: All of the tree software I listed are distributed key-value stores... I'm not saying it's a bad idea do have it native in ds. Just wondering if the option has been considered. Implementing a consensus algorithm is no small task... |
| } | ||
|
|
||
| return checkSum; | ||
| } |
There was a problem hiding this comment.
this is a very naive checksum. Sure it's good enough?
There was a problem hiding this comment.
Could you elaborate on the problem you see?
There was a problem hiding this comment.
Creating input values that return the same checksum is rather trivial... e.g. "ab" would have the same checksum as "ba".
There was a problem hiding this comment.
yup, but we're using the sum of the checksums to get a rough impression of state-consistency. This happens every time something is added to the registry, so its a trade off between performance and uniqueness. Happy to go for something more elaborate though if you've got an algorithm in mind that doesn't come with significant performance overhead
There was a problem hiding this comment.
Just doing something like:
checkSum = ( ( checkSum << 5 ) - checkSum ) + name.charCodeAt( i );
checkSum = checkSum & checkSum; // Convert to 32bit integerWould be a huge improvement.
http://werxltd.com/wp/2010/05/13/javascript-implementation-of-javas-string-hashcode-method/
There was a problem hiding this comment.
Works for me 👍 (not too sure if we need the added checkSum & checkSum though)
|
@ronag there was a discussion about whether we can leverage other technologies ability for that. Problem is however that we have a flexible set of plugins and we don't want to suddenly tie ourselves down. We are also working on a lightweight peer2peer messaging layer incase a distributed message system is considered overkill, which wouldn't work without native implementation. Hope that clears it up! |
Setting anonymous record with same name discards and resubscribes the same record, it should just ignore it Fixes deepstreamIO#312
The distributed state registry on its own. I think this might be useful for a couple of cases, e.g. keeping remote server urls for failover, keeping track of listeners etc.
There is a todo left which regards cluster-presence. In order to make sure that all nodes within a cluster are still present, we might want to implement a global registry, rather than many local ones. Let's talk about this on Monday.