DynamoDB Streams

The Case For/Against Long Polling in discfg

DynamoDB UpdateTable() is a little annoying. You can only set one option per UpdateTable operation. For example, this means change ProvisionedThroughput OR StreamSpecification. While it's updating, another UpdateTable operation can not be performed. It's an async operation.

So, a user has to issue multiple commands to change things AND do so over time, after the first update is done and the table is back in an ACTIVE state. Annoying. Not intuitive...But we can work with it for now.

Options:

• Fortunately only two options we care about here. So maybe only allow streaming to be set upon creation.
• Document the issue and just let users know they need to make multiple calls over time. Only adjust one thing at a time.
• Don't use streams ever. Then there's only one thing that can be updated.
  Note: streams can be enabled via AWS console or another app using the AWS SDK or from CLI. So maybe let the user manage that outside of discfg.

I really like the idea of streams. BUT since discfg is not meant to be a long polling kinda tool, there's no real need for them for discfg. It would definitely add something to discfg that etcd has...But discfg is no longer concerned with feature parity with etcd.

Why is there no long polling for discfg? It's designed to be used with API Gateway which doesn't have that feature and it's also designed for use with Lambda which has a max run time of 5 minutes. However, discfg can be self hosted...So it's possible to have it, just not in all use cases.

Anyone can turn on a stream for a DynamoDB table if needed so I don't think discfg needs to worry about it. Though this could change.

Other Storage Engines

Another thing discfg is looking to do is utilize other storage engines, such as S3. S3 can certainly feed into Kinesis, but other storage engines might not.

So in order to get long polling (via Kinesis anyway) for other storage engines is to now use a second storage engine that does support Kinesis OR to have discfg put data into Kinesis directly (which is a more efficient way to do it).

If discfg is to do that, then there is really less of a need for DynamoDB streams. Or none at all.

In the future, it may simply be nice to stream to Kinesis and call it a day. Perhaps also an option to trigger another Lambda with the event message being the changed key value...Which could turn things into a push system instead of a polling system. There's also SNS of course.

So is long polling the way to go? Maybe...But when you're in AWS' infrastructure, it might be less important.