Zero downtime deploy

[gte620v]

You guys are doing great work here. As a long-time subgraph user, I like everything about ponder more than the subgraph. Thanks for building!

Question/Discussion:

It seems that the ponder/Railway ZDT deploy example depends on this sequence of events:

• commit A is pushed to RW (railway), commit builds, indexes, pushes psql views to public. this is running in RW instance-A
• at some time later, commit B is pushed and RW starts building instance-B
• when instance-B finishes building, instance-A is killed
• now, the db is still serving the data from instance-A (via the public views) while instance-B catches up to the current block
• but, no new blocks are available while the catch up happens. and the catch up could take hours/days. in that time, the data will be stale, which isn't really ZDT.
• when instance-B gets to the head, its tables are used for the public views and now all is good again.

It seems that the RW mechanism for ZDT deploys is to poll a healthcheck path: https://docs.railway.app/reference/healthchecks

In that scenario, instance-A would continue to run and sequence to its table while instance-B gets to the block head. During indexing, /health returns 503s and when indexing is current, it returns 200s triggering RW to kill instance-A.

I tried this in RW and the polling of the /health endpoint works fine, but it seems that you can't have two ponder instances running at the same time. As soon as instance-B starts, I see error like the following in instance-A:

WARN  process    Received fatal error, starting shutdown sequence

WARN  sync       Failed 'insertInterval' database method after '10' attempts

error: cannot execute INSERT in a read-only transaction

and in instance-B, i see errors like:

ERROR indexing   Internal error occurred while flushing cache. Please report this error here: https://github.com/ponder-sh/ponder/issues

WARN  indexing   Failed '[EVENT].flush' database method 

ERROR indexing   Error while processing '[...]' event in '[...]' block 14817531

FlushError: cannot execute INSERT in a read-only transaction

It seems strange that there is a locking problem since the two instances are writing to different tables. Are they trying to write to the same background ponder tables and that is the problem?

Any suggestions to not have the indexing gap in my deploys?