You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Packaging and sending things over HTTP is proving to be a rate-limiting step. Instead, we should adjust eth-statediff-service to write its data directly to Postgres.
But, since we no longer need to package and send things over the wire we have an opportunity to optimize things further with a more substantial refactoring. More specifically, we could write out state/storage nodes from inside the statediff builder as it iterates over them, rather than collecting them into an object which needs to be reiterated by the ipld-eth-indexer transformer.
Keep in mind that due to the FK constraints we currently have on state and storage nodes (and the desire to publish everything at a given block into Postgres in a single atomic tx whether or not the FKs were present), this will require first publishing the rest of the block data into Postgres (or at least the header; i.e. before https://github.com/vulcanize/eth-statediff-service/blob/master/pkg/service.go#L135) using an sqlx.Tx and then passing that same sqlx.Tx into the builder along with the returned headerID.
Packaging and sending things over HTTP is proving to be a rate-limiting step. Instead, we should adjust eth-statediff-service to write its data directly to Postgres.
The quickest way to do this is to port over the transformer interface from ipld-eth-indexer: https://github.com/vulcanize/ipld-eth-indexer/blob/master/pkg/eth/transformer.go, plugging it into the output here: https://github.com/vulcanize/eth-statediff-service/blob/master/pkg/service.go#L149
But, since we no longer need to package and send things over the wire we have an opportunity to optimize things further with a more substantial refactoring. More specifically, we could write out state/storage nodes from inside the statediff builder as it iterates over them, rather than collecting them into an object which needs to be reiterated by the ipld-eth-indexer transformer.
E.g. write out the state nodes from here, here, here, and here; and the storage nodes from here, here, here, here, here, and here
Keep in mind that due to the FK constraints we currently have on state and storage nodes (and the desire to publish everything at a given block into Postgres in a single atomic tx whether or not the FKs were present), this will require first publishing the rest of the block data into Postgres (or at least the header; i.e. before https://github.com/vulcanize/eth-statediff-service/blob/master/pkg/service.go#L135) using an sqlx.Tx and then passing that same sqlx.Tx into the builder along with the returned headerID.
Ultimately need to get all blockchain data into these Postgres tables as efficiently as possible: https://github.com/vulcanize/ipld-eth-server/tree/master/db/migrations
The text was updated successfully, but these errors were encountered: