Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write statediff payload data directly to Postgres #11

Closed
i-norden opened this issue Oct 8, 2020 · 1 comment
Closed

Write statediff payload data directly to Postgres #11

i-norden opened this issue Oct 8, 2020 · 1 comment
Assignees

Comments

@i-norden
Copy link
Collaborator

i-norden commented Oct 8, 2020

Packaging and sending things over HTTP is proving to be a rate-limiting step. Instead, we should adjust eth-statediff-service to write its data directly to Postgres.

The quickest way to do this is to port over the transformer interface from ipld-eth-indexer: https://github.com/vulcanize/ipld-eth-indexer/blob/master/pkg/eth/transformer.go, plugging it into the output here: https://github.com/vulcanize/eth-statediff-service/blob/master/pkg/service.go#L149

But, since we no longer need to package and send things over the wire we have an opportunity to optimize things further with a more substantial refactoring. More specifically, we could write out state/storage nodes from inside the statediff builder as it iterates over them, rather than collecting them into an object which needs to be reiterated by the ipld-eth-indexer transformer.

E.g. write out the state nodes from here, here, here, and here; and the storage nodes from here, here, here, here, here, and here

Keep in mind that due to the FK constraints we currently have on state and storage nodes (and the desire to publish everything at a given block into Postgres in a single atomic tx whether or not the FKs were present), this will require first publishing the rest of the block data into Postgres (or at least the header; i.e. before https://github.com/vulcanize/eth-statediff-service/blob/master/pkg/service.go#L135) using an sqlx.Tx and then passing that same sqlx.Tx into the builder along with the returned headerID.

Ultimately need to get all blockchain data into these Postgres tables as efficiently as possible: https://github.com/vulcanize/ipld-eth-server/tree/master/db/migrations

@i-norden
Copy link
Collaborator Author

Deprecated by #15

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants