Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove foreign keys on entity id #817

Merged
merged 8 commits into from
Jun 11, 2020
Merged

Remove foreign keys on entity id #817

merged 8 commits into from
Jun 11, 2020

Conversation

apeksharma
Copy link
Contributor

@apeksharma apeksharma commented Jun 9, 2020

Detailed description:

  • Replace foreign keys columns - cud entity, payer account, node account - with encoded entity ids
  • Use encoded ids in REST to remove joins with t_entities table
  • Migration testing: Migrating db with ~2.93m transactions took ~20sec. Extrapolating, mainnet with ~150m txns should take less than 30min
  • Change DataGenerator to use EntityId rather than just a long for entity num
  • Remove dead functions from EntityRepository
  • Now auto renew accounts and proxy accounts are inserted in batch (with other PreparedStatements) rather than synchronously during transaction processing
  • There is one remaining entity lookup, will fix that later. It won't affect performance of crypto transfer heavy workloads

Perf testing:
Importer:
- Setup
Db preloaded with 2.93 million transactions
Workload: ~700k txns (20k tps, 7 files)
- Stats
Migration time: 20s
TPS:

2020-06-09 05:02:54,076 INFO  [scheduling-4] c.h.m.i.p.r.RecordFileParser Finished parsing 100034 transactions from record file ./data/recordstreams/valid/2020-02-09T20_00_00.004108Z.rcd in 16888ms (5923/s)
2020-06-09 05:03:06,175 INFO  [scheduling-4] c.h.m.i.p.r.RecordFileParser Finished parsing 99995 transactions from record file ./data/recordstreams/valid/2020-02-09T20_01_40.009724Z.rcd in 12061ms (8290/s)
2020-06-09 05:03:18,396 INFO  [scheduling-4] c.h.m.i.p.r.RecordFileParser Finished parsing 99997 transactions from record file ./data/recordstreams/valid/2020-02-09T20_03_20.004973Z.rcd in 12209ms (8190/s)
2020-06-09 05:03:30,431 INFO  [scheduling-4] c.h.m.i.p.r.RecordFileParser Finished parsing 100028 transactions from record file ./data/recordstreams/valid/2020-02-09T20_05_00.011814Z.rcd in 12023ms (8319/s)
2020-06-09 05:03:42,546 INFO  [scheduling-4] c.h.m.i.p.r.RecordFileParser Finished parsing 100004 transactions from record file ./data/recordstreams/valid/2020-02-09T20_06_40.019463Z.rcd in 12104ms (8262/s)
2020-06-09 05:04:00,857 INFO  [scheduling-4] c.h.m.i.p.r.RecordFileParser Finished parsing 99998 transactions from record file ./data/recordstreams/valid/2020-02-09T20_08_20.006814Z.rcd in 18300ms (5464/s)
2020-06-09 05:04:24,795 INFO  [scheduling-4] c.h.m.i.p.r.RecordFileParser Finished parsing 100008 transactions from record file ./data/recordstreams/valid/2020-02-09T20_10_00.001180Z.rcd in 23927ms (4179/s)

Which issue(s) this PR fixes:
Fixes #570
Partly addresses #548

Special notes for your reviewer:

Checklist

  • Documentation added
  • Tests updated

@apeksharma apeksharma added enhancement Type: New feature P1 performance database Area: Database labels Jun 9, 2020
@apeksharma apeksharma added this to the Mirror 0.14.0 milestone Jun 9, 2020
@apeksharma apeksharma self-assigned this Jun 9, 2020
@apeksharma apeksharma requested review from Nana-EC and steven-sheehy and removed request for Nana-EC June 9, 2020 04:12
@apeksharma apeksharma force-pushed the test_refactor branch 2 times, most recently from 5ebf23d to 132863a Compare June 9, 2020 19:37
Base automatically changed from test_refactor to master June 9, 2020 20:05
Signed-off-by: Apekshit Sharma <[email protected]>
Signed-off-by: Apekshit Sharma <[email protected]>
Signed-off-by: Apekshit Sharma <[email protected]>
Signed-off-by: Apekshit Sharma <[email protected]>
Copy link
Contributor

@steven-sheehy steven-sheehy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. There's a couple that need to be fixed and the rest can be addressed in a follow up.

@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should later convert this id to EntityId

@apeksharma apeksharma added P2 and removed P1 labels Jun 11, 2020
Copy link
Contributor

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huge change. Nice job.
Left some comments but you can address them if you so chose in a future smaller PR :)

@apeksharma
Copy link
Contributor Author

Thanks for the review guys, will post quick followup PR.

@apeksharma apeksharma merged commit 620cc9a into master Jun 11, 2020
@apeksharma apeksharma deleted the denorm_encode branch June 11, 2020 18:32
apeksharma added a commit that referenced this pull request Jun 11, 2020
Signed-off-by: Apekshit Sharma <[email protected]>
apeksharma added a commit that referenced this pull request Jun 11, 2020
Signed-off-by: Apekshit Sharma <[email protected]>
apeksharma added a commit that referenced this pull request Jun 11, 2020
Signed-off-by: Apekshit Sharma <[email protected]>
apeksharma added a commit that referenced this pull request Jun 11, 2020
Signed-off-by: Apekshit Sharma <[email protected]>
apeksharma added a commit that referenced this pull request Jun 12, 2020
Signed-off-by: Apekshit Sharma <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
database Area: Database enhancement Type: New feature P2 performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Denormalize t_transactions table
3 participants