sidechain single shard benchmark

[brenzi]

We claim to provide a solution to dramatically improve the scalability of blockchains in both tps and confirmation time latency. A benchmark will be taken out in order to prove this claim and quantify the limits of the implementation. We would expect the single-shard performance to be in the order of substrate’s tps throughput but with much lower confirmation times. It is further expected that horizontally scaling to multiple independent shards will be approximately linear until the block confirmations by themselves congest layer one. Empirically quantifying the amount of shards that can be served by a single polkadot parachain would be too much effort and is out of scope. Instead, we will provide a theoretical estimate of total tps based on the number and complexity of layer one extrinsics.
single-shard performance

• 3 workers on 3 Intel SGX machines in at least two different locations.
• layer one is a local testnet (not relevant for this test)
• send extrinsics (balance transfers) to all three workers at increasing frequency. monitor txpool and confirmation time ( Ready, InBlock ).
  *No state queries
  • No dependency on layer one state
• Repeat and add state queries (TrustedGetter) after each InBlock over RPC.
• increase number of clients until performance issues arrise -> we want to find an upper limit of number of clients.

Substrate api client example: https://github.com/scs/substrate-api-client/blob/master/src/examples/example_benchmark_bulk_xt.rs

[haerdib]

Rust intern benchmarking: https://nickb.dev/blog/guidelines-on-benchmarking-and-rust with criterion, github

[haerdib]

Linking the following issue: #231. if we use our cli, this might affect the benchmarking

[brenzi]

There is an example in the substrate-api-client which swamps a node with extrinsics. But that would require to write the entire benchmark in rust and not depend on python or bash to wrap our use case logic.

But I guess if we want the highest stress bandwidth possible, we should do that anyway

[masapr]

There exists already code in typescript to benchmark substrate:
https://github.com/paritytech/sub-flood
https://github.com/nikvolf/sub-bench
The links are from: https://wiki.polkadot.network/docs/build-tools-index#tools

It might make sense to setup a benchmark similar to sub-flood, but written in Rust. Also: to really compare the performance with substrate, the benchmark should also be run against substrate nodes, being run on the same hardware.

[masapr]

A few remarks on the current state (... in German ...):

Probleme mit Threading

• Parallelisierung von rayon Bibliothek (siehe paralleler Loop in trusted_commands.rs Zeile 263-265) funktioniert nicht richtig / Transaktionen werden nicht so häufig abgesetzt, wie man hoffen würde. Ansonsten keine Probleme, Code läuft.
• Parallelisierung mit std::thread::spawn -> hier kommt ein Fehler, dass der Code im Thread "static" sein muss.

Daher würde ich jetzt eher mehrere Prozesse starten, statt weiter am Threading von einem Prozess zu arbeiten.

Ablauf Benchmark
Folgender Ablauf ist jetzt so implementiert (und macht m. E. auch so Sinn):

• Erst werden n Tupel von Accounts angelegt (account1, account2). Beide erhalten einen fixen Betrag von Alice (100000). -> dies wird synchron durchgeführt.
• Anschliessend werden n Threads gestartet, mit folgendem Ablauf:
  account1 schickt 50000 an account2
  account2 schickt 50000 an account1
  usw.
• am Ende werden die Resultate gesammelt (println während laufendem Prozess ist nicht gut, weil sonst alles durcheinander ist). und ausgegeben und in ein file benchmark_DATUM.txt geschrieben (im bin-Ordner).

Anleitung
https://integritee.atlassian.net/wiki/spaces/INTEGRITEE/pages/307036172/Run+Performance+Benchmarks

[brenzi]

Don't forget that we also want to bloat the state size, so we need an alternative mode where we create a new account with each transfer without reaping the old one (leave the ED)

[masapr]

I don't see a difference between creating many accounts in the beginning or continously creating more accounts.

[Niederb]

See #752 regarding the issues with blocking