Split by timings for parallel run #230

trangtran-enya · 2022-07-02T12:24:13Z

📋 Add associated issues, tickets, docs URL here.

Overview

Currently, we already execute testcases in parallel. However, testcases are divided based on filename instead of execution time.
Issue 139

Changes

Add mocha-junit-reporter to collect timing data
Turn off hardhat-gas-reporter as it conflicts with mocha-junit-reporter. Need discussion?
Turn on split by timings
split the time it takes for all tests to run in half

Testing

integration tests

InoMurko · 2022-07-06T09:47:51Z

ops/up_local.sh

+  docker-compose build --parallel --no-cache -- builder l2geth l1_chain
+  docker-compose build --parallel --no-cache -- deployer dtl batch_submitter relayer integration_tests
+  docker-compose build --no-cache -- boba_message-relayer-fast
+  docker-compose build --no-cache -- gas_oracle
+  docker-compose build --no-cache -- boba_deployer
+  docker-compose build --no-cache -- fraud-detector
+  docker-compose build --no-cache -- monitor
+  docker-compose build --no-cache -- verifier
+  docker-compose build --no-cache -- replica


how come caches are disabled?

Testcases in the beginning failed, so I tried to run with image built from latest code.

In the beginning, this worked. All testcases passed. Now, some testcases using cross-chain message failed.

InoMurko · 2022-07-06T09:48:20Z

integration-tests/hardhat.config.ts

@@ -5,7 +5,8 @@ import '@nomiclabs/hardhat-ethers'
 import '@nomiclabs/hardhat-waffle'
 import 'hardhat-gas-reporter'

-const enableGasReport = !!process.env.ENABLE_GAS_REPORT
+// const enableGasReport = !!process.env.ENABLE_GAS_REPORT
+const enableGasReport = false


why doesn't this work?

Looks like the result from GasReport overwrite Timing result, which is used for future run.

Have you tried things like:
mochajs/mocha#1360
https://www.npmjs.com/package/mocha-multi-reporters
?

No, not yet. Let me try. Thanks Ino

The GasReporter is this one, configured through Hard-Hat. Maybe we will meet some problem. But I think I can figure out.

codecov-commenter · 2022-07-10T02:14:11Z

Codecov Report

Merging #230 (2085687) into develop (9a5000e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #230   +/-   ##
========================================
  Coverage    47.22%   47.22%           
========================================
  Files           99       99           
  Lines         3867     3867           
  Branches       762      762           
========================================
  Hits          1826     1826           
  Misses        2041     2041

Flag	Coverage Δ
boba-contracts	`20.39% <ø> (ø)`
contracts	`77.71% <ø> (ø)`
core-utils	`86.80% <ø> (ø)`
data-transport-layer	`31.17% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a5000e...2085687. Read the comment docs.

souradeep-das

lgtm

wsdt · 2022-07-19T14:52:06Z

integration-tests/test/boba-fee-payment.spec.ts

@@ -1044,9 +1044,9 @@ describe('Boba Fee Payment Integration Tests', async () => {
        bobaBalance
      )
      await transferBackTx.wait()
-    })
+    }).retries(3)


why do we have a retry here? Just retry three times until we don't have enough Boba tokens anymore?

flaky test or flaky infra!

wsdt

looks great! 🔥

integration-tests/test/boba-api.spec.ts

…sm 5

* try split by timings * increase retries * try to rebuild image * add mocha options * update folder * expose docker volumes * fix syntax error * update config * turn off gas report * turn off gas report * build no cache * default time for missing cases * try to reduce number of testcases * try to reduce number of testcases * change testcase path * Rewrite file path for next run * Adjust file path * Fix typo * Fix result * Fix command * Run full testcases * Increase parallel * try with existing images * try split by timings * try to rebuild image * add mocha options * update folder * expose docker volumes * try to reduce number of testcases * try to reduce number of testcases * change testcase path * Rewrite file path for next run * try mocha multi reporters * config gas reporter through mocha * try to remove hardhat reporter * switch to eth-gas-reporter * update l1 url for eth-gas-reporter * fix contracts path * Force console eth-gas-report * remove output to file * run whole testcase with both gas reporter and timing reporter * run all tests * retry * remove no cache, enable gas report, parallel docker builds, parallelism 5 * fix tests * reduce parallellism * update yarn and add dependency ignores * testing the same code twice * increase timeout for turing tests * shuf test files * harden tests * prevent parallel execution * response logs Co-authored-by: Ino Murko <[email protected]> Co-authored-by: Ino Murko <[email protected]> (cherry picked from commit dc4c6d9)

* Add BOBA API README (#209) (cherry picked from commit 5da8638) * Add new API (#196) (cherry picked from commit 5621463) * boba support for faucet (#218) * boba support for faucet * withdraw boba * Update deploy.sh Co-authored-by: undefined <undefined> (cherry picked from commit a37780f) * Feature (Gateway): Ve-Boba Lock for Vote page. (#222) * add ve endpoints * changes: - fix for avoiding the GA call on local env. - create new page for lock for vote. - Area Chart as separate common component along with other components. * changes: - created separate component for createLock - Populating the actual value of balance and connect to BOBA options. - prepare the styles for lock records. * chagnes: - Date picker on the create Lock screen - reading the value of radio button in datepicker. - redux setup for the lock for vote page - network service function schema for lock page. * changes: - removed fetchrecords from interval call - calling fetch records for lock on page and creations of new records. - tested and populated the records on lock screen - create lock to be fixed * changes- manage lock modals changes. * changes: - create lock fixes - extend time of lock - increase lock amount fix - representation of ration and voting power fixes - manage boba locks. - static data on chart representation. - cleanup * changes: - create lock fixes - extend time of lock - increase lock amount fix - representation of ration and voting power fixes - manage boba locks. - static data on chart representation. * changes: removed-address-rinkeby * remove-lock-link-from-navigation * change: - fix: pager on records listing - fix: disabled button when lock is creating - fix: disabled button when extend and increase already trigger. * - comment cleanup - added ENV var to show lock in option. * Change: 1. fix for the constants to separate file 2. fix for the chart data to 3,9,12 3. fix added env details in readme 3. fix copyright to bobanetwork * removed unused code. * removed empty file along with copyright change * Add contract address and import package * change: 1. fetchrecords on completion of lock, increase, extends time tx. 2. fix button disabling on value getting empty 3. fix for the value to empty with placeholder as '0' * rinkeby address for boba contracts * change: - fix typo in ratio. - fix in month => this month - voting power label update in increase amount. * fix: vpower calculation on increase lock * added suggested change. Co-authored-by: souradeep-das <[email protected]> Co-authored-by: cby3149 <[email protected]> (cherry picked from commit 7228802) * update sentry sdk & Zendesk Integration. (#254) 1. Update Sentry SDK. 2. ZenDesk Integration on Gateway. (cherry picked from commit 6140e5e) * updated copyrights to bobanetwork (#252) updated copyrights to bobanetwork (cherry picked from commit aa06b5f) * Update boba node for the community and remove release tag (#256) (cherry picked from commit d9e6a4f) * Add the link to community-built tools for Turing (#190) * Add the link to community-built tools for Turing * Update turing.md (cherry picked from commit bd3dc5a) * add veBoba contracts (#257) * add veBoba contracts * update deps (cherry picked from commit 9a41eaf) * Split by timings for parallel run (#230) * try split by timings * increase retries * try to rebuild image * add mocha options * update folder * expose docker volumes * fix syntax error * update config * turn off gas report * turn off gas report * build no cache * default time for missing cases * try to reduce number of testcases * try to reduce number of testcases * change testcase path * Rewrite file path for next run * Adjust file path * Fix typo * Fix result * Fix command * Run full testcases * Increase parallel * try with existing images * try split by timings * try to rebuild image * add mocha options * update folder * expose docker volumes * try to reduce number of testcases * try to reduce number of testcases * change testcase path * Rewrite file path for next run * try mocha multi reporters * config gas reporter through mocha * try to remove hardhat reporter * switch to eth-gas-reporter * update l1 url for eth-gas-reporter * fix contracts path * Force console eth-gas-report * remove output to file * run whole testcase with both gas reporter and timing reporter * run all tests * retry * remove no cache, enable gas report, parallel docker builds, parallelism 5 * fix tests * reduce parallellism * update yarn and add dependency ignores * testing the same code twice * increase timeout for turing tests * shuf test files * harden tests * prevent parallel execution * response logs Co-authored-by: Ino Murko <[email protected]> Co-authored-by: Ino Murko <[email protected]> (cherry picked from commit dc4c6d9) * Symptomatic Fix - Turing Examples Tests (#263) * symptomatic fix * lending fix * Updated config.yml * example tests spec refactoring * Updated config.yml * Updated config.yml (cherry picked from commit fd7968f) * Wsdt/erigon (#258) * erigon * submodule * Update sync-submodules.yml * Update sync-submodules.yml * Update sync-submodules.yml * Updated Erigon submodule * Update sync-submodules.yml * Update sync-submodules.yml * Updated Erigon submodule * Update sync-submodules.yml * Update sync-submodules.yml trigger * submodule * Update sync-submodules.yml * Update sync-submodules.yml * Update sync-submodules.yml * Update sync-submodules.yml * Update sync-submodules.yml * Update sync-submodules.yml * Updated Erigon submodule * Update sync-submodules.yml * Update sync-submodules.yml * Updated Erigon submodule * Update sync-submodules.yml * Update sync-submodules.yml * Updated Erigon submodule * Update sync-submodules.yml * Update sync-submodules.yml * Update sync-submodules.yml * Update sync-submodules.yml * Update sync-submodules.yml Co-authored-by: Git bot <[email protected]> Co-authored-by: Github bot <> (cherry picked from commit dc23297) * Wsdt/turing twitter (#238) * init * added tests, first draft * twitter - debugging aws, .. * twitter turing - seems to work * twitter api integration * clean up * twitter turing works - first draft * further twitter data preparation * nft -> faucet, bytes debugging, twitter gates, etc. * cooldown adapted to faucet logic * clarify boba Bubble auth flow * converted to regular claim * ui integration & testing * cleanup * minor clean * Wsdt/turing twitter meta (#157) * oz way * meta works * withdraw * ui * debug * clean * clean git * some fixes * finally works * clean Co-authored-by: Boyuan-Chen <[email protected]> * update wording * Wsdt/turing twitter meta (#161) * oz way * meta works * withdraw * ui * debug * clean * clean git * some fixes * finally works * clean * production ready, fresh deployments, etc. Co-authored-by: Boyuan-Chen <[email protected]> * move UI to token page * adding constraints to view * grammar * add proper error messages * Âease twitter constraints * remove monsters.js faucet code * test fix * Add a function for loading contracts (#170) * Wsdt/turing twitter pay (#171) * first draft * fe integration, registration done * fe send funds * Twitter Pay: First draft * collision * additional collision protection * mv to core * Delete build.toml * rm unused function * Turing TwitterPay tests * Update boba-contracts-unit.yml * Update boba-contracts-unit.yml * Update boba-contracts-unit.yml * Update boba-contracts-unit.yml * Update boba-contracts-unit.yml * Update boba-contracts-unit.yml * Update boba-contracts-unit.yml * cleanup * python clean * merge * TwitterPay added to deployer * adapted TwitterPay test (local, py, fixed pre-deploy, ..) * undo deployer add, mv to boba_examples * Update boba-contracts-unit.yml * update yarn lock Co-authored-by: CAPtheorem <[email protected]> Co-authored-by: undefined <undefined> Co-authored-by: Ino Murko <[email protected]> * rm twitterpay from gateway * Update README.md * PR review fixes: clean up Co-authored-by: CAPtheorem <[email protected]> Co-authored-by: Boyuan-Chen <[email protected]> Co-authored-by: Ino Murko <[email protected]> (cherry picked from commit 15fcfe1) * Adds reference-optimistic-geth submodule (#274) * submodule * adds new pr reviewers * Update sync-submodules.yml (cherry picked from commit bffbb93) * auth faucet done (#282) (cherry picked from commit e8f4286) * Update sync-submodules.yml (#281) Old branch replaced (cherry picked from commit 5aae0c2) * feat: modify community DAO to work with veBoba (#270) * feat: modify dao to work with veNFTs * add merge ve comment * add tests * update depl script to custom input ve addr * remove unused params and update thresholds (cherry picked from commit c80667c) * ve: update to proxy compatible contracts (#287) * update to proxy compatible contracts * update tests (cherry picked from commit 2ea7f7a) * Updated submodules (#289) Co-authored-by: Github bot <> (cherry picked from commit 85b908d) Co-authored-by: Boyuan-Chen <[email protected]> Co-authored-by: Riedl Kevin, Bsc <[email protected]> Co-authored-by: Sahil K <[email protected]> Co-authored-by: 0xEstarriol <[email protected]> Co-authored-by: Souradeep Das <[email protected]> Co-authored-by: trangtran-enya <[email protected]> Co-authored-by: Riedl Kevin, Bsc <[email protected]> Co-authored-by: bobabot-enya-ai <[email protected]>

trangtran-enya linked an issue Jul 2, 2022 that may be closed by this pull request

unreliable integration tests #193

Closed

github-actions bot added 2-reviewers M-integration M-ops labels Jul 2, 2022

InoMurko reviewed Jul 6, 2022

View reviewed changes

InoMurko force-pushed the 193-unreliable-integration-tests branch from 98bb7da to 6307e60 Compare July 6, 2022 09:58

InoMurko force-pushed the 193-unreliable-integration-tests branch from 2085687 to 33309e2 Compare July 19, 2022 09:56

InoMurko marked this pull request as ready for review July 19, 2022 11:38

InoMurko requested review from InoMurko, boyuan-chen, souradeep-das, mmontour1306 and wsdt July 19, 2022 11:38

souradeep-das approved these changes Jul 19, 2022

View reviewed changes

InoMurko force-pushed the 193-unreliable-integration-tests branch 3 times, most recently from b2632a4 to 1128ea8 Compare July 19, 2022 14:25

wsdt reviewed Jul 19, 2022

View reviewed changes

wsdt approved these changes Jul 19, 2022

View reviewed changes

boyuan-chen reviewed Jul 19, 2022

View reviewed changes

integration-tests/test/boba-api.spec.ts Show resolved Hide resolved

InoMurko added bug Something isn't working enhancement New feature or request labels Jul 19, 2022

trangtran-enya added 6 commits July 21, 2022 09:09

try split by timings

9dfcde5

increase retries

ad0fcbe

try to rebuild image

3f64a70

add mocha options

a3f24fa

update folder

562b007

expose docker volumes

5edf5d5

trangtran-enya and others added 23 commits July 21, 2022 09:09

config gas reporter through mocha

3ef1047

try to remove hardhat reporter

4d38c50

switch to eth-gas-reporter

391589f

update l1 url for eth-gas-reporter

00a82e7

fix contracts path

338471c

Force console eth-gas-report

7caca10

remove output to file

b8f70c0

run whole testcase with both gas reporter and timing reporter

4632731

run all tests

1bb0cb5

retry

a4891e6

remove no cache, enable gas report, parallel docker builds, paralleli…

ea3bfeb

…sm 5

fix tests

0f9fe2f

reduce parallellism

e1a6eb1

update yarn and add dependency ignores

8e3acf0

testing the same code twice

eb9d726

increase timeout for turing tests

67d90aa

shuf test files

c4444a9

harden tests

3655b7c

prevent parallel execution

1e10c8e

phomedir

f33194d

parallel flag

583c414

proper split

bbbedbd

proper split

4537904

InoMurko force-pushed the 193-unreliable-integration-tests branch from 66b7d42 to 4537904 Compare July 21, 2022 09:32

response logs

e23d0fd

InoMurko approved these changes Jul 21, 2022

View reviewed changes

InoMurko merged commit dc4c6d9 into develop Jul 21, 2022

InoMurko deleted the 193-unreliable-integration-tests branch July 21, 2022 11:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split by timings for parallel run #230

Split by timings for parallel run #230

trangtran-enya commented Jul 2, 2022 •

edited by InoMurko

Loading

InoMurko Jul 6, 2022

trangtran-enya Jul 6, 2022

trangtran-enya Jul 6, 2022

InoMurko Jul 6, 2022

trangtran-enya Jul 6, 2022

InoMurko Jul 6, 2022

trangtran-enya Jul 6, 2022 •

edited

Loading

codecov-commenter commented Jul 10, 2022 •

edited

Loading

souradeep-das left a comment

wsdt Jul 19, 2022

InoMurko Jul 21, 2022

wsdt left a comment

Split by timings for parallel run #230

Split by timings for parallel run #230

Conversation

trangtran-enya commented Jul 2, 2022 • edited by InoMurko Loading

Overview

Changes

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trangtran-enya Jul 6, 2022 • edited Loading

Choose a reason for hiding this comment

codecov-commenter commented Jul 10, 2022 • edited Loading

Codecov Report

souradeep-das left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wsdt left a comment

Choose a reason for hiding this comment

trangtran-enya commented Jul 2, 2022 •

edited by InoMurko

Loading

trangtran-enya Jul 6, 2022 •

edited

Loading

codecov-commenter commented Jul 10, 2022 •

edited

Loading