Dynamic Chain Test Language Package With YAML Parser #681

rauljordan · 2018-10-18T22:27:32Z

Resolves #420

Description

This PR incorporates the YAML format for fork-choice and chain tests as outlined in the Ethereum 2.0 spec and the markdown file here by @djrtwo. This PR does not yet implement the simulated backend that will act as the runner for the tests but simply includes a parser for supplied yaml files to extract test case parameters.

Requirements for Merge

Basic Refactoring to Allow for Simulated Backend
Basic Simulated Backend Definition
Refactor Config to Allow for Custom Parameters for Simulated Runs
YAML Parser From File
Include a Basic Chaintest File
README on Using the Chaintest Runner

Important

This PR does not actually run the full tests but instead just sets up the necessary framework and testing format required to do so. The test implementation will be part of #718.

codecov · 2018-10-20T05:29:39Z

Codecov Report

Merging #681 into master will decrease coverage by 0.05%.
The diff coverage is 41.86%.

@@            Coverage Diff             @@
##           master     #681      +/-   ##
==========================================
- Coverage    75.2%   75.14%   -0.06%     
==========================================
  Files          60       59       -1     
  Lines        3815     3834      +19     
==========================================
+ Hits         2869     2881      +12     
- Misses        684      692       +8     
+ Partials      262      261       -1

rauljordan · 2018-11-06T21:12:11Z

beacon-chain/casper/sharding_test.go

@@ -232,13 +232,13 @@ func TestValidatorsBySlotShardSmallValidatorSet(t *testing.T) {
 	for i := 0; i < len(shardAndCommitteeArray); i++ {
 		shardAndCommittees := shardAndCommitteeArray[i].ArrayShardAndCommittee
 		if len(shardAndCommittees) != 1 {
-			t.Fatalf("Expected %d committee per slot: got %d", params.GetConfig().MinCommiteeSize, 1)
+			t.Fatalf("Expected %d committee per slot: got %d", params.GetConfig().MinCommitteeSize, 1)


There were a few typos with Committee that needed to be fixed

prestonvanloon · 2018-11-07T11:39:36Z

.codecov.yml

+    project: 
+      default:
+        target: auto
+        threshold: 1%


Can we include this in a different PR?

nevermind, it looks like its already in master!

prestonvanloon · 2018-11-07T11:44:56Z

beacon-chain/chaintest/backend/BUILD.bazel

+        "test_format.go",
+    ],
+    importpath = "github.com/prysmaticlabs/prysm/beacon-chain/chaintest/backend",
+    visibility = ["//beacon-chain:__subpackages__"],


Please limit to //beacon-chain/chaintest:__subpackages__

prestonvanloon · 2018-11-07T11:46:14Z

beacon-chain/chaintest/backend/setup_db.go

+
+// setupDB instantiates and returns a simulated backend BeaconDB instance.
+func setupDB() (*db.BeaconDB, error) {
+	path := path.Join(os.TempDir(), "/simulateddb")


Will this cause issues if you are running tests in parallel?
Maybe a random named directory would be ideal.

prestonvanloon · 2018-11-07T11:47:14Z

beacon-chain/chaintest/backend/simulated_backend.go

@@ -0,0 +1,82 @@
+// Package backend contains utilities for simulating an entire
+// ETH 2.0 beacon chain in-memory for e2e tests and benchmarking


Is this true? The database writes to disk so its not fully in-memory. Was that the intention?

prestonvanloon · 2018-11-07T11:49:51Z

beacon-chain/chaintest/backend/simulated_backend.go

+			RandaoCommitment:  randaoReveal[:],
+		}
+	}
+	// TODO(#718): Next step is to update and save the blocks specified


Please add to your PR description that this PR does not actually run the tests and that the test implementation is part of #718

rauljordan · 2018-11-07T12:48:35Z

PTAL @prestonvanloon

prestonvanloon · 2018-11-07T13:09:33Z

beacon-chain/chaintest/backend/setup_db.go

@@ -25,7 +27,7 @@ func teardownDB(db *db.BeaconDB) {
 	if err := db.Close(); err != nil {
 		log.Fatalf("failed to close database: %v", err)
 	}
-	path := path.Join(os.TempDir(), "/simulateddb")
+	path := path.Join(os.TempDir(), fmt.Sprintf("/%d", rand.Int()))


This won't clean up the same random number will it?

I think you need to keep track of this number somehow, maybe this path could be db.DatabasePath

Yeah agreed, maybe try to keep track of the number of simulated backend instances running and assign your path that way.

prestonvanloon · 2018-11-07T13:10:16Z

beacon-chain/chaintest/backend/setup_db.go

 	"github.com/prysmaticlabs/prysm/beacon-chain/db"
 )

 // setupDB instantiates and returns a simulated backend BeaconDB instance.
 func setupDB() (*db.BeaconDB, error) {
-	path := path.Join(os.TempDir(), "/simulateddb")
+	path := path.Join(os.TempDir(), fmt.Sprintf("/%d", rand.Int()))


Please prefix with something human readable: prysmsimulateddb-%d or similar

terencechain

lgtm

terencechain · 2018-11-07T13:38:14Z

beacon-chain/chaintest/backend/setup_db.go

@@ -25,7 +27,7 @@ func teardownDB(db *db.BeaconDB) {
 	if err := db.Close(); err != nil {
 		log.Fatalf("failed to close database: %v", err)
 	}
-	path := path.Join(os.TempDir(), "/simulateddb")
+	path := path.Join(os.TempDir(), fmt.Sprintf("/%d", rand.Int()))


chaintest specifier in subpackage

6d9dbfe

rauljordan added In Progress Priority: High High priority item labels Oct 18, 2018

rauljordan added this to the Ruby - v0.1 milestone Oct 18, 2018

rauljordan self-assigned this Oct 18, 2018

rauljordan changed the title ~~Dynamic Chain Test Package With YAML Parser~~ Dynamic Chain Test Language Package With YAML Parser Oct 18, 2018

Merge branch 'master' into test-lang

82a6af8

rauljordan added 14 commits October 22, 2018 14:40

Merge branch 'master' into test-lang

8906fd1

Merge branch 'master' into test-lang

33e94c4

include basic simulated backend

f1a0178

added build files

2e0482b

Merge branch 'test-lang' of github.com:rauljordan/prysm into test-lang

f23251d

improve chain test format definitions

2699056

successful parsing of yaml chaintest

59f5530

move db test utils to internal

e0f4941

internal refactor

88e859d

chain test lang

59a3f4b

Merge branch 'master' into test-lang

074e9a9

fix import cycle

fbcd73f

Merge branch 'test-lang' of github.com:rauljordan/prysm into test-lang

72a7a60

test lang runner with end time benchmark

c28dd16

rauljordan removed the In Progress label Nov 1, 2018

rauljordan added 5 commits November 2, 2018 11:06

gazelle

0b5d2bd

fix undef

446b3a6

fix

7b2f748

handle err

50f8456

gazelle

449f307

rauljordan mentioned this pull request Nov 2, 2018

Run Fork Choice Tests Using the YAML Test Parser and Simulated Backend #718

Closed

rauljordan added 4 commits November 6, 2018 15:13

gazelle

f56777d

increase cov

a8767bd

add threshold

c360f09

Merge branch 'master' into test-lang

c5f194a

rauljordan requested review from prestonvanloon, rawfalafel, terencechain and nisdas and removed request for rawfalafel, terencechain and nisdas November 6, 2018 20:55

rauljordan commented Nov 6, 2018

View reviewed changes

rauljordan added 2 commits November 6, 2018 15:19

Merge branch 'master' into test-lang

d9c17da

Merge branch 'master' into test-lang

ccd0b2b

rauljordan added the Ready For Review label Nov 7, 2018

prestonvanloon reviewed Nov 7, 2018

View reviewed changes

rauljordan added 3 commits November 7, 2018 07:48

addr commnets

59b6fce

Merge branch 'test-lang' of github.com:rauljordan/prysm into test-lang

2087dd2

Merge branch 'master' into test-lang

17b5d14

prestonvanloon reviewed Nov 7, 2018

View reviewed changes

terencechain approved these changes Nov 7, 2018

View reviewed changes

rauljordan added 3 commits November 7, 2018 13:32

fix dep

91decbd

Merge branch 'test-lang' of github.com:rauljordan/prysm into test-lang

a4633ed

fix lint

4a8b8b4

rauljordan merged commit 6476fb5 into prysmaticlabs:master Nov 7, 2018

terencechain mentioned this pull request Nov 10, 2018

Run Shuffle Tests Using the YAML Test Parser and Simulated Backend #738

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic Chain Test Language Package With YAML Parser #681

Dynamic Chain Test Language Package With YAML Parser #681

rauljordan commented Oct 18, 2018 •

edited

Loading

codecov bot commented Oct 20, 2018 •

edited

Loading

rauljordan Nov 6, 2018

prestonvanloon Nov 7, 2018

prestonvanloon Nov 7, 2018

prestonvanloon Nov 7, 2018

prestonvanloon Nov 7, 2018

prestonvanloon Nov 7, 2018

prestonvanloon Nov 7, 2018

rauljordan commented Nov 7, 2018

prestonvanloon Nov 7, 2018

terencechain Nov 7, 2018

nisdas Nov 7, 2018

prestonvanloon Nov 7, 2018

nisdas Nov 7, 2018

terencechain left a comment

terencechain Nov 7, 2018

		@@ -0,0 +1,82 @@
		// Package backend contains utilities for simulating an entire
		// ETH 2.0 beacon chain in-memory for e2e tests and benchmarking

Dynamic Chain Test Language Package With YAML Parser #681

Dynamic Chain Test Language Package With YAML Parser #681

Conversation

rauljordan commented Oct 18, 2018 • edited Loading

Description

Requirements for Merge

Important

codecov bot commented Oct 20, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rauljordan commented Nov 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terencechain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rauljordan commented Oct 18, 2018 •

edited

Loading

codecov bot commented Oct 20, 2018 •

edited

Loading