Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Test node pkg constructor via integration test suite #2641

Merged
merged 5 commits into from
May 22, 2024
Merged

test: Test node pkg constructor via integration test suite #2641

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9aa34b6
Fix dbOpts casing
AndrewSisley May 22, 2024
14bae07
Add defra store option
AndrewSisley May 22, 2024
9ac53ef
Rework int test framework to use node constr.
AndrewSisley May 22, 2024
9011a63
Simplify NewBadgerFileDB test func
AndrewSisley May 22, 2024
ae8bf9d
Simplify NewBadgerMemoryDB test func
AndrewSisley May 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cli/server_dump.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ func MakeServerDumpCmd() *cobra.Command {
storeOpts := []node.StoreOpt{
node.WithPath(cfg.GetString("datastore.badger.path")),
}
rootstore, err := node.NewStore(storeOpts...)
rootstore, err := node.NewStore(cmd.Context(), storeOpts...)
if err != nil {
return err
}
Expand Down
3 changes: 2 additions & 1 deletion node/node.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,8 @@ func NewNode(ctx context.Context, opts ...NodeOpt) (*Node, error) {
for _, opt := range opts {
opt(options)
}
rootstore, err := NewStore(options.storeOpts...)

rootstore, err := NewStore(ctx, options.storeOpts...)
if err != nil {
return nil, err
}
Expand Down
20 changes: 19 additions & 1 deletion node/store.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,18 @@
package node

import (
"context"

"github.com/sourcenetwork/defradb/datastore"
"github.com/sourcenetwork/defradb/datastore/badger/v4"
"github.com/sourcenetwork/defradb/datastore/memory"
)

// StoreOptions contains store configuration values.
type StoreOptions struct {
path string
inMemory bool
defraStore bool
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: inMemory and defraStore are also synonyms. Maybe we could do this instead.

type Datastore int

const (
  BadgerFile Datastore = iota
  BadgerMemory
  DefraMemory 
)

type StoreOption struct {
  ...
  store Datastore
}

This way by default it will be BadgerFile and if we add more option we won't need more struct fields and we remove the memory store confusion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for the thought

Copy link
Contributor Author

@AndrewSisley AndrewSisley May 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was tempted by that initially, but I felt defra-store was more analogous to badger-store, and thus (at least long-term) both inMemory and path would be applicable to both, is just that defra-store has no file based implementation yet (and so also atm, always assumes inMemory is true).

If/when we support more stores, defraStore should be replaced by an enum that contains other store implementations (defra, badger, turtle, etc, etc), all of which can respect path and inMemory (saves multiplying the number of db-types)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That leaves a point of confusion thought as turtle or others might not support in memory store but the option would still be available. So leaving it as is in the short term creates confusion between the badger in memory and defra in memory options and in the long term leaves in an option that might be irrelevant for some stores. I do have a preference to remove the confusion but I'm not blocking the PR for this.

valueLogFileSize int64
encryptionKey []byte
}
Expand All @@ -41,6 +45,16 @@ func WithInMemory(inMemory bool) StoreOpt {
}
}

// WithDefraStore sets the defra store flag.
//
// Setting this to true will result in the defra node being created with
// the a custom defra implementation of the rootstore instead of badger.
func WithDefraStore(defraStore bool) StoreOpt {
return func(o *StoreOptions) {
o.defraStore = defraStore
}
}

// WithPath sets the datastore path.
func WithPath(path string) StoreOpt {
return func(o *StoreOptions) {
Expand All @@ -63,12 +77,16 @@ func WithEncryptionKey(encryptionKey []byte) StoreOpt {
}

// NewStore returns a new store with the given options.
func NewStore(opts ...StoreOpt) (datastore.RootStore, error) {
func NewStore(ctx context.Context, opts ...StoreOpt) (datastore.RootStore, error) {
options := DefaultStoreOptions()
for _, opt := range opts {
opt(options)
}

if options.defraStore {
return memory.NewDatastore(ctx), nil
}

badgerOpts := badger.DefaultOptions
badgerOpts.InMemory = options.inMemory
badgerOpts.ValueLogFileSize = options.valueLogFileSize
Expand Down
2 changes: 1 addition & 1 deletion tests/bench/bench_util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ func newBenchStoreInfo(ctx context.Context, t testing.TB) (client.DB, error) {
case "memory":
db, err = testutils.NewBadgerMemoryDB(ctx)
case "badger":
db, _, err = testutils.NewBadgerFileDB(ctx, t)
db, err = testutils.NewBadgerFileDB(ctx, t)
default:
return nil, errors.New(fmt.Sprintf("invalid storage engine backend: %s", storage))
}
Expand Down
123 changes: 49 additions & 74 deletions tests/integration/db.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,8 @@ import (
"strconv"
"testing"

badger "github.com/sourcenetwork/badger/v4"

"github.com/sourcenetwork/defradb/client"
"github.com/sourcenetwork/defradb/crypto"
badgerds "github.com/sourcenetwork/defradb/datastore/badger/v4"
"github.com/sourcenetwork/defradb/datastore/memory"
"github.com/sourcenetwork/defradb/internal/db"
"github.com/sourcenetwork/defradb/node"
changeDetector "github.com/sourcenetwork/defradb/tests/change_detector"
Expand Down Expand Up @@ -74,93 +70,46 @@ func init() {
}
}

func NewBadgerMemoryDB(ctx context.Context, dbopts ...db.Option) (client.DB, error) {
opts := badgerds.Options{
Options: badger.DefaultOptions("").WithInMemory(true),
}
if encryptionKey != nil {
opts.Options.EncryptionKey = encryptionKey
opts.Options.IndexCacheSize = 100 << 20
}
rootstore, err := badgerds.NewDatastore("", &opts)
if err != nil {
return nil, err
}

acp, err := node.NewACP(ctx)
if err != nil {
return nil, err
}

db, err := db.NewDB(ctx, rootstore, acp, dbopts...)
if err != nil {
return nil, err
func NewBadgerMemoryDB(ctx context.Context) (client.DB, error) {
opts := []node.NodeOpt{
node.WithStoreOpts(node.WithInMemory(true)),
node.WithDatabaseOpts(db.WithUpdateEvents()),
}
return db, nil
}

func NewInMemoryDB(ctx context.Context, dbopts ...db.Option) (client.DB, error) {
acp, err := node.NewACP(ctx)
node, err := node.NewNode(ctx, opts...)
if err != nil {
return nil, err
}

db, err := db.NewDB(ctx, memory.NewDatastore(ctx), acp, dbopts...)
if err != nil {
return nil, err
}
return db, nil
return node.DB, err
}

func NewBadgerFileDB(ctx context.Context, t testing.TB, dbopts ...db.Option) (client.DB, string, error) {
var dbPath string
switch {
case databaseDir != "":
// restarting database
dbPath = databaseDir

case changeDetector.Enabled:
// change detector
dbPath = changeDetector.DatabaseDir(t)

default:
// default test case
dbPath = t.TempDir()
}

opts := &badgerds.Options{
Options: badger.DefaultOptions(dbPath),
}
if encryptionKey != nil {
opts.Options.EncryptionKey = encryptionKey
opts.Options.IndexCacheSize = 100 << 20
}
rootstore, err := badgerds.NewDatastore(dbPath, opts)
if err != nil {
return nil, "", err
}
func NewBadgerFileDB(ctx context.Context, t testing.TB) (client.DB, error) {
path := t.TempDir()

acp, err := node.NewACP(ctx, node.WithACPPath(dbPath))
if err != nil {
return nil, "", err
opts := []node.NodeOpt{
node.WithStoreOpts(node.WithPath(path)),
}

db, err := db.NewDB(ctx, rootstore, acp, dbopts...)
node, err := node.NewNode(ctx, opts...)
if err != nil {
return nil, "", err
return nil, err
}

return db, dbPath, err
return node.DB, err
}

// setupDatabase returns the database implementation for the current
// testing state. The database type on the test state is used to
// select the datastore implementation to use.
func setupDatabase(s *state) (impl client.DB, path string, err error) {
dbopts := []db.Option{
func setupDatabase(s *state) (client.DB, string, error) {
dbOpts := []db.Option{
db.WithUpdateEvents(),
db.WithLensPoolSize(lensPoolSize),
}
storeOpts := []node.StoreOpt{}
acpOpts := []node.ACPOpt{}
opts := []node.NodeOpt{}

if badgerEncryption && encryptionKey == nil {
key, err := crypto.GenerateAES256()
Expand All @@ -170,22 +119,48 @@ func setupDatabase(s *state) (impl client.DB, path string, err error) {
encryptionKey = key
}

if encryptionKey != nil {
storeOpts = append(storeOpts, node.WithEncryptionKey(encryptionKey))
}

var path string
switch s.dbt {
case badgerIMType:
impl, err = NewBadgerMemoryDB(s.ctx, dbopts...)
storeOpts = append(storeOpts, node.WithInMemory(true))

case badgerFileType:
impl, path, err = NewBadgerFileDB(s.ctx, s.t, dbopts...)
switch {
case databaseDir != "":
// restarting database
path = databaseDir

case changeDetector.Enabled:
// change detector
path = changeDetector.DatabaseDir(s.t)

default:
// default test case
path = s.t.TempDir()
}

storeOpts = append(storeOpts, node.WithPath(path))
acpOpts = append(acpOpts, node.WithACPPath(path))

case defraIMType:
impl, err = NewInMemoryDB(s.ctx, dbopts...)
storeOpts = append(storeOpts, node.WithDefraStore(true))

default:
err = fmt.Errorf("invalid database type: %v", s.dbt)
return nil, "", fmt.Errorf("invalid database type: %v", s.dbt)
}

opts = append(opts, node.WithDatabaseOpts(dbOpts...))
opts = append(opts, node.WithStoreOpts(storeOpts...))
opts = append(opts, node.WithACPOpts(acpOpts...))

node, err := node.NewNode(s.ctx, opts...)
if err != nil {
return nil, "", err
}
return

return node.DB, path, nil
}
3 changes: 1 addition & 2 deletions tests/integration/events/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ import (
"github.com/stretchr/testify/require"

"github.com/sourcenetwork/defradb/client"
"github.com/sourcenetwork/defradb/internal/db"
testUtils "github.com/sourcenetwork/defradb/tests/integration"
)

Expand Down Expand Up @@ -70,7 +69,7 @@ func ExecuteRequestTestCase(
) {
ctx := context.Background()

db, err := testUtils.NewBadgerMemoryDB(ctx, db.WithUpdateEvents())
db, err := testUtils.NewBadgerMemoryDB(ctx)
require.NoError(t, err)

_, err = db.AddSchema(ctx, schema)
Expand Down
3 changes: 1 addition & 2 deletions tests/integration/net/order/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ import (

"github.com/sourcenetwork/defradb/client"
"github.com/sourcenetwork/defradb/errors"
coreDB "github.com/sourcenetwork/defradb/internal/db"
"github.com/sourcenetwork/defradb/net"
netutils "github.com/sourcenetwork/defradb/net/utils"
testutils "github.com/sourcenetwork/defradb/tests/integration"
Expand Down Expand Up @@ -81,7 +80,7 @@ func setupDefraNode(
ctx := context.Background()

log.InfoContext(ctx, "Building new memory store")
db, err := testutils.NewBadgerMemoryDB(ctx, coreDB.WithUpdateEvents())
db, err := testutils.NewBadgerMemoryDB(ctx)
if err != nil {
return nil, nil, err
}
Expand Down
Loading