Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Consolidate node-related fields into a struct #3232

Conversation

islamaliev
Copy link
Contributor

Relevant issue(s)

Resolves #3208

Description

All node-related fields are moved into a separate nodeState struct so now we don't need to maintain all slices and node indexes and it's also easier to reason about a node's state.

@islamaliev islamaliev requested review from AndrewSisley and a team November 11, 2024 15:08
@islamaliev islamaliev self-assigned this Nov 11, 2024
@islamaliev islamaliev added area/testing Related to any test or testing suite refactor This issue specific to or requires *notable* refactoring of existing codebases and components labels Nov 11, 2024
@islamaliev islamaliev added this to the DefraDB v0.15 milestone Nov 11, 2024
@islamaliev islamaliev changed the title Consolidate node-related fields into a struct refactor: Consolidate node-related fields into a struct Nov 11, 2024
Copy link

codecov bot commented Nov 11, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.51%. Comparing base (909c4af) to head (983cd4f).
Report is 1 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #3232      +/-   ##
===========================================
- Coverage    77.53%   77.51%   -0.02%     
===========================================
  Files          382      382              
  Lines        35263    35263              
===========================================
- Hits         27341    27334       -7     
- Misses        6296     6302       +6     
- Partials      1626     1627       +1     
Flag Coverage Δ
all-tests 77.51% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 9 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 909c4af...983cd4f. Read the comment docs.

Copy link
Contributor

@AndrewSisley AndrewSisley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a nice change, thanks Islam :)

Couple of optional suggestions from me.

// The node active in this test.
node *node.Node
// The node's client active in this test.
client clients.Client
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: If nodeState embeds *node.Node calling node.client the node will be a little nicer. E.g.:

type nodeState struct {
  *node.Node
  ...
}

suggestion: node.Node and clients.Client seem to basically be the same thing, it looks like it would be minimal hassle to get rid of clients.Client and just call nodeState.node.Peer.Connect(...).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: node.Node and clients.Client seem to basically be the same thing, it looks like it would be minimal hassle to get rid of clients.Client and just call nodeState.node.Peer.Connect(...).

You mean to get rid of nodeState's client and instead of for example:

node.client.GetCollections(s.ctx, client.CollectionFetchOptions{})

call

node.DB.GetCollections(s.ctx, client.CollectionFetchOptions{})

(assuming your first suggestion is applied).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah - my bad, when I made that suggestion I though node.Node implemented client.DB and you could have done node.GetCollections. Don't bother with that suggestion :)

Copy link
Member

@nasdf nasdf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nice improvement to the test framework!

Copy link
Member

@shahzadlone shahzadlone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it.

Comment on lines 754 to 778
if len(s.nodeConfigs) == 0 {
// If there are no explicit node configuration actions the node will be
// basic (i.e. no P2P stuff) and can be yielded now.
c, err := setupClient(s, node)
require.NoError(s.t, err)
s.nodes[nodeIndex] = c

eventState, err := newEventState(c.Events())
require.NoError(s.t, err)
s.nodeEvents[nodeIndex] = eventState
continue
}

// We need to make sure the node is configured with its old address, otherwise
// a new one may be selected and reconnection to it will fail.
var addresses []string
for _, addr := range s.nodeAddresses[nodeIndex].Addrs {
addresses = append(addresses, addr.String())
}

nodeOpts := s.nodeConfigs[nodeIndex]
nodeOpts = append(nodeOpts, net.WithListenAddresses(addresses...))

node.Peer, err = net.NewPeer(s.ctx, node.DB.Blockstore(), node.DB.Encstore(), node.DB.Events(), nodeOpts...)
require.NoError(s.t, err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: What happened to this logic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's all inside setupNode

@islamaliev islamaliev force-pushed the refactor/consolidate-node-fields-in-test branch from e24d2df to 2a010ba Compare November 13, 2024 16:04
Copy link
Collaborator

@fredcarle fredcarle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this is a nice change Islam. There are a few issues that I've spotted that caused the tests you pointed out to me to fail. You'll find todos in the relavent areas.

One thing I want to highlight is that this PR managed to make all ci test jobs use the go client regardless of the client selected by the matrix. It would be nice if we could add some kind of validation to prevent this form happening again (out of scope)

While troubleshooting, I've applied changes that fix the issues so if you want to use any or all of it feel free to do so: b3f4fbe#diff-31fde4ee4ab77f7265d4f25d4941c99e626d8b1969c9ff3667e1c6f0c7c4878f

I also noticed that even after fixing, some of the test were a little flaky due to Dial backoff. I'm not sure yet what to do about this but if it turns out to be a problem in the CI, I'll look deeper into it.

tests/integration/db.go Outdated Show resolved Hide resolved
tests/integration/db.go Outdated Show resolved Hide resolved
event: eventState,
p2p: newP2PState(),
dbPath: path,
netOpts: netOpts,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: netOpts is store on the struct but never reused. These options should be passed to setupNode on restart.

@islamaliev islamaliev force-pushed the refactor/consolidate-node-fields-in-test branch from 2a010ba to 8c953d4 Compare November 14, 2024 18:01
@islamaliev islamaliev requested a review from fredcarle November 14, 2024 20:29
Copy link
Collaborator

@fredcarle fredcarle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank for applying the changes.

@islamaliev islamaliev merged commit 198454b into sourcenetwork:develop Nov 14, 2024
42 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/testing Related to any test or testing suite refactor This issue specific to or requires *notable* refactoring of existing codebases and components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor test state to make working with nodes more convenient
5 participants