Fix nonreliable method of validating shardawareness in e2e's

[zimnx]

Gocql doesn't expose any way of checking if shardaware ports are used successfully, hence I implemented simple driver within a test, able to send initial packet and reading to which shard connection was established. Connections are established from within the client Pod which better reflects how clients are connecting in real environments.

Requires:

• Add netcat to node-setup helper image scylla-operator-images#76

Fixes #1028

[tnozicka]

Can you elaborate on why

scylla-operator/vendor/github.com/gocql/gocql/scylla.go

Lines 784 to 796 in 027d4b3

	func (d *ScyllaShardAwareDialer) DialContext(ctx context.Context, network, addr string) (conn net.Conn, err error) {
	sourcePort := ScyllaGetSourcePort(ctx)
	if sourcePort == 0 {
	return d.Dialer.DialContext(ctx, network, addr)
	}
	dialerWithLocalAddr := d.Dialer
	dialerWithLocalAddr.LocalAddr, err = net.ResolveTCPAddr(network, fmt.Sprintf(":%d", sourcePort))
	if err != nil {
	return nil, err
	}
	return dialerWithLocalAddr.DialContext(ctx, network, addr)
	}

is not enough? (failing for 0, or failing to dial)

[zimnx]

is not enough? (failing for 0, or failing to dial)

You will dial successfully to shard aware port, but how do you know if you actually got the right shard without parsing the protocol? Gocql driver doesn't allow you to check it.
When there's NAT in between client and server, the only side effect is log message which cannot be validated.

Current approach used injected Dialer, but it can't determine if correct shard was chosen so it depended on particular gocql behavior - first connection is to any shard and the rest should use shard aware range.
But sometimes driver didn't try to establish connections to all shards within test timeout.
Why? I don't know, but relying on it isn't the best idea. We can do better by sending frames on our own and checking how shards are assigned.

[tnozicka]

/approve

[zimnx]

E2E's are failing because image is missing netcat. Once it's there I will retrigger them.
Local runs with image having it are passing.

[tnozicka]

thanks for the update
(lgtm)

/assign @rzetelskik

[rzetelskik]

/lgtm

[zimnx]

scylladb/scylla-operator-images#76 landed
/retest

[zimnx]

Bumped node-setup image to 0.0.3

[zimnx]

One minor change was made to remove -x from client Pod to prevent printing commands to stderr which is later validated and expected to be empty. There's no benefit of having this print. Expanded command is available in e2e dump.
@rzetelskik ptal

[rzetelskik]

/lgtm

[scylla-operator-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rzetelskik, tnozicka, zimnx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tnozicka,zimnx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment