explicitly close socat socket

[git-phu]

What

closes #11621

Currently the way we handle sending messages from the source pod to the orchestrator pod is:

• The orchestrator loops forever trying to read messages from the source until it detects that the source pod has terminated
• The source pod uses socat to transfer a stream of data to the orchestrator.
  • Once the stream's EOF is reached, socat shuts down its write stream.
  • After a 0.5 second timeout, socat closes the connection to the orchestrator as well and the source pod terminates

We can see this in a sample of the socat logs.

2022/05/05 02:03:29 socat[9] I transferred 8192 bytes from 0 to 5
2022/05/05 02:03:29 socat[9] I transferred 5306 bytes from 0 to 5
2022/05/05 02:03:29 socat[9] N socket 1 (fd 0) is at EOF
2022/05/05 02:03:29 socat[9] I shutdown(5, 1)
2022/05/05 02:03:30 socat[9] I poll timed out (no data within 0.500000 seconds)
2022/05/05 02:03:30 socat[9] I shutdown(5, 2)
2022/05/05 02:03:30 socat[9] N exiting with status 0

The problem with this approach is that we have seen cases where it appears that the source pod has already terminated before the orchestrator is able to fully process all the incoming data. This leaves the network socket in a bad state and the orchestrator pod simply hangs. #11621 (comment) for more details on how this might happen.

A more reliable way to handle this interaction should be:

• The orchestrator loops forever trying to read messages from the source
• The source pod uses socat to transfer a stream of data to the orchestrator.
  • Once the stream's EOF is reached, socat shuts down its write stream.
• Once the orchestrator receives the EOF from socat, the orchestrator should initiate shutting down the network socket
  • after socat receives the signal to shut down the read stream from the orchestrator, the network socket is closed and the source pod returns.

This new flow should avoid the problems arising from the source initiating the shutdown. This is also how socat is intended to work.
http://www.dest-unreach.org/socat/doc/socat.html

When one of the streams effectively reaches EOF, the closing phase begins. Socat transfers the EOF condition to the other stream, i.e. tries to shutdown only its write stream, giving it a chance to terminate gracefully. For a defined time socat continues to transfer data in the other direction, but then closes all remaining channels and terminates.

How

• Change DefaultReplicatonWorker#getReplicationRunnable so that when there are no more messages from the source, the orchestrator initiate closing the network connection.
• Increase the timeout on socat to 60 seconds (from the default 0.5 seconds) to give the orchestrator time to receive all messages and then initiate shutdown of the network connection.

With this change the socat logs now look like

2022/05/05 16:09:02 socat[8] N reading from and writing to stdio
2022/05/05 16:09:02 socat[8] N opening connection to AF=2 172.25.37.3:9877
2022/05/05 16:09:02 socat[8] N successfully connected from local address AF=2 172.25.32.134:38950
2022/05/05 16:09:02 socat[8] N starting data transfer loop with FDs [0,1] and [5,5]
2022/05/05 16:14:17 socat[8] N socket 1 (fd 0) is at EOF
2022/05/05 16:14:19 socat[8] N socket 2 (fd 5) is at EOF
2022/05/05 16:14:19 socat[8] N exiting with status 0

(note that socket 2 also has an EOF now)

[supertopher]

looks great Peter

[supertopher]

I love this exception. it sounds like Godzilla is coming from the water and cannot be contained

[git-phu]

did a final verification in a dev env that everything works as expected (discovers, syncs, etc.)