Help debug: invalid_argument: protocol error: incomplete envelope: unexpected EOF

[D4ryl00]

I have a client and a server using Golang 1.21 in a Docker container on the same server. Both are using connect-go 1.16.2 with the connect protocol.
The client calls a RPC method that returns a stream by the server.
Randomly, when the server sends data, the client gets invalid_argument: protocol error: incomplete envelope: unexpected EOF and the server receives context canceled.
Do you have any idea to what I can do to have more information to figure out what happens?

If you are curious:

• the server method: https://github.com/D4ryl00/dsocial/blob/feat/expo-push-notif/tools/indexer-service/api.go#L34-L49
• the client: https://github.com/D4ryl00/dsocial/blob/feat/expo-push-notif/tools/notification-service/service.go#L84-L114

Thanks

[emcfarlane]

Hey @D4ryl00 , will need more information to help debug your setup but I would guess the endpoint is not being correctly routed to the handler. Could you check that each method is invoked correctly by adding some logging? The unexpected EOF sounds seems like an empty response from the server.

[D4ryl00]

Hi @emcfarlane !

I would guess the endpoint is not being correctly routed to the handler

I think if it had been the case, I would have the error every time, no?

Could you check that each method is invoked correctly by adding some logging?

Yes I've already put a log at the beginning of the method, after stream.Receive() and at the end. I have a log that print stream.Err(). The outcome is that I enter every time in that method, but randomly, after the stream.Receive() loop, I have the error ...unexpected EOF when the server try to send something. I tried to identify a pattern to reproduce it, but it seems to be random.

The unexpected EOF sounds seems like an empty response from the server.

Interresting. Do you think that msg in stream.Send(msg) would be empty? I'll try to log it.

Thanks

[emcfarlane]

My mistake, I missed that this occurred intermittently. Might be related to networking, what is between the client and the server? Could there also be any middleware that could cancel the context?

[D4ryl00]

I don't have any middleware setup.
The client and the server are connected to the same network in the same machine, they are in two separate Docker containers. So the connection shouldn't be an issue I suppose.

[sudorandom]

@D4ryl00 I do have an angle for this. Is it actually random when this happens or is it 5 minutes after the request is started?

[D4ryl00]

It's completely random. Sometimes it's after 1 mn, sometimes 10 mn. I tried to identify a pattern to reproduce it, but it seems to be random.

[D4ryl00]

@sudorandom You were right! I've tried to test again more rigorously with a timer and the error happens after 5 minutes.
What's the problem, a misconfiguration?

[D4ryl00]

So after some tests, it's WriteTimeout that triggers that error.
Is it a good practice to put it to zero?

[sudorandom]

Yeah, reference this documentation: https://connectrpc.com/docs/faq#stream-error

The error, strangely, doesn't match what you're seeing but it appears to be caused by the same thing. Maybe they need to update this documentation.

However, here's the issue: When you configure http.Server with a ReadTimeout or WriteTimeout it will terminate the HTTP connection after the configured amount of time, regardless if there's regular messages being passed. This doesn't work well with long running streaming calls since the connection will just be terminated like you're seeing.

Is it good practice to set these values to zero? Eh, it's a bit hard to say. I think there's two lines of thinking:

• Publicly exposed services where you have little control over client behavior, you should probably always set these values. You may want to run streaming RPC requests using a different http.Server instance for long running streaming calls OR you can use a timeout middleware on every "normal" RPC call and omit this middleware for long-running streaming calls. That should give you most of the safety of the Read/WriteTimeout.
• For privately exposed services or in situations where you control the clients then it's probably fine to set the timeouts to zero.

[D4ryl00]

Ok thank you for the help 😀
I'll open an issue to update the documentation.

[emcfarlane]

Thanks both!