Possible backwards incompatible change or regression in v1.42.0

[blkperl]

What version of gRPC are you using?

1.42.0

What version of Go are you using (go version)?

1.17

What operating system (Linux, Windows, …) and version?

Linux

What did you do?

The Argo Workflows project is trying to update from grpc 1.41.1 to 1.42.0 and it looks like there is a regression or behavior change not documented in the release notes. The upgrade pr is argoproj/argo-workflows#7658. The tests pass on 1.41.1 argoproj/argo-workflows@afd59bc and fail on 1.42.0 argoproj/argo-workflows@e8ac272

The following tests fails after update to grpc 1.42.0: https://github.com/argoproj/argo-workflows/blob/master/test/e2e/argo_server_test.go#L1091-L1102

The test sets the Accept header as text/event-stream and asserts that the response header Content-Type is event-stream. On grpc v1.42.0 the test suite fails because the content type is application/json and is a 500 instead of a 200 response.

=== RUN   TestArgoServerSuite/TestEventSourcesService/WatchEventSources
    argo_server_test.go:1101: 
        	Error Trace:	argo_server_test.go:1101
        	            				argo_server_test.go:1638
        	            				suite.go:77
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 500
        	Test:       	TestArgoServerSuite/TestEventSourcesService/WatchEventSources
    argo_server_test.go:1102: 
        	Error Trace:	argo_server_test.go:1102
        	            				argo_server_test.go:1638
        	            				suite.go:77
        	Error:      	Not equal: 
        	            	expected: "text/event-stream"
        	            	actual  : "application/json"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-text/event-stream
        	            	+application/json
        	Test:       	TestArgoServerSuite/TestEventSourcesService/WatchEventSources

Any assistance would be greatly appreciated.

What did you expect to see?

I expected 1.42.0 to not cause a test failure or to contain a note in the release notes on what to change.

What did you see instead?

A test failure indicating a behavior change.

[menghanl]

Is there a proxy in the server?
The content-type (text/event-stream or application/json) is not gRPC's context-type. And the test sends an http request, not gRPC.

Can you also try turning on the gRPC logs https://github.com/grpc/grpc-go#how-to-turn-on-logging

[blkperl]

@menghanl Hi, yes using grpc-gateway as a proxy to the gRPC server

https://github.com/argoproj/argo-workflows/blob/master/server/apiserver/argoserver.go#L296-L298

I will try enabling the gRPC logs and report back

[blkperl]

Hi @menghanl ,

I think this is the relevant log

argo-server | time="2022-02-02T20:13:42.897Z" level=warning msg="Non-transient error: failed to evaluate workflow template expression: unable to evaluate expression '': unexpected token EOF (1:1)"
argo-server | time="2022-02-02T20:13:42.897Z" level=error msg="failed to dispatch from event" error="failed to evaluate workflow template expression: unable to evaluate expression '': unexpected token EOF (1:1)" event=malformed namespace=argo
argo-server | time="2022-02-02T20:13:42.897Z" level=error msg="finished unary call with code Unknown" error="failed to dispatch event: [failed to evaluate workflow template expression: unable to evaluate expression '': unexpected token EOF (1:1)]" grpc.code=Unknown grpc.method=ReceiveEvent grpc.service=event.EventService grpc.start_time="2022-02-02T20:13:42-08:00" grpc.time_ms=9.869 span.kind=server system=grpc
argo-server | time="2022-02-02T20:13:43.083Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=CreateEventSource grpc.service=eventsource.EventSourceService grpc.start_time="2022-02-02T20:13:43-08:00" grpc.time_ms=10.457 span.kind=server system=grpc
argo-server | time="2022-02-02T20:13:43.094Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListEventSources grpc.service=eventsource.EventSourceService grpc.start_time="2022-02-02T20:13:43-08:00" grpc.time_ms=9.695 span.kind=server system=grpc
argo-server | 2022/02/02 20:13:43 ERROR: [transport] transport: http2Server.operateHeaders parsed a :connection header which makes a request malformed as per the HTTP/2 spec

[menghanl]

transport: http2Server.operateHeaders parsed a :connection header which makes a request malformed as per the HTTP/2 spec

This looks like grpc-ecosystem/grpc-gateway#2447

There was a behavior change in the server transport. Now requests with connection header will be rejected (this was missed in the 1.42.0 release notes, I just added it).

The problem was fixed in grpc-gateway. Can you try the newer version? Thanks!

[blkperl]

Thanks @menghanl for your assistance. That explains it. We are unfortunately unable to update to the 2.x grpc-gateway until Kubernetes updates due to the dependencies between the grpc eco system and gogo-protobuf/protofbuf.