Can't receive messages larger than 4MiB

[drigz]

Cloud Pub/Sub has a maximum message size of around 10 MiB (source). gRPC has a default max message size of 4MiB, so there are valid messages that cannot be pulled by google-cloud-python.

These messages appear to be silently dropped, as the pull request generates <_Rendezvous of RPC that terminated with (StatusCode.RESOURCE_EXHAUSTED, Received message larger than max (8106187 vs. 4194304))> but this is considered to be a transient error so the request is retried. When the next call to Pull() returns no messages, the application receives no messages. (behaviour may be different with production Pub/Sub servers, as I encountered this testing against an emulator)

This limit was lifted for the BigTable client in #2907, but this change didn't affect other clients. For PubSub, it can be fixed by setting the max message size options (as done for BigTable) in make_gax_publisher_api / make_gax_subscriber_api.

[tseaver]

Thanks for the report! One thing I note in the changes for #2907 is the comment that "''grpc.max_message_length' will no longer be recognized in grpcio 1.1 and later". I'm unable to verify that assertion from a scan of the grpc issue tracker.

[drigz]

It'll be ignored, and the send/receive-specific settings will be used, which is why BigQuery sets both. Pub/Sub should set all three (old and both new (send/receive)) for max compatibility.

[lukesneeringer]

And it should set it to 20 MB + 1 byte.

[lukesneeringer]

I am trying to run down how to set this properly in GAX. In theory, I would expect it to be configurable, but not sure if it is. If not, we need to add that.

[lukesneeringer]

Trip from @geigerj, recording here for my personal note-taking: https://github.com/googleapis/gax-python/blob/master/google/gax/__init__.py#L206

[lukesneeringer]

Addressed in the upcoming new library: eae7e14

[ensonic]

Thanks, you are setting the limit to 20Mb - is this configurable (overridable)? Please also note the comment @drigz posted on your change.