Hang in MutationsBatcher.mutate() #949
Comments
product-auto-label
bot
added
the
api: bigtable
|
I should add that this hang is not deterministic. It happens about 50% of the time that the code runs. |
|
A bit more information. This issue only occurs with Given that |
* feat: Add attempt_direct_path argument to create_channel * add more test cases * fix docstring * fix docstring * update docstring of attempt_direct_path arg * update docstring of target arg * Add comment for dns_prefix local variable * Set the default value of attempt_direct_path to False * simplify conditional statement * use warnings.warn instead of _LOGGER.debug * update docstring of target arg in _modify_target_for_direct_path * s/direct_path_prefix/direct_path_separator * default->google_auth_default * parametrize target in def test_create_channel_implicit * Add github issue for TODO * filter deprecation warning related to grpcio-gcp * format docstring
|
I was able to reproduce the issue. It looks like it comes up when table.name is called in a background thread. Your observation on it being related to @parthea let me know how you want to proceed. I can probably make some patches to fix this issue on the bigtable side, but it's probably better to fix this in api_core |
|
Can you share the code that you used to create the issue, and the version of gRPC that was used? I can check to see if this is an issue in other clients as well. |
|
I reproduced it yesterday using similar code to Patrick above. But this simpler, more generic sample also hangs: In both cases, I used would run python script in a bash loop. The Python code doesn't hang every time, but when looped, it will consistently hang within a couple minutes: I have |
|
Thanks for the sample code! I can produce the issue in versions of The issue does not appear with |
|
This could be related to grpc/grpc#34672 which was fixed for |
parthea
added
type: bug
|
Marking as P2 since there is a workaround which is to use |
…th nogil (#36266) This fix is similar to #34712 but for `grpc_google_default_credentials_create` rather than `grpc_ssl_credentials_create` Fixes #36265 Fixes googleapis/python-bigtable#949 Closes #36266 COPYBARA_INTEGRATE_REVIEW=#36266 from parthea:repro-issue-34672 d736f6f PiperOrigin-RevId: 623291826
…th nogil (grpc#36266) This fix is similar to grpc#34712 but for `grpc_google_default_credentials_create` rather than `grpc_ssl_credentials_create` Fixes grpc#36265 Fixes googleapis/python-bigtable#949 Closes grpc#36266 COPYBARA_INTEGRATE_REVIEW=grpc#36266 from parthea:repro-issue-34672 d736f6f PiperOrigin-RevId: 623291826
…th nogil (v1.62.x backport) (#36376) Backport of #36266 to v1.62.x. --- This fix is similar to #34712 but for `grpc_google_default_credentials_create` rather than `grpc_ssl_credentials_create` Fixes #36265 Fixes googleapis/python-bigtable#949 Co-authored-by: Anthonios Partheniou <[email protected]>
|
@parthea and @daniel-sanche - thanks to you both! Sorry I've gone a bit silent; I was on vacation the past two weeks. If there's anything else I can do to help let me know, but it looks like y'all got to the bottom of this. |
|
Yeah I think it should be safe to close this, thanks for the detailed bug report! |
Environment details
google-cloud-bigtableversion: 2.23.0Steps to reproduce
Recently, we started seeing this usage of the MutationsBatcher hang indefinitely in
mutate(). This issue seems to be reproducible both in my local development environment (Mac) as well as in our deployed dev environment (Linux).Based on the stack traces I included below, I'm guessing there's some problem with the creation of the gRPC channel to perform the flush but it's unclear to me how to get any insight into what might be going on there.
Code example
Stack trace
The text was updated successfully, but these errors were encountered: