airbyte-server StatusRuntimeException: DEADLINE_EXCEEDED on Initial Deployment #18516
Labels
area/server
community
frozen
Not being actively worked on
team/platform-move
type/bug
Something isn't working
Comments
octavia-squidington-iv
added
team/triage
autoteam
community
team/tse
|
Hey @nullmaxwell , do you have any failure logs from the temporal container? |
|
@sh4sh here's a log dump from the Let me know if you need anything to complement these |
|
To add some context to the timeline, I am adding an excerpt that describes the first instance of the From |
|
Hmm interesting, this is hard to pin down. As far as I can tell temporal started successfully, after this snippet it appears to successfully talk to the airbyte db: So if it's not temporal container failure, or resource limit related, then my guess is that there's something weird going on with communication between the temporal and airbyte-server containers. Few quick questions:
Also, depending on how you last shut down Airbyte, it might still have some volumes and networks leftover in Docker. To completely remove these resources, you may need to run |
|
Thanks for taking the time to look into this with me. To answer your questions:
It's reassuring that you suspect there's no temporal failure here, but on the consideration that it could be a communication issue between containers, do you have any troubleshooting I can perform? |
|
I am not too sure about troubleshooting that, but if restarting doesn't fix it then I'm not sure if that's the issue to be honest. Something we haven't checked is the worker container, which Temporal appears to communicate with successfully. Can you share logs from that container to see if there is anything there? (At this point, if you can share logs from any/all containers other than the server and temporal that would be very helpful!) Also, it might be worth trying the latest Airbyte in case there has been relevant changes. It looks like the same issue here: #10053 Just to double check, can you confirm you have >= 30GB disk space? Also, if you're able to take a look at the Scaling Airbyte document and try some settings that might improve the behaviour. |
|
Sure, I've attached archives of logs from all containers on I made a reference to #10053 in my original post as well. Also, for the sake of being thorough here are the results of airbyte_deployment_logs_v04018.zip |
sh4sh
added
area/server
team/platform-move
and removed
needs-triage
team/tse
|
I am stuck on this, I've assigned it to the team's backlog in hopes they might know what's going on. 🙏🏽 Feel free to ping me if I can help with anything. |
|
@benmoriceau Can you take a look? I'd timebox the investigation. Thanks! |
|
I'm happy to provide more information if it's needed. Thanks for taking the time to triage |
|
any update on this ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment
Current Behavior
airbyte-servercontainer hangs ongrpc.StatusRuntimeException: DEADLINE_EXCEEDEDas a result ofi.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failurewhich stalls the UI with the "Cannot reach server" splash page.Expected Behavior
airbyte-servershould create the GRPC client, initialize the temporal namespace default and finish deploying.Logs
From
airbyte-servercontainer:Steps to Reproduce
git clone https://github.com/airbytehq/airbyte.gitcd air bytedocker-compose up -dAre you willing to submit a PR?
Yes
Additional Notes:
The following issues mention similar behavior as a result of a heartbeat failure (
ActivityExecutionContextImpl(doHeartBeat):153 - Heartbeat failed) :Error messages are looking similar to that of those mentioned in the following comment that references issues #4536 and #6417: #4975 (comment)
However, the main issue I am experiencing is unrelated to a heartbeat failure, suggesting this as not a duplicate of #4975, #6417, or #10053.
This discussion thread is also relevant.
Each of these issues were closed with the suggestion of upgrading to the latest version/release or increasing the amount of resources, however, doing either seems to have made no difference in standing up a deployment.
Troubleshooting Performed
Deployment on another machine
I have deployed the equivalent Airbyte version on a smaller machine with less resources using the same environment without any issues.
Asserting no memory limit on containers
$ docker stats --no-streamEnsuring no memory limit set within repository's
.envWhile is genuinely unclear to me what the problem could be, any help would be greatly appreciated as I would love to experiment with Airbyte more.
The text was updated successfully, but these errors were encountered: