Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

integration: log goroutine trace on test timeout #5568

Merged
merged 3 commits into from
Dec 6, 2024

Conversation

tonistiigi
Copy link
Member

Add maximum timeout to each test and log all buildkitd
goroutines when a test gets stuck. Currently tests can time out
and will print daemon logs but these would already be after
shutdown signal is sent to the daemon.

Is there any other runtime info that is also worth printing?

Test with fake deadlock:

/src # go test -v --run /TestUserAdditionalGids/worker=oci$/frontend=builtin --count 1 ./frontend/dockerfile/
=== RUN   TestIntegration
=== RUN   TestIntegration/TestUserAdditionalGids/worker=oci/frontend=builtin
=== PAUSE TestIntegration/TestUserAdditionalGids/worker=oci/frontend=builtin
=== CONT  TestIntegration/TestUserAdditionalGids/worker=oci/frontend=builtin
    sandbox.go:129: sandbox timeout reached, stopping worker
    sandbox.go:169: buildkitd debug logs:
        goroutine 254 [running]:
        runtime/pprof.writeGoroutineStacks({0x1943840, 0x4000896000})
        	/usr/local/go/src/runtime/pprof/pprof.go:761 +0x6c
        runtime/pprof.writeGoroutine({0x1943840?, 0x4000896000?}, 0x4000782838?)
        	/usr/local/go/src/runtime/pprof/pprof.go:750 +0x2c
        runtime/pprof.(*Profile).WriteTo(0x260c330?, {0x1943840?, 0x4000896000?}, 0xc?)
        	/usr/local/go/src/runtime/pprof/pprof.go:374 +0x148

--- FAIL: TestIntegration (0.02s)
    --- FAIL: TestIntegration/TestUserAdditionalGids/worker=oci/frontend=builtin (53.20s)
FAIL
FAIL	github.com/moby/buildkit/frontend/dockerfile	53.238s

Copy link
Collaborator

@profnandaa profnandaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Add maximum timeout to each test and log all buildkitd
goroutines when a test gets stuck. Currently tests can time out
and will print daemon logs but these would already be after
shutdown signal is sent to the daemon.

Signed-off-by: Tonis Tiigi <[email protected]>
@tonistiigi tonistiigi force-pushed the intergration-test-timeout-traces branch from 47642a2 to 58b5e6a Compare December 5, 2024 22:54
@@ -18,6 +22,8 @@ import (

const buildkitdConfigFile = "buildkitd.toml"

const maxSandboxTimeout = 5 * time.Minute
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5min might not be enough for this test TestIntegration/TestBasicGhaCacheImportExport: https://github.com/moby/buildkit/actions/runs/12189447998/job/34004770018?pr=5568#step:8:2218

Default timeout is 10m:

defaultTimeout = 10 * time.Minute

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added extra time to GHA test

This test can hit rate-limiting from Github and run long.

Signed-off-by: Tonis Tiigi <[email protected]>
@crazy-max crazy-max merged commit 55a7483 into moby:master Dec 6, 2024
96 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants