sql/importer: TestImportDefaultWithResume failed [ingestKVs hangs with BulkAdderFlushesEveryBatch] #92910

cockroach-teamcity · 2022-12-02T12:58:38Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ 7cb778506d75bbef2eb90abccaa75b9dc7e3fb91:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume2888010391
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 298969 [semacquire]:
        sync.runtime_Semacquire(0x11bbf05?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc0043bbdf0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc00487fbc0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00487fbc0?, {0x6b20dd0?, 0xc00487fb80?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b20e78, 0xc007ce73e0}, 0xc002c47400, 0xc005abb8d0, 0xc008b5ba40, 0xc00487fa40?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7b9
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 298971 [semacquire]:
        sync.runtime_Semacquire(0x11bbf05?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc002c8b808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc00487fcc0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00487fcc0?, {0x6b20dd0?, 0xc00487fc80?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b20dd0?, 0xc00487fb80?}, 0xc002c47400, 0xc005abb8d0, 0xc008b5ba40, 0xc004855ec0)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b20dd0, 0xc00487fb80})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 298973 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x311
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b20dd0, 0xc00487fc80})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (5.74s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-experience _{This test on roachdash | Improve this report!

Jira issue: CRDB-22045}

The text was updated successfully, but these errors were encountered:

andreimatei · 2022-12-02T20:01:38Z

happened here too

rafiss · 2022-12-02T21:27:30Z

This new failure looks to be the same as #91828 (comment)

The ingestKvs function is getting stuck in the BulkAdderFlushesEveryBatch testing knob:

cockroach/pkg/sql/importer/import_processor.go

Lines 544 to 548 in a16106e

    
           if flowCtx.Cfg.TestingKnobs.BulkAdderFlushesEveryBatch { 
        
           	_ = pkIndexAdder.Flush(ctx) 
        
           	_ = indexAdder.Flush(ctx) 
        
           	pushProgress() 
        
           }

@adityamaru or @dt - do you have any idea why this might happen?

adityamaru · 2022-12-02T21:33:59Z

Let me take a look, not sure off the top of my head.

rafiss · 2022-12-02T21:55:30Z

Since it seems related to the BulkAdder behavior, I'm putting this in the DR team board. Let us know if you want to give it back.

blathers-crl · 2022-12-02T21:55:45Z

cc @cockroachdb/disaster-recovery

cockroach-teamcity · 2022-12-03T13:06:00Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ ca079ac1fc8a8f7a5360f5b1f4fa210c14dc8410:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume1553331737
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 52425960 [semacquire]:
        sync.runtime_Semacquire(0x11bee65?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc003195df0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0147911c0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0147911c0?, {0x6b40b10?, 0xc014791180?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b40bb8, 0xc02069adb0}, 0xc006f52200, 0xc00a1dc6d0, 0xc01e57f560, 0x4891e5?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7b9
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 52425963 [semacquire]:
        sync.runtime_Semacquire(0x11bee65?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc00980b808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc014791580)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc014791580?, {0x6b40b10?, 0xc014791540?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b40b10?, 0xc014791180?}, 0xc006f52200, 0xc00a1dc6d0, 0xc01e57f560, 0xc00dd89b60)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b40b10, 0xc014791180})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 52425992 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x325
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b40b10, 0xc014791540})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (9.42s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-04T13:14:02Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ 859e1e1f26d79af43cce16519f9ba984baaa495c:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume2933919849
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 282140 [semacquire]:
        sync.runtime_Semacquire(0x11bbc05?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc00608bdf0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc003376140)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc003376140?, {0x6b39cf0?, 0xc003376080?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b39d98, 0xc00d358090}, 0xc0031e1000, 0xc00712fd50, 0xc009eaa540, 0xc003bb9fc0?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7b9
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 282142 [semacquire]:
        sync.runtime_Semacquire(0x11bbc05?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc005f37808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc003376380)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc003376380?, {0x6b39cf0?, 0xc003376300?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b39cf0?, 0xc003376080?}, 0xc0031e1000, 0xc00712fd50, 0xc009eaa540, 0xc00dd05f20)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b39cf0, 0xc003376080})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 282144 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x311
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b39cf0, 0xc003376300})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (5.76s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-05T12:19:29Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ 8a5cb5109d6b7e122e996a44f940bfd61698cd81:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume297126914
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 284283 [semacquire]:
        sync.runtime_Semacquire(0x11bbc25?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc002cf5df0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0072050c0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0072050c0?, {0x6b39d70?, 0xc007205080?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b39e18, 0xc004c819b0}, 0xc00565f800, 0xc0034a98d0, 0xc00b11d0e0, 0xc007204fc0?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7b9
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 284285 [semacquire]:
        sync.runtime_Semacquire(0x11bbc25?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc002e67808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0072051c0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0072051c0?, {0x6b39d70?, 0xc007205180?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b39d70?, 0xc007205080?}, 0xc00565f800, 0xc0034a98d0, 0xc00b11d0e0, 0xc00427e060)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b39d70, 0xc007205080})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 284287 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x311
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b39d70, 0xc007205180})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (5.96s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

adityamaru · 2022-12-05T17:02:46Z

I'll start a bisect today to figure out what started causing this to fail.

adityamaru · 2022-12-05T22:51:32Z

The failure started after #89649, now trying to look into the patch but from the description the failure we're saying makes sense.

cockroach-teamcity · 2022-12-07T11:34:59Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ 1b4947bf4e1036a7c55996d2cb6ab815254a3c8c:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume3951419972
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 297732 [semacquire]:
        sync.runtime_Semacquire(0x11c8165?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc005d71df0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0038b5fc0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0038b5fc0?, {0x6b74d30?, 0xc0038b5f80?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b74dd8, 0xc00383c7b0}, 0xc003d44400, 0xc0038306d0, 0xc003c98ae0, 0xc0038b5e00?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7b9
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 297734 [semacquire]:
        sync.runtime_Semacquire(0x11c8165?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc002d3f808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc000a04540)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000a04540?, {0x6b74d30?, 0xc000a04440?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b74d30?, 0xc0038b5f80?}, 0xc003d44400, 0xc0038306d0, 0xc003c98ae0, 0xc008be87e0)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b74d30, 0xc0038b5f80})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 297736 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x311
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b74d30, 0xc000a04440})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (6.28s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

adityamaru · 2022-12-07T16:09:29Z

I have a patch that fixes the flake that looks very similar to the bigger change @stevendanna has around import processor shutdown in #91615. We plan on merging this soon, so I'm going to hold off on sending an independent patch.

cockroach-teamcity · 2022-12-08T11:47:28Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ ec095bc2fdbe4e518b076db20e4920fab67222bf:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume620298263
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 285560 [semacquire]:
        sync.runtime_Semacquire(0x11c8185?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc005dfbdf0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0040f2b40)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0040f2b40?, {0x6b7a210?, 0xc0040f2b00?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b7a2b8, 0xc0097e10b0}, 0xc0086eca00, 0xc003ee0b50, 0xc007510780, 0xc0040f2840?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7b9
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 285562 [semacquire]:
        sync.runtime_Semacquire(0x11c8185?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc005ded808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0040f2c40)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0040f2c40?, {0x6b7a210?, 0xc0040f2c00?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b7a210?, 0xc0040f2b00?}, 0xc0086eca00, 0xc003ee0b50, 0xc007510780, 0xc00f6203c0)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b7a210, 0xc0040f2b00})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 285564 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x311
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b7a210, 0xc0040f2c00})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (5.99s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-09T08:51:04Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ 3910ac409195ab6ab2ec3e9f6eda553048afa23f:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume4169183669
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 268008 [semacquire]:
        sync.runtime_Semacquire(0x11c81e5?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc004e6ddf0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0059dadc0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0059dadc0?, {0x6b7e6d0?, 0xc0059dad80?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b7e778, 0xc003d898f0}, 0xc0039cd200, 0xc001dda250, 0xc002ed9080, 0xc0088f3000?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7b9
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 268010 [semacquire]:
        sync.runtime_Semacquire(0x11c81e5?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc004f99808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0059daec0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0059daec0?, {0x6b7e6d0?, 0xc0059dae80?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b7e6d0?, 0xc0059dad80?}, 0xc0039cd200, 0xc001dda250, 0xc002ed9080, 0xc007f03f20)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b7e6d0, 0xc0059dad80})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 268012 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x311
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b7e6d0, 0xc0059dae80})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (6.31s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-10T11:44:03Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ f2b00e8039af6ea8887ec124dad8daf19da6fbf1:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume176071449
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 267298 [semacquire]:
        sync.runtime_Semacquire(0x11c8265?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc0047d3df0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0036ab500)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0036ab500?, {0x6b78a50?, 0xc0036ab4c0?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b78af8, 0xc0074e9fb0}, 0xc0050ad600, 0xc00337dd50, 0xc00a0827e0, 0xc0036ab240?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7a6
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 267300 [semacquire]:
        sync.runtime_Semacquire(0x11c8265?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc0060cf808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc0036ab600)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0036ab600?, {0x6b78a50?, 0xc0036ab5c0?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b78a50?, 0xc0036ab4c0?}, 0xc0050ad600, 0xc00337dd50, 0xc00a0827e0, 0xc009ede840)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b78a50, 0xc0036ab4c0})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 267302 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x311
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b78a50, 0xc0036ab5c0})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (6.01s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-11T10:25:23Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ c050c9b4b57ecf2ceb5d449c31c617fe12c920e0:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume1357980605
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 269235 [semacquire]:
        sync.runtime_Semacquire(0x11c8265?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc0053a1df0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc007f51d80)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc007f51d80?, {0x6b79130?, 0xc007f51d40?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b791d8, 0xc00d3ddaa0}, 0xc003ff4400, 0xc003e41d50, 0xc009fd0120, 0xc007f51c80?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7a6
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 269237 [semacquire]:
        sync.runtime_Semacquire(0x11c8265?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc003b2d808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc007f51ec0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc007f51ec0?, {0x6b79130?, 0xc007f51e80?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b79130?, 0xc007f51d40?}, 0xc003ff4400, 0xc003e41d50, 0xc009fd0120, 0xc00451aba0)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b79130, 0xc007f51d40})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 269239 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x311
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b79130, 0xc007f51e80})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (5.92s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-12-12T11:04:17Z

sql/importer.TestImportDefaultWithResume failed with artifacts on master @ 942a4d468e9c8ad0ef45a7be33f0a326dfb19fef:

    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportDefaultWithResume43282058
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestImportDefaultWithResume
    import_stmt_test.go:4712: -- test log scope end --
    import_stmt_test.go:4712: Leaked goroutine: goroutine 268478 [semacquire]:
        sync.runtime_Semacquire(0x11c84e5?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc005b51df0?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc00338fcc0)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00338fcc0?, {0x6b799d0?, 0xc00338fc80?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport({0x6b79a78, 0xc00f2bd530}, 0xc0030c1a00, 0xc002e058d0, 0xc000b86600, 0xc00338fa00?)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:127 +0x7a6
        github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start.func1()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:183 +0x8b
        created by github.com/cockroachdb/cockroach/pkg/sql/importer.(*readImportDataProcessor).Start
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:181 +0xeb
        Leaked goroutine: goroutine 268480 [semacquire]:
        sync.runtime_Semacquire(0x11c84e5?)
        	GOROOT/src/runtime/sema.go:62 +0x25
        sync.(*WaitGroup).Wait(0xc0034e5808?)
        	GOROOT/src/sync/waitgroup.go:139 +0x52
        golang.org/x/sync/errgroup.(*Group).Wait(0xc00338fe80)
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:53 +0x27
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00338fe80?, {0x6b799d0?, 0xc00338fe40?}})
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:144 +0x4a
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs({0x6b799d0?, 0xc00338fc80?}, 0xc0030c1a00, 0xc002e058d0, 0xc000b86600, 0xc00d9b1500)
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:553 +0x1028
        github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func2({0x6b799d0, 0xc00338fc80})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:108 +0x6a
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
        Leaked goroutine: goroutine 268482 [chan send]:
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func3()
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:468 +0x311
        github.com/cockroachdb/cockroach/pkg/sql/importer.ingestKvs.func5({0x6b799d0, 0xc00338fe40})
        	github.com/cockroachdb/cockroach/pkg/sql/importer/import_processor.go:547 +0x2cf
        github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
        	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
        created by golang.org/x/sync/errgroup.(*Group).Go
        	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestImportDefaultWithResume (6.15s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

stevendanna · 2022-12-20T16:25:44Z

Should be closed by #93782

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Dec 2, 2022

cockroach-teamcity added this to the 23.1 milestone Dec 2, 2022

blathers-crl bot added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Dec 2, 2022

rafiss removed the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Dec 2, 2022

blathers-crl bot added the T-disaster-recovery label Dec 2, 2022

rafiss changed the title ~~sql/importer: TestImportDefaultWithResume failed~~ sql/importer: TestImportDefaultWithResume failed [ingestKVs hangs with BulkAdderFlushesEveryBatch] Dec 5, 2022

rafiss assigned adityamaru Dec 5, 2022

msbutler added X-noreuse Prevent automatic commenting from CI test failures and removed X-noreuse Prevent automatic commenting from CI test failures labels Dec 6, 2022

adityamaru mentioned this issue Dec 16, 2022

importer: use ctx in progress push, use ctxgroup #93782

Merged

stevendanna closed this as completed Dec 20, 2022

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql/importer: TestImportDefaultWithResume failed [ingestKVs hangs with BulkAdderFlushesEveryBatch] #92910

sql/importer: TestImportDefaultWithResume failed [ingestKVs hangs with BulkAdderFlushesEveryBatch] #92910

cockroach-teamcity commented Dec 2, 2022 •

edited by cockroach-jira-scripts

Loading

andreimatei commented Dec 2, 2022

rafiss commented Dec 2, 2022

adityamaru commented Dec 2, 2022

rafiss commented Dec 2, 2022

blathers-crl bot commented Dec 2, 2022

cockroach-teamcity commented Dec 3, 2022

cockroach-teamcity commented Dec 4, 2022

cockroach-teamcity commented Dec 5, 2022

adityamaru commented Dec 5, 2022

adityamaru commented Dec 5, 2022

cockroach-teamcity commented Dec 7, 2022

adityamaru commented Dec 7, 2022

cockroach-teamcity commented Dec 8, 2022

cockroach-teamcity commented Dec 9, 2022

cockroach-teamcity commented Dec 10, 2022

cockroach-teamcity commented Dec 11, 2022

cockroach-teamcity commented Dec 12, 2022

stevendanna commented Dec 20, 2022

sql/importer: TestImportDefaultWithResume failed [ingestKVs hangs with BulkAdderFlushesEveryBatch] #92910

sql/importer: TestImportDefaultWithResume failed [ingestKVs hangs with BulkAdderFlushesEveryBatch] #92910

Comments

cockroach-teamcity commented Dec 2, 2022 • edited by cockroach-jira-scripts Loading

andreimatei commented Dec 2, 2022

rafiss commented Dec 2, 2022

adityamaru commented Dec 2, 2022

rafiss commented Dec 2, 2022

blathers-crl bot commented Dec 2, 2022

cockroach-teamcity commented Dec 3, 2022

cockroach-teamcity commented Dec 4, 2022

cockroach-teamcity commented Dec 5, 2022

adityamaru commented Dec 5, 2022

adityamaru commented Dec 5, 2022

cockroach-teamcity commented Dec 7, 2022

adityamaru commented Dec 7, 2022

cockroach-teamcity commented Dec 8, 2022

cockroach-teamcity commented Dec 9, 2022

cockroach-teamcity commented Dec 10, 2022

cockroach-teamcity commented Dec 11, 2022

cockroach-teamcity commented Dec 12, 2022

stevendanna commented Dec 20, 2022

cockroach-teamcity commented Dec 2, 2022 •

edited by cockroach-jira-scripts

Loading