Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CLI tests failure due to data race #1594

Closed
shahzadlone opened this issue Jun 27, 2023 · 1 comment
Closed

CLI tests failure due to data race #1594

shahzadlone opened this issue Jun 27, 2023 · 1 comment
Assignees
@nasdf
Labels
area/testing Related to any test or testing suite bug Something isn't working

Comments

@shahzadlone
Copy link
Member

The cli tests have a race condition, that makes them fail on some slower machines.

Was introduced here: #928

There has been an attempt to fix this in this PR: #1415

Failure Output:

=== FAIL: tests/integration/cli TestRequestWithErrorNoType (5.05s)
Created config file at /tmp/TestRequestWithErrorNoType2187137018/001/config.yaml
Starting DefraDB service...
Opening badger store
2023-06-22T16:22:04.538-0400    ERROR   badger  [email protected]/db.go:1101 Failure while flushing memtable to disk: error while creating table err: while creating table: /tmp/TestRequestSimple2316297766/001/data/000012.sst error: open /tmp/TestRequestSimple2316297766/001/data/000012.sst: no such file or directory
unable to open: /tmp/TestRequestSimple2316297766/001/data/000012.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:80
github.com/dgraph-io/badger/v3/table.CreateTable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:259
github.com/dgraph-io/badger/v3.(*DB).handleFlushTask
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1062
github.com/dgraph-io/badger/v3.(*DB).flushMemtable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1084
github.com/dgraph-io/badger/v3.Open.func5
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:357
runtime.goexit
        /home/shahzad/.local/bin/go1.19.5/src/runtime/asm_amd64.s:1594. Retrying...

==================
WARNING: DATA RACE
Write at 0x000005f0bff0 by goroutine 2162:
  github.com/sourcenetwork/defradb/tests/integration/cli.captureOutput()
      /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:166 +0xd0
  github.com/sourcenetwork/defradb/tests/integration/cli.runDefraCommand()
      /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:121 +0x4f6
  github.com/sourcenetwork/defradb/tests/integration/cli.TestRequestWithErrorNoType()
      /home/shahzad/~~/clefradb/tests/integration/cli/client_query_test.go:45 +0x32a
  testing.tRunner()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1493 +0x47

Previous read at 0x000005f0bff0 by goroutine 2165:
  github.com/spf13/cobra.(*Command).OutOrStdout()
      /home/shahzad/go/pkg/mod/github.com/spf13/[email protected]/command.go:363 +0x554
  github.com/spf13/cobra.(*Command).InitDefaultCompletionCmd()
      /home/shahzad/go/pkg/mod/github.com/spf13/[email protected]/completions.go:687 +0x548
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/shahzad/go/pkg/mod/github.com/spf13/[email protected]/command.go:1023 +0x152
  github.com/spf13/cobra.(*Command).ExecuteContextC()
      /home/shahzad/go/pkg/mod/github.com/spf13/[email protected]/command.go:1001 +0x1ee
  github.com/sourcenetwork/defradb/cli.(*DefraCommand).Execute()
      /home/shahzad/~~/clefradb/cli/cli.go:109 +0x171
  github.com/sourcenetwork/defradb/tests/integration/cli.runDefraNode.func1()
      /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:93 +0x1db

Goroutine 2162 (running) created at:
  testing.(*T).Run()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1493 +0x75d
  testing.runTests.func1()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1846 +0x99
  testing.tRunner()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1446 +0x216
  testing.runTests()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1844 +0x7ec
  testing.(*M).Run()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1726 +0xa84
  main.main()
      _testmain.go:119 +0x2e9

Goroutine 2165 (running) created at:
  github.com/sourcenetwork/defradb/tests/integration/cli.runDefraNode()
      /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:88 +0x731
  github.com/sourcenetwork/defradb/tests/integration/cli.TestRequestWithErrorNoType()
      /home/shahzad/~~/clefradb/tests/integration/cli/client_query_test.go:42 +0x157
  testing.tRunner()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1493 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x000005f0bfe0 by goroutine 2162:
  github.com/sourcenetwork/defradb/tests/integration/cli.captureOutput()
      /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:167 +0x10a
  github.com/sourcenetwork/defradb/tests/integration/cli.runDefraCommand()
      /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:121 +0x4f6
  github.com/sourcenetwork/defradb/tests/integration/cli.TestRequestWithErrorNoType()
      /home/shahzad/~~/clefradb/tests/integration/cli/client_query_test.go:45 +0x32a
  testing.tRunner()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1493 +0x47

Previous read at 0x000005f0bfe0 by goroutine 2165:
  github.com/sourcenetwork/defradb/logging.(*logger).ApplyConfig()
      /home/shahzad/~~/clefradb/logging/logger.go:178 +0x5a4
  github.com/sourcenetwork/defradb/logging.updateLoggers()
      /home/shahzad/~~/clefradb/logging/registry.go:51 +0x264
  github.com/sourcenetwork/defradb/logging.SetConfig()
      /home/shahzad/~~/clefradb/logging/logging.go:76 +0x119
  github.com/sourcenetwork/defradb/config.(*LoggingConfig).load()
      /home/shahzad/~~/clefradb/config/config.go:641 +0x331
  github.com/sourcenetwork/defradb/config.(*Config).load()
      /home/shahzad/~~/clefradb/config/config.go:239 +0x284
  github.com/sourcenetwork/defradb/config.(*Config).LoadWithRootdir()
      /home/shahzad/~~/clefradb/config/config.go:152 +0x28e
  github.com/sourcenetwork/defradb/cli.MakeStartCommand.func1()
      /home/shahzad/~~/clefradb/cli/start.go:61 +0x1e4
  github.com/spf13/cobra.(*Command).execute()
      /home/shahzad/go/pkg/mod/github.com/spf13/[email protected]/command.go:915 +0xa02
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/shahzad/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068 +0x5fa
  github.com/spf13/cobra.(*Command).ExecuteContextC()
      /home/shahzad/go/pkg/mod/github.com/spf13/[email protected]/command.go:1001 +0x1ee
  github.com/sourcenetwork/defradb/cli.(*DefraCommand).Execute()
      /home/shahzad/~~/clefradb/cli/cli.go:109 +0x171
  github.com/sourcenetwork/defradb/tests/integration/cli.runDefraNode.func1()
      /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:93 +0x1db

Goroutine 2162 (running) created at:
  testing.(*T).Run()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1493 +0x75d
  testing.runTests.func1()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1846 +0x99
  testing.tRunner()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1446 +0x216
  testing.runTests()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1844 +0x7ec
  testing.(*M).Run()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1726 +0xa84
  main.main()
      _testmain.go:119 +0x2e9

Goroutine 2165 (running) created at:
  github.com/sourcenetwork/defradb/tests/integration/cli.runDefraNode()
      /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:88 +0x731
  github.com/sourcenetwork/defradb/tests/integration/cli.TestRequestWithErrorNoType()
      /home/shahzad/~~/clefradb/tests/integration/cli/client_query_test.go:42 +0x157
  testing.tRunner()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:1493 +0x47
==================
    utils.go:123: executing defra command with args [--url localhost:65021 client query query { User { whatever } } --rootdir /tmp/TestRequestWithErrorNoType2187137018/003]
2023-06-22T16:22:05.555-0400    ERROR   badger  [email protected]/db.go:1101 Failure while flushing memtable to disk: error while creating table err: while creating table: /tmp/TestRequestSimple2316297766/001/data/000013.sst error: open /tmp/TestRequestSimple2316297766/001/data/000013.sst: no such file or directory
unable to open: /tmp/TestRequestSimple2316297766/001/data/000013.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:80
github.com/dgraph-io/badger/v3/table.CreateTable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:259
github.com/dgraph-io/badger/v3.(*DB).handleFlushTask
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1062
github.com/dgraph-io/badger/v3.(*DB).flushMemtable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1084
github.com/dgraph-io/badger/v3.Open.func5
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:357
runtime.goexit
        /home/shahzad/.local/bin/go1.19.5/src/runtime/asm_amd64.s:1594. Retrying...

2023-06-22T16:22:06.666-0400    ERROR   badger  [email protected]/db.go:1101 Failure while flushing memtable to disk: error while creating table err: while creating table: /tmp/TestRequestSimple2316297766/001/data/000014.sst error: open /tmp/TestRequestSimple2316297766/001/data/000014.sst: no such file or directory
unable to open: /tmp/TestRequestSimple2316297766/001/data/000014.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:80
github.com/dgraph-io/badger/v3/table.CreateTable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:259
github.com/dgraph-io/badger/v3.(*DB).handleFlushTask
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1062
github.com/dgraph-io/badger/v3.(*DB).flushMemtable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1084
github.com/dgraph-io/badger/v3.Open.func5
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:357
runtime.goexit
        /home/shahzad/.local/bin/go1.19.5/src/runtime/asm_amd64.s:1594. Retrying...

    client_query_test.go:47: expected [""] to contain "Cannot query field"
2023-06-22T16:22:08.091-0400    ERROR   badger  [email protected]/db.go:1101 Failure while flushing memtable to disk: error while creating table err: while creating table: /tmp/TestRequestSimple2316297766/001/data/000015.sst error: open /tmp/TestRequestSimple2316297766/001/data/000015.sst: no such file or directory
unable to open: /tmp/TestRequestSimple2316297766/001/data/000015.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:80
github.com/dgraph-io/badger/v3/table.CreateTable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:259
github.com/dgraph-io/badger/v3.(*DB).handleFlushTask
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1062
github.com/dgraph-io/badger/v3.(*DB).flushMemtable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1084
github.com/dgraph-io/badger/v3.Open.func5
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:357
runtime.goexit
        /home/shahzad/.local/bin/go1.19.5/src/runtime/asm_amd64.s:1594. Retrying...

    testing.go:1319: race detected during execution of test

=== FAIL: tests/integration/cli TestPingCommandNoHost (unknown)
    utils.go:123: executing defra command with args [--url localhost:61504 client ping --url localhost:55092 --rootdir /tmp/TestPingCommandNoHost3295764822/002]
2023-06-22T16:22:09.281-0400    ERROR   badger  [email protected]/db.go:1101 Failure while flushing memtable to disk: error while creating table err: while creating table: /tmp/TestRequestSimple2316297766/001/data/000016.sst error: open /tmp/TestRequestSimple2316297766/001/data/000016.sst: no such file or directory
unable to open: /tmp/TestRequestSimple2316297766/001/data/000016.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:80
github.com/dgraph-io/badger/v3/table.CreateTable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:259
github.com/dgraph-io/badger/v3.(*DB).handleFlushTask
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1062
github.com/dgraph-io/badger/v3.(*DB).flushMemtable
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1084
github.com/dgraph-io/badger/v3.Open.func5
        /home/shahzad/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:357
runtime.goexit
        /home/shahzad/.local/bin/go1.19.5/src/runtime/asm_amd64.s:1594. Retrying...

panic: Fail in goroutine after TestRequestWithErrorNoType has completed

goroutine 1724 [running]:
testing.(*common).Fail(0xc00105a340)
        /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:824 +0x1a8
testing.(*common).Errorf(0xc00105a340, {0x22e5156, 0x3}, {0xc00121c3b0, 0x1, 0x1})
        /home/shahzad/.local/bin/go1.19.5/src/testing/testing.go:941 +0xaf
github.com/stretchr/testify/assert.Fail({0x28e5060, 0xc00105a340}, {0xc0006d5900, 0x48a}, {0x0, 0x0, 0x0})
        /home/shahzad/go/pkg/mod/github.com/stretchr/[email protected]/assert/assertions.go:333 +0x438
github.com/stretchr/testify/assert.NoError({0x28e5060, 0xc00105a340}, {0x28e3060, 0xc008076050}, {0x0, 0x0, 0x0})
        /home/shahzad/go/pkg/mod/github.com/stretchr/[email protected]/assert/assertions.go:1495 +0x127
github.com/sourcenetwork/defradb/tests/integration/cli.runDefraNode.func1()
        /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:94 +0x1fb
created by github.com/sourcenetwork/defradb/tests/integration/cli.runDefraNode
        /home/shahzad/~~/clefradb/tests/integration/cli/utils.go:88 +0x732
@shahzadlone shahzadlone added bug Something isn't working area/testing Related to any test or testing suite labels Jun 27, 2023
@AndrewSisley AndrewSisley added this to the DefraDB v0.7 milestone Oct 4, 2023
@AndrewSisley
Copy link
Contributor

Fixed by @nasdf in #1839

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nasdf nasdf

Labels
area/testing Related to any test or testing suite bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nasdf @shahzadlone @AndrewSisley