Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: hibernate failed #34693

Closed
cockroach-teamcity opened this issue Feb 7, 2019 · 13 comments
Closed

roachtest: hibernate failed #34693

cockroach-teamcity opened this issue Feb 7, 2019 · 13 comments
Assignees
@BramGruneir
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/66f13c1d9a12c31e18a198da4ff5ac0bbe2db781

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1127849&tab=buildLog

The test failed on provisional_201902062125_v2.2.0-alpha.20190211:
	test.go:748,hibernate.go:285,hibernate.go:296: 
		Tests run on Cockroach v2.2.0-alpha.20181217-1103-g66f13c1
		6285 Total Tests Run
		5241 tests passed
		1044 tests failed
		0 tests passed unexpectedly
		744 tests failed unexpectedly
		3 tests expected failed, but not run 
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_2) is available in the artifacts' hibernate log
@cockroach-teamcity cockroach-teamcity added this to the 2.2 milestone Feb 7, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Feb 7, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/4e2e450ecf5c3f73c7de46e48696528468803db7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1128035&tab=buildLog

The test failed on master:
	test.go:748,hibernate.go:285,hibernate.go:296: 
		Tests run on Cockroach v2.2.0-alpha.20181217-1110-g4e2e450
		3801 Total Tests Run
		1817 tests passed
		1984 tests failed
		0 tests passed unexpectedly
		1854 tests failed unexpectedly
		173 tests expected failed, but not run 
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_2) is available in the artifacts' hibernate log

@asubiotto
Copy link
Contributor

The second failures seem to be predominantly:

org.hibernate.test.resulttransformer.ResultTransformerTest > classMethod FAILED
    org.hibernate.testing.junit4.CallbackException
        Caused by: org.hibernate.service.spi.ServiceException
            Caused by: org.hibernate.exception.JDBCConnectionException
                Caused by: org.postgresql.util.PSQLException
                    Caused by: java.net.ConnectException

@asubiotto asubiotto mentioned this issue Feb 7, 2019
Closed
@BramGruneir
Copy link
Member

BramGruneir commented Feb 7, 2019 via email

@BramGruneir
Copy link
Member

From the logs, it looks like this crashed the node:

SELECT
  count(*) AS col_0_0_
FROM
  human AS human0_
  INNER JOIN mammal AS human0_1_ ON human0_.mammal = human0_1_.animal
  INNER JOIN animal AS human0_2_ ON human0_.mammal = human0_2_.id
  CROSS JOIN human_family AS family1_,
  human AS human2_
  INNER JOIN mammal AS human2_1_ ON human2_.mammal = human2_1_.animal
  INNER JOIN animal AS human2_2_ ON human2_.mammal = human2_2_.id
WHERE
  (human0_.mammal = family1_.human1 AND family1_.human2 = human2_.mammal)
  AND (
      SELECT
        family1_.human2
      FROM
        human_family AS family1_,
        human AS human2_
        INNER JOIN mammal AS human2_1_ ON human2_.mammal = human2_1_.animal
        INNER JOIN animal AS human2_2_ ON human2_.mammal = human2_2_.id
      WHERE
        human0_.mammal = family1_.human1 AND family1_.human2 = human2_.mammal
    )
    = $1;

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/4bdf0ad7b3190e1836e311d2b05478e8cda0bfe6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1129727&tab=buildLog

The test failed on provisional_201902072331_v2.2.0-alpha.20190211:
	hibernate.go:285,hibernate.go:296,test.go:1206: 
		Tests run on Cockroach v2.2.0-alpha.20181217-1133-g4bdf0ad
		5406 Total Tests Run
		4667 tests passed
		739 tests failed
		0 tests passed unexpectedly
		508 tests failed unexpectedly
		72 tests expected failed, but not run 
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_2) is available in the artifacts' hibernate log

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/e4127466ae0a44efb624afcaf0861903028b6e18

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1129825&tab=buildLog

The test failed on release-2.1:
	hibernate.go:285,hibernate.go:296,test.go:1206: 
		Tests run on Cockroach v2.1.4-57-ge412746
		6637 Total Tests Run
		6325 tests passed
		312 tests failed
		0 tests passed unexpectedly
		2 tests failed unexpectedly
		0 tests expected failed, but not run 
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_1) is available in the artifacts' hibernate log

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/70cd0458348655fb74f4e2cb89aa202b5dd89ed0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1129807&tab=buildLog

The test failed on master:
	hibernate.go:285,hibernate.go:296,test.go:1206: 
		Tests run on Cockroach v2.2.0-alpha.20181217-1136-g70cd045
		5992 Total Tests Run
		5509 tests passed
		483 tests failed
		0 tests passed unexpectedly
		261 tests failed unexpectedly
		81 tests expected failed, but not run 
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_2) is available in the artifacts' hibernate log

@BramGruneir
Copy link
Member

The 2.1 failure worries me. It doesn't seem to be related to the sql panic.

I'm going to spend monday getting to the bottom of both.

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/40e403544d60b1a44b8b1ed961a817c77d67aa31

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1131411&tab=buildLog

The test failed on master:
	hibernate.go:285,hibernate.go:296,test.go:1206: 
		Tests run on Cockroach v2.2.0-alpha.20181217-1158-g40e4035
		4256 Total Tests Run
		2353 tests passed
		1903 tests failed
		0 tests passed unexpectedly
		1760 tests failed unexpectedly
		160 tests expected failed, but not run 
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_2) is available in the artifacts' hibernate log

@BramGruneir BramGruneir mentioned this issue Feb 11, 2019
Closed
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/b80d241d5693d6ba2bde01ae9167b49b04b47226

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1133193&tab=buildLog

The test failed on master:
	hibernate.go:285,hibernate.go:296,test.go:1206: 
		Tests run on Cockroach v2.2.0-alpha.20190211-57-gb80d241
		5641 Total Tests Run
		4832 tests passed
		809 tests failed
		0 tests passed unexpectedly
		568 tests failed unexpectedly
		62 tests expected failed, but not run 
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_2) is available in the artifacts' hibernate log

@BramGruneir
Copy link
Member

So I've investigated the 2.1 failure and it was a transient failure.
Once the panic in #34795 is addressed, this should fix the 2.2 hibernate test.

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/10f8010fa5778e740c057905e2d7664b5fd5d647

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1135549&tab=buildLog

The test failed on master:
	hibernate.go:285,hibernate.go:296,test.go:1212: 
		Tests run on Cockroach v2.2.0-alpha.20190211-100-g10f8010
		3867 Total Tests Run
		2525 tests passed
		1342 tests failed
		0 tests passed unexpectedly
		1227 tests failed unexpectedly
		188 tests expected failed, but not run 
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_2) is available in the artifacts' hibernate log

@BramGruneir
Copy link
Member

Fixed via #34795.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BramGruneir BramGruneir

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.1
Development

No branches or pull requests
3 participants
@BramGruneir @asubiotto @cockroach-teamcity