Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flake - random tests are failing due to timeout on write #1996

Closed
zimnx opened this issue Jun 28, 2024 · 6 comments
Closed

Flake - random tests are failing due to timeout on write #1996

zimnx opened this issue Jun 28, 2024 · 6 comments
Assignees
@tnozicka
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@zimnx
Copy link
Collaborator

zimnx commented Jun 28, 2024 •
edited
Loading

This was on -clusterip job which has slower persistent network attached ssds. We might want to reevaluate whether we still want to use them.

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1991/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1806721597633990656

Snippet of what failed.

   [FAILED] Unexpected error:
      <*fmt.wrapError | 0xc0006967a0>: 
      can't insert data: Operation timed out for 58qkpsqt.test - received only 1 responses from 2 CL=ALL.
      {
          msg: "can't insert data: Operation timed out for 58qkpsqt.test - received only 1 responses from 2 CL=ALL.",
          err: <*gocql.RequestErrWriteTimeout | 0xc0004f4380>{
              errorFrame: {
                  frameHeader: {version: 132, flags: 0, stream: 576, op: 0, length: 104, warnings: nil},
                  code: 4352,
                  message: "Operation timed out for 58qkpsqt.test - received only 1 responses from 2 CL=ALL.",
              },
              Consistency: 5,
              Received: 1,
              BlockFor: 2,
              WriteType: "SIMPLE",
          },
      }
  occurred
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/verify.go:312 @ 06/28/24 16:30:32.464
@zimnx zimnx added the kind/flake Categorizes issue or PR as related to a flaky test. label Jun 28, 2024
@scylla-operator-bot scylla-operator-bot bot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Jun 28, 2024
@zimnx zimnx mentioned this issue Jun 28, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Jun 28, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1991/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1806749115925663744

@tnozicka
Copy link
Contributor

tnozicka commented Jul 1, 2024

We don' have a choice on some platforms - preferably we'd adjust the timeout or concurrency based o what exactly goes so slow.
/priority important-longterm
/triage accepted

@scylla-operator-bot scylla-operator-bot bot added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 1, 2024
@tnozicka
Copy link
Contributor

tnozicka commented Jul 2, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1871/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1808031823570145280#1:test-build-log.txt%3A1205

@tnozicka tnozicka mentioned this issue Jul 2, 2024
Merged
@tnozicka
Copy link
Contributor

tnozicka commented Jul 9, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1971/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1810558454176157696#1:test-build-log.txt%3A962

@tnozicka tnozicka self-assigned this Jul 9, 2024
@tnozicka tnozicka added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Jul 9, 2024
@tnozicka tnozicka mentioned this issue Jul 9, 2024
Merged
1 task
@tnozicka
Copy link
Contributor

tnozicka commented Jul 9, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1971/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1810586982485594112#1:test-build-log.txt%3A750

@tnozicka
Copy link
Contributor

tnozicka commented Jul 9, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/1971/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1810605342397042688#1:test-build-log.txt%3A1258

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tnozicka tnozicka

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zimnx @tnozicka