Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

distsqlrun: race in TCS #30769

Closed
jordanlewis opened this issue Sep 28, 2018 · 2 comments
Closed

distsqlrun: race in TCS #30769

jordanlewis opened this issue Sep 28, 2018 · 2 comments
Assignees
@andreimatei
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

Comments

@jordanlewis
Copy link
Member 

==================
Read at 0x00c421abe390 by goroutine 1415:
  reflect.Value.Len()
      /usr/local/go/src/reflect/value.go:1050 +0x19f
  encoding/json.isEmptyValue()
      /usr/local/go/src/encoding/json/encode.go:309 +0x1d5
  encoding/json.(*structEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:628 +0x319
  encoding/json.(*structEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:653 +0x7b
  encoding/json.(*structEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:639 +0x2dc
  encoding/json.(*structEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:653 +0x7b
  encoding/json.(*ptrEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:780 +0x127
  encoding/json.(*ptrEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:785 +0x7b
  encoding/json.(*structEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:639 +0x2dc
  encoding/json.(*structEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:653 +0x7b
  encoding/json.(*ptrEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:780 +0x127
  encoding/json.(*ptrEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:785 +0x7b
  encoding/json.(*encodeState).reflectValue()
      /usr/local/go/src/encoding/json/encode.go:325 +0x93
  encoding/json.interfaceEncoder()
      /usr/local/go/src/encoding/json/encode.go:611 +0xed
  encoding/json.(*structEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:639 +0x2dc
  encoding/json.(*structEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:653 +0x7b
  encoding/json.(*arrayEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:761 +0x128
  encoding/json.(*arrayEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:768 +0x7b
  encoding/json.(*sliceEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:735 +0xe1
  encoding/json.(*sliceEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:747 +0x7b
  encoding/json.(*structEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:639 +0x2dc
  encoding/json.(*structEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:653 +0x7b
  encoding/json.(*ptrEncoder).encode()
      /usr/local/go/src/encoding/json/encode.go:780 +0x127
  encoding/json.(*ptrEncoder).(encoding/json.encode)-fm()
      /usr/local/go/src/encoding/json/encode.go:785 +0x7b
  encoding/json.(*encodeState).reflectValue()
      /usr/local/go/src/encoding/json/encode.go:325 +0x93
  encoding/json.(*encodeState).marshal()
      /usr/local/go/src/encoding/json/encode.go:298 +0xb2
  encoding/json.(*Encoder).Encode()
      /usr/local/go/src/encoding/json/stream.go:200 +0xd7
  github.com/cockroachdb/cockroach/pkg/kv.GRPCTransportFactory.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport_race.go:129 +0x21f
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:324 +0xf3

Previous write at 0x00c421abe390 by goroutine 828:
  github.com/cockroachdb/cockroach/pkg/roachpb.(*EndTransactionRequest).SetHeader()
      /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.go:358 +0x80
  github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_sequence_nums.go:55 +0x20c
  github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeat).SendLocked()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeat.go:230 +0x1cb
  github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:643 +0x6ba
  github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:623 +0x123
  github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:838 +0x1cf
  github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).commit()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:525 +0x30e
  github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Commit()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:573 +0x4c
  github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).exec()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:743 +0x466
  github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:585 +0x15d
  github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.WriteResumeSpan()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/backfiller.go:243 +0x24d
  github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*backfiller).mainLoop()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/backfiller.go:142 +0xba4
  github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*backfiller).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/backfiller.go:82 +0x239
  github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*columnBackfiller).Run()
      <autogenerated>:1 +0x61
@jordanlewis jordanlewis added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Sep 28, 2018
@jordanlewis jordanlewis self-assigned this Sep 28, 2018
This was referenced Sep 28, 2018
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
@jordanlewis jordanlewis changed the title distsqlrun: race in WriteResumeSpan distsqlrun: race in TCS Sep 28, 2018
@jordanlewis
Copy link
Member Author

@andreimatei do you think this has to do with recent changes you've been involved with?

@jordanlewis jordanlewis mentioned this issue Sep 28, 2018
Closed
andreimatei added a commit to andreimatei/cockroach that referenced this issue Sep 28, 2018
@andreimatei
Revert "client: don't allocate some commit batches"
208f59b 
This reverts commit 06c1adf.

Revert the last commit from cockroachdb#30485 - the one that pre-allocation and
possible reuse of EndTransaction batches. It's causing races.
I'll figure it out and re-send the original patch.
Touches cockroachdb#30769

Release note: None
@andreimatei andreimatei mentioned this issue Sep 28, 2018
Merged
@andreimatei
Copy link
Contributor

I'm sure it has :|
Reverting while I figure it out.

craig bot pushed a commit that referenced this issue Sep 28, 2018
@andreimatei
Merge #30773
4ce1f0e 
30773: Revert "client: don't allocate some commit batches" r=andreimatei a=andreimatei

This reverts commit 06c1adf.

Revert the last commit from #30485 - the one that pre-allocation and
possible reuse of EndTransaction batches. It's causing races.
I'll figure it out and re-send the original patch.
Touches #30769

Release note: None

Co-authored-by: Andrei Matei <[email protected]>
@jordanlewis jordanlewis mentioned this issue Sep 28, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andreimatei andreimatei

Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jordanlewis @andreimatei