XATTRS: Panic in updateAndReturnDoc

[sethrosetter]

Sync Gateway version

1.5.0-452

Operating system

CentOS7

Config file

{
        "adminInterface": "0.0.0.0:4985", 
        "compressResponses": false, 
        "databases": {
            "db": {
                "bucket": "data-bucket", 
                "import_docs": "continuous", 
                "password": "password", 
                "server": "http://XXXX:8091", 
                "unsupported": {
                    "enable_extended_attributes": true
                }, 
                "username": "data-bucket"
            }
        }, 
        "interface": ":4984", 
        "log": [
            "CRUD+", 
            "Cache+", 
            "HTTP+", 
            "Changes+", 
            "Import+"
        ], 
        "maxCouchbaseConnections": 16, 
        "maxFileDescriptors": 90000, 
        "maxIncomingConnections": 0, 
        "slowServerCallWarningThreshold": 500
    }

Log output

http://uberjenkins.sc.couchbase.com:8080/job/cen7-sync-gateway-functional-tests-base-cc/767/artifact/results/logs/test_sg_sdk_interop_shared_docs%5Bsync_gateway_default_functional_tests-1-1000%5D-2017-06-26-11-28-40-sglogs.zip

Expected behavior

Sync Gateway should not panic during concurrent updates with the SDK

Actual behavior

Sync Gateway panics during concurrent updates with the SDK

Steps to reproduce

1. http://uberjenkins.sc.couchbase.com:8080/job/cen7-sync-gateway-functional-tests-base-cc/767/testReport/junit/testsuites.syncgateway.functional.tests/test_xattrs/test_sg_sdk_interop_shared_docs_sync_gateway_default_functional_tests_1_1000_/

[tleyden]

Panic message:

2017-06-26T14:28:38.447-04:00 CRUD+: Got old revision "doc_set_one_0" / "1974-2ec50de4d72fc937c6786c6c085538f3" --> 130 bytes
2017-06-26T14:28:38.447-04:00 CRUD: 	Doc "doc_set_two_0" in channels "{}"
2017-06-26T14:28:38.448-04:00 CRUD: 	Doc "doc_set_one_0" in channels "{}"
2017-06-26T14:28:38.448-04:00 CRUD+: updateDoc("doc_set_two_0"): Pruned 1001 old revisions
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7df70a]

goroutine 147 [running]:
panic(0xae5aa0, 0xc420010060)
	/usr/local/go/1.7.4/go/src/runtime/panic.go:500 +0x1a1
github.com/couchbase/sync_gateway/db.(*Database).updateAndReturnDoc.func2(0x0, 0x0, 0x0, 0xc420f84000, 0xd78b, 0xe000, 0x14cbbf5edd1b0000, 0x0, 0x0, 0xf629a0, ...)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:965 +0x1aa
github.com/couchbase/sync_gateway/base.CouchbaseBucketGoCB.WriteUpdateWithXattr(0xc42027a750, 0xc420188d50, 0x28, 0xbaa23e, 0x7, 0xc42018d1b0, 0xb, 0x0, 0x0, 0xf629a0, ...)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/bucket_gocb.go:1232 +0x27e
github.com/couchbase/sync_gateway/base.(*CouchbaseBucketGoCB).WriteUpdateWithXattr(0xc42027a7e0, 0xc420255b60, 0xd, 0xba7c38, 0x5, 0x0, 0xc420f199b0, 0x42f5f0, 0xc42017a3f8, 0x300000002)
	<autogenerated>:484 +0xde
github.com/couchbase/sync_gateway/db.(*Database).updateAndReturnDoc(0xc420f16dc0, 0xc420255b60, 0xd, 0x1, 0xc420f1af50, 0x1, 0x1, 0xc42011e400, 0x4102ee, 0xc4204f9b90)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:971 +0x1f37
github.com/couchbase/sync_gateway/db.(*Database).ImportDoc(0xc420f16dc0, 0xc420255b60, 0xd, 0xc420f19950, 0xc420255c01, 0x14cbbf5edd1b0000, 0x0, 0x7a6673, 0xc42011e400, 0x7fb76d1bf000)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:683 +0x274
github.com/couchbase/sync_gateway/db.(*Database).ImportDocRaw(0xc420f16dc0, 0xc420255b60, 0xd, 0x0, 0x0, 0x0, 0x1, 0x14cbbf5edd1b0000, 0x0, 0x0, ...)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:624 +0x199
github.com/couchbase/sync_gateway/db.(*changeCache).DocChanged.func1()
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/change_cache.go:371 +0x1233
created by github.com/couchbase/sync_gateway/db.(*changeCache).DocChanged
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/change_cache.go:471 +0x2b3

[sethrosetter]

This is also observed in the DI tests - http://uberjenkins.sc.couchbase.com:8080/job/cen7-sync-gateway-functional-tests-base-di/661/testReport/testsuites.syncgateway.functional.tests/test_xattrs/test_sg_sdk_interop_shared_docs_sync_gateway_default_functional_tests_1_1000_/

[tleyden]

I reproduced the same panic locally against vagrant via:

pytest --xattrs --mode=cc --server-version=5.0.0-3193 --sync-gateway-version=1.5.0-452 -k "test_sg_sdk_interop_shared_docs[sync_gateway_default_functional_tests-1-1000]"  testsuites/syncgateway/functional/tests/

[tleyden]

PR up: #2693