spanner: add object caching #2362

callmehiphop · 2017-06-06T20:34:07Z

callmehiphop · 2017-06-06T20:35:24Z

@stephenplusplus I still need to write unit tests but I wanted your approval before I did that (in case you can think of a better way)

@Chipintoza is there any chance you could check this branch out and test with your application to see if you're still seeing the same issues?

Chipintoza · 2017-06-07T05:28:29Z

Yes today i will do this and comment you. (We have big Time Zone difference :()

Chipintoza · 2017-06-07T12:18:39Z

@callmehiphop i update these files and transaction.js from #2358
The importing process has passed but during the process there was two type of errors:

(node:4049) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 55): TypeError: Cannot set property 'beginTime_' of undefined
this errors was continue made after the process was stopped.
noStatus - TypeError: this.stream.end is not a function
at StreamProxy._onFinish (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/google-gax/lib/streaming.js:103:17)
at StreamProxy. (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/google-gax/lib/streaming.js:76:10)
at emitNone (events.js:86:13)
at StreamProxy.emit (events.js:185:7)
at finishMaybe (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/readable-stream/lib/_stream_writable.js:504:14)
at endWritable (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/readable-stream/lib/_stream_writable.js:514:3)
at StreamProxy.Writable.end (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/readable-stream/lib/_stream_writable.js:484:41)
at StreamProxy. (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules//src/transaction.js:374:23)
at emitOne (events.js:96:13)
at StreamProxy.emit (events.js:188:7)

and data imported in Users Table 21903 instead of 33742

Chipintoza · 2017-07-06T06:49:44Z

please look comment

callmehiphop · 2017-07-10T18:42:54Z

@Chipintoza sorry this kind of slipped away from me. I put in some additional error handling and rebased the code against master. Could you give this another test run? If the errors persist could you point me to the files & lines where the errors originate from? I couldn't determine that from the stacks you provided.

Chipintoza · 2017-07-11T07:53:25Z

@stephenplusplus i run my import and during the process there was:

noStatus - TypeError: this.stream.end is not a function
at StreamProxy._onFinish (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/google-gax/lib/streaming.js:103:17)
at StreamProxy. (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/google-gax/lib/streaming.js:76:10)
at emitNone (events.js:86:13)
at StreamProxy.emit (events.js:185:7)
at finishMaybe (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/readable-stream/lib/_stream_writable.js:503:14)
at endWritable (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/readable-stream/lib/_stream_writable.js:513:3)
at StreamProxy.Writable.end (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/readable-stream/lib/_stream_writable.js:483:41)
at StreamProxy. (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@gss-llc/database-utils.gss.ge/node_modules/@google-cloud/spanner/src/transaction.js:375:19)
at emitOne (events.js:96:13)
at StreamProxy.emit (events.js:188:7)

and

(node:33729) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 19): TypeError: Cannot read property 'run' of undefined
(node:33729) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 20): TypeError: Cannot read property 'run' of undefined
(node:33729) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 21): TypeError: Cannot read property 'run' of undefined
(node:33729) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 22): TypeError: Cannot read property 'run' of undefined
(node:33729) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 23): TypeError: Cannot read property 'run' of undefined

at the end - Too many active sessions, limit is 10000

(node:33729) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 60): TypeError: Cannot read property 'run' of undefined
(node:33729) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 61): TypeError: Cannot read property 'run' of undefined
(node:33729) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 62): TypeError: Cannot read property 'run' of undefined
(node:33729) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 63): TypeError: Cannot read property 'run' of undefined

req.url - http://accounts.gss.local:9001/gss-api/environments/upsert
InternalServerError: err.code: 8
err.message: Too many active sessions in database projects/spanner-gss-ge/instances/development-data/databases/system-development, limit is 10000. Increase the node count to allow more sessions.
err.status: undefined
err location:
Environment.upsert\database.runTransaction((err, tx) => {...
data:
{"environmentId":"23e8c345-d5a4-442e-8092-e47f695662c5","isActive":true,"created":1427575695237,"timestamp":1499759051813,"modifierId":"system-id","modifierName":"system","name":"ჩემი 'GSS გარემო' - [email protected]|en:My 'GSS Environment' - [email protected]|ru:Моя 'GSS Среда' - [email protected]","databaseID":"39aa56bc-f926-4df2-afe9-619c1e527afe","isUnderControl":false,"environmentSyncURL":"","linkedGSSContactID":null,"linkedContactID":null,"inputLanguages":"en ka ru","usersInputLanguages":"","registrationCountry":"","rsgeServiceUser":"","rsgeServiceUserPassword":"","mainCurrencySignification":null,"accountingCurrencySignifications":"","domain":null,"gssEducationIsActive":false}
apiResponse:
at Function.dbUtils.getDsError (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@gss-llc/database-utils.gss.ge/index.js:213:16)
at error (/Users/Chipintoza/GSS Projects/accounts.gss.ge/server/tables/system/environments-table.js:327:64)
at database.runTransaction (/Users/Chipintoza/GSS Projects/accounts.gss.ge/server/tables/system/environments-table.js:342:27)
at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@gss-llc/database-utils.gss.ge/node_modules/@google-cloud/spanner/src/database.js:816:7
at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@gss-llc/database-utils.gss.ge/node_modules/@google-cloud/spanner/src/database.js:1040:7
at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@gss-llc/database-utils.gss.ge/node_modules/@google-cloud/spanner/src/session-pool.js:471:9
at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@gss-llc/database-utils.gss.ge/node_modules/@google-cloud/spanner/src/session-pool.js:230:9
at process._tickCallback (internal/process/next_tick.js:103:7)

packages/spanner/src/transaction.js

@@ -371,7 +371,6 @@ Transaction.prototype.requestStream = function(config) {
      return;
    }

-    requestStream.end();


coveralls · 2017-07-14T11:36:08Z

Coverage increased (+2.0e-05%) to 99.988% when pulling 5ecdc9f on callmehiphop:dg--spanner-obj-cache into 4b5158e on GoogleCloudPlatform:master.

stephenplusplus · 2017-07-14T13:35:24Z

LGTM -- @Chipintoza do you want to give this PR a try before we merge?

Chipintoza · 2017-07-14T21:16:15Z

@stephenplusplus i run my imports, it take a much time, in my country now is 01:14 AM, tomorrow i will report everything.

Chipintoza · 2017-07-15T15:26:48Z

@stephenplusplus as i understand everything works fine!

I did not have optimal importing process and the request for update for same record was parallel many times, maybe more than 1000 time. it looks like hotspots occurred.

There was two types of errors

req.url - http://accounts.gss.local:9001/gss-api/environments/upsert
DsError: err.code: 4
err.message: Deadline for Transaction exceeded.
err.status: undefined
err.stack: undefined
err location:
Environment.upsert\database.runTransaction((err, tx) => {...
data:
{"environmentId":"4ff7a78e-eff4-4ee4-a5ec-ee8d2f1ed8dc","isActive":true,"created":1413465452830,"timestamp":1500123845696,"modifierId":"system-id","modifierName":"system","name":"en:My 'GSS Environment' - [email protected]|ru:Моя 'GSS Среда' - [email protected]|ka:საქართველოს ეროვნული ბანკი","databaseID":"39aa56bc-f926-4df2-afe9-619c1e527afe","isUnderControl":false,"environmentSyncURL":"","linkedGSSContactID":null,"linkedContactID":null,"inputLanguages":"en ka ru","usersInputLanguages":"","registrationCountry":"","rsgeServiceUser":"","rsgeServiceUserPassword":"","mainCurrencySignification":null,"accountingCurrencySignifications":"","domain":null,"gssEducationIsActive":false}
apiResponse:

    at Function.dbUtils.getDsError (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@gss-llc/database-utils.gss.ge/index.js:224:28)
    at error (/Users/Chipintoza/GSS Projects/accounts.gss.ge/server/tables/system/environments-table.js:325:64)
    at Transaction.database.runTransaction [as runFn_] (/Users/Chipintoza/GSS Projects/accounts.gss.ge/server/tables/system/environments-table.js:340:27)
    at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@google-cloud/spanner/src/transaction.js:338:14
    at _combinedTickCallback (internal/process/next_tick.js:71:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

req.url - http://accounts.gss.local:9001/gss-api/environments/upsert
DsError: err.code: undefined
err.message: Unable to create Transaction.
err.status: undefined
err.stack: Error: Unable to create Transaction.
    at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@google-cloud/spanner/src/database.js:892:26
    at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@google-cloud/spanner/src/database.js:450:13
    at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@google-cloud/spanner/src/session-pool.js:234:7
    at process._tickCallback (internal/process/next_tick.js:103:7)
err location:
Environment.upsert\database.runTransaction((err, tx) => {...
data:
{"environmentId":"19c0c143-24af-4567-9b24-95d84a08d176","isActive":true,"created":1458117721470,"timestamp":1500123919337,"modifierId":"system-id","modifierName":"system","name":"ჩემი 'GSS გარემო' - [email protected]|en:My 'GSS Environment' - [email protected]|ru:Моя 'GSS Среда' - [email protected]","databaseID":"39aa56bc-f926-4df2-afe9-619c1e527afe","isUnderControl":false,"environmentSyncURL":"","linkedGSSContactID":null,"linkedContactID":null,"inputLanguages":"en ka ru","usersInputLanguages":"","registrationCountry":"","rsgeServiceUser":"","rsgeServiceUserPassword":"","mainCurrencySignification":null,"accountingCurrencySignifications":"","domain":null,"gssEducationIsActive":false}
apiResponse:

    at Function.dbUtils.getDsError (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@gss-llc/database-utils.gss.ge/index.js:224:28)
    at error (/Users/Chipintoza/GSS Projects/accounts.gss.ge/server/tables/system/environments-table.js:325:64)
    at database.runTransaction (/Users/Chipintoza/GSS Projects/accounts.gss.ge/server/tables/system/environments-table.js:340:27)
    at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@google-cloud/spanner/src/database.js:892:13
    at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@google-cloud/spanner/src/database.js:450:13
    at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@google-cloud/spanner/src/session-pool.js:234:7
    at process._tickCallback (internal/process/next_tick.js:103:7)

As i understand all 100 sessions were in retry process and because of this Transactions could not be created.

Am i right?

When i optimised import, everything ran well.

Thank you everyone

stephenplusplus · 2017-07-15T21:27:09Z

Thank you for testing!

@callmehiphop does everything described above sound expected?

callmehiphop · 2017-07-17T09:08:28Z

@stephenplusplus actually I made a small change in a recent commit, but otherwise everything sounds good. PTAL!

coveralls · 2017-07-17T09:13:54Z

Coverage increased (+2.0e-05%) to 99.988% when pulling e4872ec on callmehiphop:dg--spanner-obj-cache into 4b5158e on GoogleCloudPlatform:master.

callmehiphop added api: spanner Issues related to the Spanner API. status: do not merge labels Jun 6, 2017

callmehiphop requested a review from stephenplusplus June 6, 2017 20:34

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 6, 2017

Chipintoza mentioned this pull request Jul 11, 2017

Spanner - noStatus - TypeError: requestStream.destroy is not a function #2356

Closed

callmehiphop added 5 commits July 14, 2017 02:27

spanner: add object caching

aa51918

move database cache to instance property

0df24e7

add check for missing txn object

6a081df

update call to public method

67c273f

remove destroy call to request stream

1031864

callmehiphop commented Jul 14, 2017

View reviewed changes

packages/spanner/src/transaction.js

@@ -371,7 +371,6 @@ Transaction.prototype.requestStream = function(config) {

return;

}

requestStream.end();

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

callmehiphop added 2 commits July 14, 2017 07:27

add caching tests

684969e

linting error

5ecdc9f

callmehiphop removed the status: do not merge label Jul 14, 2017

store txn options when polling for session

e4872ec

stephenplusplus merged commit 6b7c948 into googleapis:master Jul 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spanner: add object caching #2362

spanner: add object caching #2362

callmehiphop commented Jun 6, 2017

callmehiphop commented Jun 6, 2017

Chipintoza commented Jun 7, 2017

Chipintoza commented Jun 7, 2017

Chipintoza commented Jul 6, 2017

callmehiphop commented Jul 10, 2017

Chipintoza commented Jul 11, 2017

This comment was marked as spam.

This comment was marked as spam.

coveralls commented Jul 14, 2017

stephenplusplus commented Jul 14, 2017

Chipintoza commented Jul 14, 2017 •

edited

Loading

Chipintoza commented Jul 15, 2017

stephenplusplus commented Jul 15, 2017

callmehiphop commented Jul 17, 2017

coveralls commented Jul 17, 2017

spanner: add object caching #2362

spanner: add object caching #2362

Conversation

callmehiphop commented Jun 6, 2017

callmehiphop commented Jun 6, 2017

Chipintoza commented Jun 7, 2017

Chipintoza commented Jun 7, 2017

Chipintoza commented Jul 6, 2017

callmehiphop commented Jul 10, 2017

Chipintoza commented Jul 11, 2017

This comment was marked as spam.

This comment was marked as spam.

coveralls commented Jul 14, 2017

stephenplusplus commented Jul 14, 2017

Chipintoza commented Jul 14, 2017 • edited Loading

Chipintoza commented Jul 15, 2017

stephenplusplus commented Jul 15, 2017

callmehiphop commented Jul 17, 2017

coveralls commented Jul 17, 2017

Chipintoza commented Jul 14, 2017 •

edited

Loading