TypeError: Cannot read property 'pop' of null

[tallguys]

Environment details

• OS: Google Kubernetes Container
• Node.js version: 12.15.0
• npm version: to be update.
• @google-cloud/spanner version: 5.16.1

Steps to reproduce

1. ?
2. ?

We're seeing an issue in our production environment. It happens pretty inconsistently, so I'm not sure of how exactly to reproduce it. The error message and stack:

TypeError: Cannot read property 'pop' of null
    at Function.mergeLists (/deploy/my-project/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:255:65)
    at Function.merge (/deploy/my-project/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:227:41)
    at Function.mergeLists (/deploy/my-project/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:255:44)
    at Function.merge (/deploy/my-project/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:227:41)
    at Function.mergeLists (/deploy/my-project/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:255:44)
    at Function.merge (/deploy/my-project/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:227:41)
    at PartialResultStream._addChunk (/deploy/my-project/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:145:48)
    at PartialResultStream._transform (/deploy/my-project/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:84:24)
    at PartialResultStream.Transform._read (_stream_transform.js:189:10)
    at PartialResultStream.Transform._write (_stream_transform.js:177:12)
    at PartialResultStream.obj.<computed> [as _write] (/deploy/my-project/node_modules/stubs/index.js:26:26)
    at doWrite (_stream_writable.js:435:12)
    at writeOrBuffer (_stream_writable.js:419:5)
    at PartialResultStream.Writable.write (_stream_writable.js:309:11)
    at Pumpify.ondata (/deploy/my-project/node_modules/readable-stream/lib/_stream_readable.js:619:20)
    at Pumpify.emit (events.js:223:5)

The error seems can not be caught. I wrapped try-catch on all of my spanner operations but still got restart of my nodejs process.

Please let me know what other information would be helpful.

[asthamohta]

Hi @tallguys. If possible could you share, usually when you are trying to read/write the data and this error gets thrown are any of the columns of type array or struct(if yes, which kind)? Also does the issue seem to disappear when you rerun it the application or it persists?

[tallguys]

Thanks for your response @asthamohta. I'm not sure which query cause this error. However, we have the scenario to read columns in type array and struct.
The SQL is like this:

SELECT table_A.fields,
ARRAY(SELECT AS STRUCT table_B_fields FROM table_B WHERE table_A.fields_1 = table_B.fields_1),
ARRAY(SELECT AS STRUCT table_C_fields FROM table_C WHERE table_A.fields_1 = table_C.fields_1),
ARRAY(SELECT AS STRUCT table_D_fields FROM table_D WHERE table_A.fields_1 = table_D.fields_1),
FROM table_A WHERE table_A.fields_1 = "xxx"

[asthamohta]

Thanks @tallguys. Is this a persistent issue or does it disappear when you rerun the query?

[tallguys]

@asthamohta Sorry, I can't rerun because I'm not sure which query exactly is. It happens around 10 times daily. I think it might relate to specific data or network.
Would it help if I query without ARRAY and STRUCT?

[asthamohta]

@tallguys I think the query looks okay and there is an issue on our end. Can you confirm that you are able to run it on cloud console without errors? Also if possible it would be great if you could provide a db schema where you think we can replicate this issue.

[tallguys]

I can confirm we are able to run it on the cloud console without errors. It doesn't happen consistently in the code. Please see if this could help you. @asthamohta

// PARENT
CREATE TABLE table_a (
  table_a_id STRING(36) NOT NULL,
  field_1 STRING(36) NOT NULL,
  ....
  field_100 STRING(36) NOT NULL,
  ...
  field_120 ARRAY<STRING(256)>,
) PRIMARY KEY(table_a_id);

CREATE TABLE table_b (
  table_a_id STRING(36) NOT NULL,
  table_b_id STRING(36) NOT NULL,
  field_1
  ...
  field_30
) PRIMARY KEY(table_a_id, table_b_id),
  INTERLEAVE IN PARENT table_a ON DELETE CASCADE;

CREATE TABLE table_c (
  table_a_id STRING(36) NOT NULL,
  table_c_id STRING(36) NOT NULL,
  field_1
  ...
  field_20 ARRAY<STRING(256)>
) PRIMARY KEY(table_a, table_c),
  INTERLEAVE IN PARENT table_a ON DELETE CASCADE;

CREATE TABLE table_d (
  table_a_id STRING(36) NOT NULL,
  table_d_id STRING(36) NOT NULL,
  field_1
  ...
  field_20
) PRIMARY KEY(table_a_id, table_d_id),
  INTERLEAVE IN PARENT table_a ON DELETE CASCADE;

[asthamohta]

Hi @tallguys. We haven't been able to replicate your issue exactly and hence can't test it. We have tried deploying a solution to this branch: fixForPartialResultSet . Could you try running your query using this and let us know if it works?

[tallguys]

I haven't installed the new version package yet. I can't replicate it on our testing envrourment and can't do the test on production. @asthamohta

[asthamohta]

@tallguys you can install the library using
npm install git://github.com/googleapis/nodejs-spanner.git#fixForPartialResultSet --save
and then simply run the query by connecting to the database in a new js file using:

const instanceId = 'my-instance';
const databaseId = 'my-database';
const projectId = 'my-project-id';

// Imports the Google Cloud Spanner client library
const {Spanner} = require('@google-cloud/spanner');

// Instantiates a client
const spanner = new Spanner({
projectId: projectId,
});

// Gets a reference to a Cloud Spanner instance and database
const instance = spanner.instance(instanceId);
const database = instance.database(databaseId);

// Logic for running query

[tallguys]

I see, but I can't replicate it either. 😂 Could you release a stable version then I can run on the production? @asthamohta I will run the #fixForPartialResultSet version on our testing first to see if there is any other issue.

[asthamohta]

Thanks for your patience. We were able to replicate the issue and the solution works. Will merge the changes today. 😊

[tallguys]

@asthamohta Thanks. Please let me know once you guys publish a new package. I can't wait to try. :D

[asthamohta]

The new version is out. Let me know if it works for you. Thanks