Datastore fails when saving entity with null attribute value

[toboid]

When trying to save an object that has null property values gcloud-node errors:

dataset.insert({
  key: dataset.key('Test'),
  data: { foo: 'bar', baz: null }
}, err => {
  if (err) return console.log(err);
});

prints:

Error: Unsupported field value, null, is provided.

I can see that the definition for Value in the Datastore proto buffers api does not provide for null values. The Ruby and Python clients create a message with no value attribute, is this an option for gcloud-node? I'd be really pleased to take a look at this if you'll take a pull request?

[stephenplusplus]

Definitely, that would be very welcome!

// @pcostell - makes sense, right?

[pcostell]

Makes sense to me. Can we make sure this gets updated for the v1beta3 branch as well? In particular the next version of the API includes NullValue as an explicit field: https://github.com/google/googleapis/blob/master/google/datastore/v1beta3/entity.proto#L119

[toboid]

OK great. Correctly saving the null to datastore is straightforward, however when the entity is retrieved the null value comes back out as an empty array because this function isn't able to distinguish between a property representing an empty array and a property representing a null value since they have the exact same structure:

{
  "key": {
    ...
  },
  "property": [
    {
      "name": "i-am-a-null",
      "value": {
        "boolean_value": null,
        "integer_value": null,
        "double_value": null,
        "timestamp_microseconds_value": null,
        "key_value": null,
        "blob_key_value": null,
        "string_value": null,
        "blob_value": null,
        "entity_value": null,
        "list_value": [],
        "meaning": null,
        "indexed": true
      }
    },
    {
      "name": "i-am-an-empty-array",
      "value": {
        "boolean_value": null,
        "integer_value": null,
        "double_value": null,
        "timestamp_microseconds_value": null,
        "key_value": null,
        "blob_key_value": null,
        "string_value": null,
        "blob_value": null,
        "entity_value": null,
        "list_value": [],
        "meaning": null,
        "indexed": true
      }
    }
  ]
}

I'm still looking into how to handle this, do you have any suggestions? Should be easier for the v1beta3 branch.

[stephenplusplus]

That makes for a tricky problem, indeed. Can we glean any insight into handling this from the other client libraries?

[stephenplusplus]

@pcostell added nullValue support to the v1beta3 PR 👍

[mcfarljw]

I'm also finding it a bit quirky to deal with nulls being returned as empty arrays.

[toboid]

I have the changes to correctly store null values here but still can't see a solution for getting them back out correctly at the moment

[pcostell]

The Google versions of protobuf support this using the hasField method, which allows querying for whether a field has been explicitly set or not. There is an issue in the protobufjs issue tracker for this.

As mentioned above, this is fixed in v1beta3 (mostly because v1beta3 uses proto3, where unset and the default value are considered the same thing).

[stephenplusplus]

Let's just wait for v1beta3 to have official support for this. It sounds like it's quite a hassle to support it now and v1beta3 is on the horizon. Support is ready in my v1beta3 PR, so I'm going to close the issue, but please re-open if there's something we can do to support it now.

[toboid]

Makes sense, I can't see a non-hacky way to correctly retrieve nulls at the moment.

[krisnye]

I have a fork which adds support for writing null values to the datastore today via a trivial fix. It does NOT fix the problem with reading them as empty arrays. (Not a problem for my use case.)

https://github.com/krisnye/gcloud-node