Additional Support For Nullable Attributes

[nguyenv]

Background

As detailed in sc-34754, this fixes a bug found by a customer using the TileDB-SOMA Python API where the SOMADataFrame containing an enumerated nullable attribute was not being readback correctly. This highlights a larger deficit in the TileDB-Py codebase in which we have little support for writing nullable attributes outside of utilizing tiledb.from_pandas with Pandas's ExtensionDtype.

Changes

• This PR supports writing Pyarrow arrays and Pandas dataframes that contain nullable values (pd.NA, pa.na, None, etc.).
• Nullable attributes are now represented in Numpy as masked arrays.
• PyQuery results now also return the validity buffer.
• Note that in Pyarrow, the validity values represent 0 = invalid, 1 = valid, whereas in Numpy, this is inverted and mask values represent 0 = valid, 1 = invalid.

Future Proposals

• Support writing numpy.ma for nullable attributes.

with tiledb.open(uri, "w') as A:
   A[:] = np.ma.array(data, mask)

• Support writing with built-in sequences (eg. list, tuple). Internally, we check if the attribute .isnullable() and then cast using np.ma.masked_invalid().

with tiledb.open(uri, "w') as A:
   A[:] = [1, 2, None, 3]

[shortcut-integration]

This pull request has been linked to Shortcut Story #34754: Not-in-enum handled incorrectly.