HL API additions for dask #129
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ihnorton
commented
Apr 2, 2019
tiledb/libtiledb.pyx
Outdated
| check_error(ctx, | ||
| tiledb_vfs_fh_is_closed(ctx.ptr, self.ptr, &isclosed)) | ||
| if not isclosed: | ||
| with nogil: |
There was a problem hiding this comment.
This seems like the behavior we want, need to be careful about resource use here:
https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#controlling-deallocation-and-garbage-collection-in-cpython
This commit adds an 'attr` specifier to DenseArray, which is used to open multi-attribute array as a single-attribute view. Needed for use-cases which do not support multi-attribute arrays. - Add pickle/unpickle support for DenseArray - Add serialization for Config so that head node can control worker node thread settings.
- tiledb.{open,save}: convenience functions to load and save an array
- tiledb.schema_like: create a schema from an existing numpy array or
{dtype, shape}
- empty_like: create a writable array based on specifiers to
'schema_like', but do not materialize anything in memory.
jakebolewski
reviewed
Apr 8, 2019
jakebolewski
reviewed
Apr 8, 2019
jakebolewski
approved these changes
Apr 8, 2019
|
Some comments, otherwise it lgtm |
|
🎉 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains
rework DenseArray to support single-attribute view on a multi-attribute array
set of API additions, motivated by dask support (but hopefully generally useful):
open,savefrom_numpyto allow use as simplytiledb.from_numpyDim,Domain)schema_like: create ArraySchema based on a numpy ndarray template, or numpy-styleshapeanddtypekwargsempty_like: returns a writable, emtpy array similar to an existing ndarray-like object (.dtype,.shape).basic CI for dask integration to make sure we don't break that here, going forward