[REVIEW] Python redesign for libcudf++ #3254

shwina · 2019-10-30T20:43:47Z

See the "redesign document" here.

See the corresponding RMM PR rapidsai/rmm#163

Rewrite Buffer to use rmm.DeviceBuffer instead of rmm.device_array underneath
Move Column to Cython and enable it to talk to both legacy libcudf and new libcudf++
~~[ ] Add example of porting Python functionality to libcudf++~~

…mple

…ibcudfxx-pythonexample

…mple

…ibcudfxx-python

…yops

python/cudf/cudf/_lib/copying.pyx

raydouglass · 2020-01-07T16:42:59Z

python/cudf/setup.py

+
+CUDA_HOME = os.environ.get("CUDA_HOME", False)
+if not CUDA_HOME:
+    CUDA_HOME = os.path.dirname(os.path.dirname(shutil.which("cuda-gdb")))


It's possible that the user doesn't have cuda-gdb on their path, so CUDA_HOME could still not be found at this point. Might want to check it again and if not found, raise an exception telling the user to set CUDA_HOME.

It's also possible the user has set the CUDA_HOME environment variable to a path which doesn't exist. @shwina is this needed long term or is this just a temporary workaround similar to cub / libcudacxx below?

@raydouglass any opposition to us tackling this in a follow up pr?

@kkraus14 Nope, it's just a nice-to-have to error out early on in my opinion since the user will get a compile error later that should make it obvious the CUDA includes are missing.

rjzamora

dask_cudf-specific changes LGTM - Thanks for the hard work on this @shwina !

python/cudf/cudf/_lib/cudf.pyx

python/cudf/cudf/_lib/rolling.pyx

python/cudf/cudf/_lib/unaryops.pyx

python/cudf/cudf/_lib/utils.pyx

python/cudf/cudf/core/column/categorical.py

Co-Authored-By: jakirkham <[email protected]>

python/cudf/cudf/core/column/column.py

…nto libcudfxx-python

python/cudf/cudf/core/series.py

python/cudf/setup.py

python/cudf/cudf/core/column/column.py

Co-Authored-By: jakirkham <[email protected]>

jakirkham

LGTM. Thanks for all of your hard work on this @shwina! 😄

shwina added 13 commits October 18, 2019 10:25

Initial libcudf++ bindings

762efcd

Add bindings for mutable_column_view

7bec7b3

Add initial bindings for table classes

d5d9bb7

Rework Column bindings

80f5ea5

Merge branch 'port-libcudf++-scatter-gather' into libcudfxx-pythonexa…

868399c

…mple

Initial gather libcudf++ bindings

ea27470

Add Buffer.from_array_like()

bc9c01f

Iterating on Python libcudf++ design

aebf743

Merge branch 'branch-0.11' of https://github.com/rapidsai/cudf into l…

6d83e5f

…ibcudfxx-pythonexample

Replace use of _Table and _Column with release_table and release_column

6bb4aa8

Merge branch 'branch-0.11' of https://github.com/rapidsai/cudf into l…

bcfbd3d

…ibcudfxx-pythonexample

Merge branch 'port-libcudf++-scatter-gather' into libcudfxx-pythonexa…

cf61707

…mple

Move release_*() utilities to methods

c7165bf

shwina added 2 - In Progress Currently a work in progress Python Affects Python cuDF API. Cython labels Oct 30, 2019

shwina changed the title ~~Python redesign for libcudf++~~ [DRAFT] Python redesign for libcudf++ Oct 30, 2019

shwina added 13 commits October 31, 2019 09:24

Merge branch 'branch-0.11' of https://github.com/rapidsai/cudf into l…

1047cbe

…ibcudfxx-python

Merge branch 'branch-0.11' of https://github.com/rapidsai/cudf into l…

60c65fc

…ibcudfxx-python

Merge branch 'port-libcudf++-scatter-gather' into libcudfxx-python

e1d4cae

Rudimentary null support in Python Column

ad62bce

Merge branch 'branch-0.11' of https://github.com/rapidsai/cudf into l…

efa01b7

…ibcudfxx-python

Merge branch 'port-libcudf++-scatter-gather' into libcudfxx-python

369a297

Starting on migrating Column/Buffer to libcudf++

9c472d8

progress

54925dd

Initial port of as_column

066415d

Initial port of nans_to_nulls

2193034

Add method to translate gdf_column to Column and initial port of unar…

30e7940

…yops

Update series constructor

947fe87

Initial port of unaryops

208ed89

nvdbaranec approved these changes Jan 7, 2020

View reviewed changes

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/_lib/copying.pyx Show resolved Hide resolved

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/_lib/copying.pyx Outdated Show resolved Hide resolved

Remove print statements

151475a

raydouglass approved these changes Jan 7, 2020

View reviewed changes

rjzamora approved these changes Jan 7, 2020

View reviewed changes

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/_lib/cudf.pyx Outdated Show resolved Hide resolved

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/_lib/rolling.pyx Show resolved Hide resolved

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/_lib/unaryops.pyx Show resolved Hide resolved

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/_lib/utils.pyx Show resolved Hide resolved

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/core/column/categorical.py Outdated Show resolved Hide resolved

shwina and others added 2 commits January 7, 2020 12:32

Update python/cudf/cudf/_lib/cudf.pyx

ba37d79

Co-Authored-By: jakirkham <[email protected]>

Update python/cudf/cudf/core/column/categorical.py

3514463

Co-Authored-By: jakirkham <[email protected]>

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/core/column/column.py Outdated Show resolved Hide resolved

shwina added 4 commits January 7, 2020 12:38

Better handling of CUDA_HOME in setup.py

917154b

Remove unused variable

6fbf958

Explicitly check if a valid is not NULL

3808a31

Merge branch 'libcudfxx-python' of git+ssh://github.com/shwina/cudf i…

20b1781

…nto libcudfxx-python

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/core/series.py Show resolved Hide resolved

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/setup.py Show resolved Hide resolved

jakirkham reviewed Jan 7, 2020

View reviewed changes

python/cudf/cudf/core/column/column.py Outdated Show resolved Hide resolved

Update python/cudf/cudf/core/column/column.py

201e9eb

Co-Authored-By: jakirkham <[email protected]>

jakirkham approved these changes Jan 7, 2020

View reviewed changes

kkraus14 merged commit 26ade6d into rapidsai:branch-0.12 Jan 7, 2020

kkraus14 mentioned this pull request Jan 9, 2020

[FEA] Cython wrappers for libcudf++ classes #3198

Closed

jakirkham mentioned this pull request Jan 14, 2020

[REVIEW] Drop cython from run requirements rapidsai/cugraph#667

Merged

shwina mentioned this pull request Jan 21, 2020

[BUG] Possible memory leak when merging DataFrames #3848

Closed

rjzamora mentioned this pull request Jan 28, 2020

[REVIEW] Add dask_cudf dispatch function to hash cudf.Series #3959

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[REVIEW] Python redesign for libcudf++ #3254

[REVIEW] Python redesign for libcudf++ #3254

shwina commented Oct 30, 2019 •

edited

Loading

raydouglass Jan 7, 2020

kkraus14 Jan 7, 2020

kkraus14 Jan 7, 2020

raydouglass Jan 7, 2020

rjzamora left a comment

jakirkham left a comment

[REVIEW] Python redesign for libcudf++ #3254

[REVIEW] Python redesign for libcudf++ #3254

Conversation

shwina commented Oct 30, 2019 • edited Loading

raydouglass Jan 7, 2020

Choose a reason for hiding this comment

kkraus14 Jan 7, 2020

Choose a reason for hiding this comment

kkraus14 Jan 7, 2020

Choose a reason for hiding this comment

raydouglass Jan 7, 2020

Choose a reason for hiding this comment

rjzamora left a comment

Choose a reason for hiding this comment

jakirkham left a comment

Choose a reason for hiding this comment

shwina commented Oct 30, 2019 •

edited

Loading