Pandas, Pytorch & Tensorflow Shared Memory Support (Issue #499)

[danielwetzel]

Added Pandas Shared Memory Support for Frames

• Different Pandas Frame Types (eg. Series, Sparse, Categorical) are automatically transformed to standard frames
• With the Argument "keepIndex=True" in the from_pandas function, the original DF index is stored as first column named "index"
• With the Argument "useIndexColumn=True" the Index column from a Daphne Frame is stored as Index of the Pandas DF and no longer as separate column
• Updated Frame Operators within Python

Added Pytorch Shared Memory Support for 2d & nd Tensors (nd Tensors will be flattened)
Added Tensorflow Shared Memory Support for 2d & nd Tensors (nd Tensors will be flattened)

• Tensors are Transformed to Matrixes, the original shape can be returned with the Argument "return_shape=True" in the from_pytorch & from_tensorflow method
• Matrixes from Daphne can be returned as Torch & Tensorflow Tensors, with optional function Arguments for the compute function: "isTensorflow: bool, isPytorch: bool, shape (original shape of the tensor)"

Added Updates to the Numpy functions & to the Matrix Operators in Python

Added Support for Daphne SQL and Joins for Frames

• Only InnerJoins are working at the moment, as Semi Joins and Group Joins unexpectedly return multiple results in Daphne (Demonstrated in "scripts/examples/daphnelib/issues-with-joins.daph")
• The other two Join functions are commented out for future adjustments

Added a delete function for Daphne Objects in Python to prevent Memory Overflow

Designed all functions in a zero copy manner with strong focus on performance

Examples for all the added functions

• "scripts/examples/daphnelib/"
• with a real world example "scripts/examples/daphnelib/daphne-python-realworld-example.py"

Benchmarks & Benchmark Results

• "scripts/benchmarks/"
• "scripts/benchmarks/testoutputs/"

Function Tests

• "test/api/python/"

[danielwetzel]

Solution for Issue #499 in the Context of the LDE Project Summer 2023

[pdamme]

I completely forgot to answer here, since we had already discussed the topic offline. Thanks for your contribution, @danielwetzel and @Niklas-Ventker. Connecting DAPHNE better to the data science ecosystem by enabling efficient data transfer with widely used Python libraries is a great new feature, and highly anticipated by our use-case partners.

I will finalize your code and merge it soon.

[pdamme]

Again, thanks a lot for this valuable contribution @danielwetzel and @Niklas-Ventker! I finally found the time to get back to it now. It will be included in our upcoming v0.3 release.

I rebased your commits on the latest state of main and tidied up your code to make it ready to be merged. In that context, I also reduced it a bit to the core of the contribution regarding data exchange with Python libs. Some of the frame ops you added are also still in it.

We also appreciate the remaining things you started implementing. These were valuable for the LDE project. However, I removed some of them from this PR because they’re not in the state to be merged into main, e.g.:

• SQL processing in DaphneLib through a separate compute_sql()
• freeing of the DaphneLibResult
• the partly finished work on MultiReturn
• the scripts for your project experiments
• some example scripts

Furthermore, I can confirm that all test cases run successfully in this PR, if TensorFlow and PyTorch are installed. Since these dependencies are not included in the container we use for CI testing, I commented out all DaphneLib test cases for now. After the merge I will make a follow-up commit that makes these dependencies optional and switches on the test cases again.