Make PyResultSet and PyRecordBatch serializable/picklable

[frank-lsf]

The first step towards implementing shuffle using Ray is that we need a way to put the execution result into the Ray Object Store. Currently https://github.com/datafusion-contrib/ray-sql/blob/main/raysql/worker.py#L22 returns a string representation of the result set which would not work if we want to recover the result set from the worker return value. We need to make the ResultSet and therefore RecordBatch classes picklable in Python.

[frank-lsf]

@andygrove I'm working on this. Is there a way to serialize a Rust RecordBatch into raw bytes?

Also do you know how to make this work in pyo3? I'm reading PyO3/pyo3#100 which led me to https://gist.github.com/ethanhs/fd4123487974c91c7e5960acc9aa2a77.

[andygrove]

Hi @franklsf95. Yes, it is possible to serialize RecordBatch in Arrow IPC format. DataFusion has an IPCWriter struct that does this. You can see how this is used in the current Ray SQL ShuffleWriter.

To make this work with Pyo3, I think I would just delegate to Rust to serialize the batches to bytes then have a Python class that contains the bytes. This is what I did with the query plan - see https://github.com/datafusion-contrib/ray-sql/blob/main/raysql/context.py#L74

[frank-lsf]

Done in #28.