df.write_xxx no longer working in ballista

[yarenty]

Describe the bug

Since the update to Datafusion 30 ( rev: 7d774481aedc027b7f68226b2c3a4fc0db959fc2 )
as Dataframe moved to use LogicalPlan::Copy when executing write (csv, parquet,json)
df. write_xxx is no longer working in Ballista.

To Reproduce
One can use any sql and then df.write ie: in examples standalone_sql.rs add write_csv() instead show():

    let df = ctx.sql("select count(1) from test").await?;

    df.write_csv("output.csv", DataFrameWriteOptions::default(), None).await?;

cargo run --example standalone_sql

Output:

Error: DataFusionError(Internal("failed to serialize logical plan: Internal(\"LogicalPlan serde is not yet implemented for Copy\")"))

Expected behavior
File created.

Additional context
See PR: apache/datafusion#7283

Looks like this is todo - there are no serde / proto changes in datafusion.

Question: should I ask about it in datafusion repo ?

[andygrove]

Depends on the following DataFusion issues:

• Implement protobuf serialization for LogicalPlan::CopyTo datafusion#8596
• Add serde support for CSV FileTypeWriterOptions datafusion#8597
• Add serde support for Parquet FileTypeWriterOptions datafusion#8598
• Add serde support for Arrow FileTypeWriterOptions datafusion#8599
• Add serde support for CSV and Parquet FileSinkExec nodes datafusion#8645