Can't delete from Spark DeltaLake table if it has a timestamp field

[bhuacret]

Environment

Delta-rs version:0.12.0
Binding: Rust

Environment:dev

• Cloud provider:Azure
• OS:Linux/Windows
• Other:

---

Bug

What happened:
I'm trying to delete some data from a delta table in Rust, using with_predicate.

let (table,_) = DeltaOps(table).delete().with_predicate(col("fechabloqueo").eq(cast(lit("2022-11-06"),Date32))).await.unwrap();

but im getting "Execution error: Failed to map column projection for field fechacreacion. Incompatible data types Timestamp(Nanosecond, None) and Timestamp(Microsecond, None)"

What you expected to happen:

Delete successfully with that condition.

How to reproduce it:

More details:
The table was created using default INT96 outputTimestampType option.

If i rewrite the table using spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")
still fails:
"Execution error: Failed to map column projection for field fechacreacion. Incompatible data types Timestamp(Microsecond, Some(\"UTC\")) and Timestamp(Microsecond, None)"

[cmackenzie1]

Out of curiosity, is spark.sql.parquet.timestampNTZ.enabled set to true of false?

[bhuacret]

Just tested with both spark.sql.parquet.timestampNTZ.enabled true and false at Spark 3.4.0, i keep getting the same error.

[Blajda]

This is caused by ambiguity on how to store timestamps in the serialization section of the protocol. The older version of the protocol specified that timestamp columns do not specified the timezone but this change has modified it to be adjusted to UTC.

I did some local testing. Updating the code here to include the timezone of UTC and performing of casts in the user supplied query (e.g (cast(lit("2022-11-06"),arrow_schema::DataType::Timestamp(arrow_schema::TimeUnit::Microsecond, Some("UTC".into())))) will fix this issue. I'm just not certain if this change is a breaking change.

related: delta-io/delta#643

[ion-elgreco]

Resolved by #2615, and using logical plans in delete and datafusion/arrow-rs can do type coercion during parquet reading