feat(csharp/src/Drivers/Apache/Spark): poc - Support for Apache Spark over HTTP (non-Arrow)

[birschick-bq]

Proof-of-concept

Adds support for standard Apache Spark data source

• Requires adbc.spark.type parameter option to specify the Spark server type.
• Handles the possibility that data source does not support "catalog" in the table object namespace (i.e., only schema and table names).

Closes #2015

[CurtHagenlocher]

Thanks for your contribution! This is only a partial review as I've exceeded the time I had allocated for a first pass.

[CurtHagenlocher]

Is this a fundamental limitation or an implementation issue? Are we getting back enough information to do the conversion locally when the data is returned in a Thrift vs Arrow format?

[birschick-bq]

@CurtHagenlocher
This is a "Thrift" limitation - if we don't do any further conversion. We have the opportunity to convert the string types to native type - at the expense of memory/performance. Should we do always do the conversion on the client driver? Or possibly allow an option to decide?

[CurtHagenlocher]

I think the correct thing is to do the conversion locally, yes. I think it would be very surprising for a user to run a query that produces a decimal or date value and to get that back as a string. I understand that we've been avoiding this for structured types where it's perhaps a little more justifiable but this just seems wrong.

One thing we could do to unblock this checkin is to add an option for converting scalar types and to fail if the value is not explicitly passed as false. This would let us add support in a subsequent change without breaking consumers.

[birschick-bq]

Introduced option data_type_conv with value none to indicate that no conversion should be done. Indicated future support for scalar to convert date, decimal and timestamp types.

[CurtHagenlocher]

Thanks for the changes! I do think we can't just return e.g. dates or decimals as strings, but I've made a suggestion for a small change that would unblock this change and let us fix that separately.

[CurtHagenlocher]

For .NET 6+ we could use e.g. BinaryPrimitives.WriteInt32LittleEndian for what I suspect is an efficiency gain. This could be done as a followup and/or performance-tested.

[birschick-bq]

Okay will create a follow-up to .NET 6+ support for BinaryPrimitives.WriteInt32LittleEndian

[CurtHagenlocher]

I think the correct thing is to do the conversion locally, yes. I think it would be very surprising for a user to run a query that produces a decimal or date value and to get that back as a string. I understand that we've been avoiding this for structured types where it's perhaps a little more justifiable but this just seems wrong.

One thing we could do to unblock this checkin is to add an option for converting scalar types and to fail if the value is not explicitly passed as false. This would let us add support in a subsequent change without breaking consumers.