chore(rust): make Arrow version selection more flexible

[felipecrv]

The driver manager doesn't have to necessarily require the same version that the datafusion driver requires (due to datafusion's specific arrow version requirements).

Fixes #2524

[lidavidm]

Hmm, should we set up CI to make sure things (minus Datafusion) build with a few versions of the Rust Arrow crate?

[mbrobbel]

Testing this in CI won't work when the datafusion driver is in the workspace (because it locks the arrow version in the workspace).

[felipecrv]

Testing this in CI won't work when the datafusion driver is in the workspace (because it locks the arrow version in the workspace).

That's what I suspected. It's a tricky issue. We could discuss on the issue page the best way to decouple adbc_core from driver/datafusion.

[mbrobbel]

Testing this in CI won't work when the datafusion driver is in the workspace (because it locks the arrow version in the workspace).

That's what I suspected. It's a tricky issue. We could discuss on the issue page the best way to decouple adbc_core from driver/datafusion.

We could maybe move the datafusion driver to the datafusion repo or use two workspaces in this repo (one with the core crate and the other with the drivers)?

[felipecrv]

The 2-workspace solution sounds good. Moving to datafusion sounds even better.

Does Rust tooling assume one-repo==one-workspace anywhere?

[mbrobbel]

Moving to datafusion sounds even better.

+1

Does Rust tooling assume one-repo==one-workspace anywhere?

No.