Support DBFS?

[wjones127]

Description

Currently, it looks like we error with:

/local_disk0/.ephemeral_nfs/envs/pythonEnv-ff27c9d1-006a-4c3f-9028-6a49bdc901e2/lib/python3.9/site-packages/deltalake/writer.py in write_deltalake(table_or_uri, data, schema, partition_by, filesystem, mode, file_options, max_open_files, max_rows_per_file, min_rows_per_group, max_rows_per_group, name, description, configuration, overwrite_schema, storage_options)
    267 
    268     if table is None:
--> 269         _write_new_deltalake(
    270             table_uri,
    271             schema,

PyDeltaTableError: Failed to read delta log object: Generic DeltaLocalObjectStore error: Function not implemented (os error 38)

I suspect this is because DBFS does not support replace_if_not_exist. So I think we’d only be able to support it if we added an option to use non-atomic commits (you would lose support for concurrent writers).

The alternative is we tell people to connect to object stores directly. We should provide clear instructions on that.

Use Case

Related Issue(s)

[christophergrant]

The use case for this would be to write to Delta Lake tables on DBFS when a Spark Context is not available. This is rare, but it does happen.

[rtyler]

There's a more recent duplicate #1376 which I am going to close this in favor of