Empty strings in CSV files aren't being interpreted as null when using a Dictionary(_, Utf8)

[rumpuslabs]

Describe the bug

Related to #7797

Empty strings in CSV files aren't being interpreted as null when using a Dictionary(_, Utf8)

To Reproduce

Create a simple input.csv file like this:

id,name
1,
2,bob

Run the following code:

#[tokio::main]
async fn main() -> Result<(), DataFusionError> {
    let ctx = SessionContext::new();

    let format = CsvFormat::default();
    let listing_options = ListingOptions::new(Arc::new(format));
    ctx.register_listing_table(
        "input",
        "input.csv",
        listing_options.clone(),
        Some(Arc::new(Schema::new(vec![
            Field::new("id", DataType::Utf8, false),
            Field::new(
                "name",
                DataType::Dictionary(Box::new(DataType::UInt8), Box::new(DataType::Utf8)),
                true,
            ),
        ]))),
        None,
    )
    .await?;

    let results = ctx
        .table("input")
        .await?
        .filter(col("name").is_not_null())?
        .collect()
        .await?;

    let pretty_results = arrow::util::pretty::pretty_format_batches(&results)?.to_string();

    println!("{}", pretty_results);

    Ok(())
}

Expected behavior

I was expecting the output to look like this:

+----+------+
| id | name |
+----+------+
| 2  | bob  |
+----+------+

But the full dataset is returned instead:

+----+------+
| id | name |
+----+------+
| 1  |      |
| 2  | bob  |
+----+------+

Additional context

Tested on v41.0.0

Replace DataType::Dictionary(Box::new(DataType::UInt8), Box::new(DataType::Utf8)) with DataType::Utf8 and it works.

[edmondop]

take

[edmondop]

@alamb shouldn't the csv reader also throw an error because "bob" is not a valid dictionary?

[alamb]

I agree the discrepancy between UTf8 and Dictionary looks like a bug

@alamb shouldn't the csv reader also throw an error because "bob" is not a valid dictionary?

I think "bob" is a valid value for a DictionaryArray (whose values are Strings)

[edmondop]

@alamb before I file an issue to arrow-csv, why is "bob" a valid value for DictionaryArray? don't you need a key and a value for a dictionary?

[alamb]

@alamb before I file an issue to arrow-csv, why is "bob" a valid value for DictionaryArray? don't you need a key and a value for a dictionary?

I think a more precise version of this would be "bob" is a valid value for DictionaryArray(Int32, Utf8) -- that is if the dictionary's value type is strings (Utf8) then the dictionary should be able to hold string values

[edmondop]

@alamb I have a reproducer on Arrow CSV, see this https://github.com/apache/arrow-rs/compare/main...edmondop:arrow-rs:datafusion-12041?expand=1

However, I don't know if we should change the behavior of the Arrow CSV reader to return null for empty nullable dictionaries, I filed apache/arrow-rs#6821 to track the discussion

[edmondop]

Pending apache/arrow-rs#6830 merge and release in upstream arrow-rs dependency