write_parquet encoding no longer recognized by PBI Service parquet connector after Polars 1.5.0 onwards

[darrylthom]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

df.write_parquet("data.parquet")

Log output

No response

Issue description

Just to explain the setup a bit:
Parquet gets written to a network drive. Report published to PBI Service connects to this parquet file using an on-premises gateway.

Refreshing works on local copy of PBI file, but through PBI Service specifically, it is now giving an error:
Data source error: {"error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","pbi.error":{"code":"DM_GWPipeline_Gateway_MashupataAccessError","parameters":{},,"details":[{"code":"DM_errorDetailNameCode_UnderlyingErrorCode","detail":{"type":1,"value":"-2147467259"}},{"code":"DM_ErrorDetailNameCode_UnderlyingErrorMessage","detail":{"type":1,"value":"Parquet: class parquet::ParquetException (message: 'Unknown encoding type.'"}}, {"code": "DM_ErrorDetailNameCode_UnderlyingHResult", "detail":{"type":1,"value":"-2147467259"}},"code":"Microsoft.Data.Mashup.ValueError.Reason","detail":{"type":1,"value":"DataFormat.Error"}}]"eceptionCulprit":1}}}

This refreshes fine locally -- the problem is PBI Service specifically. I tested generating my parquet files version to version from Polars 1.2 up until current, and I start getting these messages as of Polars 1.5.0's write_parquet specifically.

I believe something changed specifically in the write_parquet output that is causing it to no longer be compatible with the PBI Service's parquet connector in newer versions. I have analyzed the schema and the meta data and they are exactly the same in the old output versus new output.

Expected behavior

As nothing has changed in my schema or meta data, the files should be refreshing, but it seems like write_parquet's encoding is not recognized by PBI Service as of 1.5.0 onwards.

Installed versions

polars 1.5.0 to 1.7.1 (tested version by version)

[coastalwhite]

My guess is that this has to do with the Boolean Hybrid-RLE encoding.

[darrylthom]

My guess is that this has to do with the Boolean Hybrid-RLE encoding.

Yes, this is it exactly. When I drop my boolean columns from my parquet file in the latest Polars, PBI Service refreshes the file successfully.

[ritchie46]

It seems that the service only supports older parquet formats/encodings. For now you can circumvent the issue by writing via pyarrow which allows you to select different encodings.

This is something we could also support to a limited extend.

[darrylthom]

It seems that the service only supports older parquet formats/encodings. For now you can circumvent the issue by writing via pyarrow which allows you to select different encodings.

This is something we could also support to a limited extend.

Writing with pyarrow for the meantime worked. I tried creating an issue with the PowerBI team, but it got caught with their triaging vendor who was claiming it had to do with Polars and not Power BI so they wouldn't escalate it to the product team and recommended I downgrade instead.

[s1lvester]

I came across the same error. Here's how I worked around it:

import polars.selectors as cs
(
    df
    .with_columns(cs.boolean().cast(pl.Int8))
    .write_parquet("fname.parquet")
)```

[darrylthom]

I came across the same error. Here's how I worked around it:

import polars.selectors as cs
(
    df
    .with_columns(cs.boolean().cast(pl.Int8))
    .write_parquet("fname.parquet")
)```

I ended up using the following, as I wanted to keep booleans in their proper data type:

import pyarrow.parquet as pq
...
pq.write_table(df.to_arrow(), OUTPUT_PATH, use_dictionary=True) # use_dictionary I think is True by default and might not even be necessary for this to work on PBI (I did not test without it)

In fact, my file was compressing more than Polars when I tested writing compression='zstd' and compression_level=22 in both at the time.

Hopefully Polars adds a way to make itself compatible with PBI service when booleans are in the dataset in the future, as I almost guarantee based off of my awful experience dealing with their "support team" that they will not fix it on their end.

[cBiscuitSurprise]

This same issue breaks Amazon Redshift import from parquet files. Working around for now with pyarrow.