Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for pipes in read_csv #17614

Open
2 tasks done
raayu83 opened this issue Jul 13, 2024 · 1 comment
Open
2 tasks done

Support for pipes in read_csv #17614

raayu83 opened this issue Jul 13, 2024 · 1 comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars

Comments

@raayu83
Copy link

raayu83 commented Jul 13, 2024

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import os
import pandas as pd
import polars as pl

r_fd, w_fd = os.pipe()

r_file = os.fdopen(r_fd, 'rb', buffering=0)
w_file = os.fdopen(w_fd, 'wb', buffering=0)

df = pl.DataFrame({'Hello': 'World'})

with w_file as f:
    df.write_csv(w_file)

with r_file as f:
    df = pl.read_csv(f)
    #df = pd.read_csv(f) # works fine in pandas
    print(df)

Log output

Traceback (most recent call last):
  File "/Users/klst/Library/Application Support/JetBrains/PyCharmCE2024.1/scratches/scratch_1.py", line 16, in <module>
    df = pl.read_csv(f)
         ^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/polars/io/csv/functions.py", line 418, in read_csv
    df = _read_csv_impl(
         ^^^^^^^^^^^^^^^
  File "/Users/klst/Library/Caches/pypoetry/virtualenvs/pyexasol-cQLUeQM1-py3.12/lib/python3.12/site-packages/polars/io/csv/functions.py", line 564, in _read_csv_impl
    pydf = PyDataFrame.read_csv(
           ^^^^^^^^^^^^^^^^^^^^^
OSError: Invalid argument (os error 22)

Issue description

Unlike pandas, Polars does currently not support pipes in read_csv.
This makes integration with libraries that use pipes as a base for exchanging data (e.g. pyexasol, the Exasol database driver library) impossible.
In other words, fixing this bug would help increasing adoption of polars

Expected behavior

Polars should also support pipes.

Installed versions

--------Version info---------
Polars:               1.1.0
Index type:           UInt32
Platform:             macOS-14.5-arm64-arm-64bit
Python:               3.12.1 (main, Jan  5 2024, 19:05:58) [Clang 15.0.0 (clang-1500.1.0.2.5)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
great_tables:         <not installed>
hvplot:               <not installed>
matplotlib:           <not installed>
nest_asyncio:         <not installed>
numpy:                1.26.4
openpyxl:             <not installed>
pandas:               2.2.2
pyarrow:              <not installed>
pydantic:             <not installed>
pyiceberg:            <not installed>
sqlalchemy:           <not installed>
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>
@raayu83 raayu83 added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Jul 13, 2024
@deanm0000 deanm0000 added enhancement New feature or an improvement of an existing feature and removed bug Something isn't working needs triage Awaiting prioritization by a maintainer labels Jul 19, 2024
@ThomasRebeleKeebo
Copy link

It seems that quite some effort would be required to implement this enhancement. A good starting point might be crates/polars-plan/src/plans/conversion/scans.rs line 179.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars
Projects
None yet
Development

No branches or pull requests

3 participants