Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Don't silently produce null values from invalid input to pl.datetime and pl.date #21013

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

mcrumiller
Copy link
Contributor

@mcrumiller mcrumiller commented Jan 30, 2025

Fixes #20977.

I am not sure if this is a breaking change or not.

Example of new behavior:

>>> import polars as pl
>>> pl.select(pl.date(2025, 13, 1))  # month 13 is invalid (pl.Date)
polars.exceptions.ComputeError: Invalid date components (2025, 13, 1) supplied

>>> pl.select(pl.datetime(2025, 13, 1))  # month 13 is invalid (pl.Datetime)
polars.exceptions.ComputeError: Invalid date components (2025, 13, 1) supplied


>>> pl.select(pl.datetime(2025, 1, 1, 25))  # hour 25 is invalid
polars.exceptions.ComputeError: Invalid time components (25, 0, 0, 0) supplied

@github-actions github-actions bot added fix Bug fix rust Related to Rust Polars labels Jan 30, 2025
Copy link

codecov bot commented Jan 30, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.27%. Comparing base (98ccb09) to head (1cdac2c).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #21013      +/-   ##
==========================================
+ Coverage   79.21%   79.27%   +0.06%     
==========================================
  Files        1583     1583              
  Lines      225545   225584      +39     
  Branches     2586     2586              
==========================================
+ Hits       178664   178842     +178     
+ Misses      46291    46152     -139     
  Partials      590      590              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mcrumiller mcrumiller marked this pull request as ready for review January 30, 2025 17:26
@ritchie46 ritchie46 changed the title fix(rust): Don't silently produce null values from invalid input to pl.datetime and pl.date fix: Don't silently produce null values from invalid input to pl.datetime and pl.date Feb 1, 2025
@github-actions github-actions bot added the python Related to Python Polars label Feb 1, 2025
Copy link
Member

@ritchie46 ritchie46 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We really should return an error here, not a panic.

@mcrumiller mcrumiller marked this pull request as draft February 1, 2025 14:59
@mcrumiller
Copy link
Contributor Author

mcrumiller commented Feb 1, 2025

@ritchie46 I wasn't quite sure how to go about this, so let me know if it needs rework. Right now, we have an iterator that yields Option<i32>s that are collected by the trusted len iterator. I could have returned Result<Option<i32>> for each item and updated the trusted len iterator to collect this into a Result<Int32Chunked> but this would require messing with the legacy module in polars-arrow which I didn't really want to do, since FromTrustedLenIterator doesn't collect Results.

Instead, I opted to flag if an invalid value is observed during the array creation, and proceed as we used to (filling with nulls). Then I check in hindsight if an invalid value was observed, and at that point throw a ComputeError. The downside is that the iterator doesn't halt immediately, but since the operation is failing anyway this doesn't seem that awful to me. Let me know if this sounds reasonable, and if not I can try to rework it some more.

@mcrumiller mcrumiller marked this pull request as ready for review February 1, 2025 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pl.datetime does not raise with argument values outside of the specified domain
2 participants