fix: Datetime parser was incorrectly parsing 8-digit fractional seconds when format specified to expect 9

[MarcoGorelli]

closes #22167

Exploring this further, as this nominally fixes the issue but isn't the end of the story done, much happier with the current fix

[codecov]

Codecov Report

Attention: Patch coverage is 71.42857% with 4 lines in your changes missing coverage. Please review.

Project coverage is 81.15%. Comparing base (19ae9ec) to head (4f51c2b).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
...es/polars-time/src/chunkedarray/string/strptime.rs	71.42%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #22180      +/-   ##
==========================================
- Coverage   81.16%   81.15%   -0.01%     
==========================================
  Files        1638     1638              
  Lines      235247   235253       +6     
  Branches     2714     2714              
==========================================
- Hits       190931   190924       -7     
- Misses      43676    43688      +12     
- Partials      640      641       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kdn36]

FWIW - I am not familiar with the history or detail, so consider the following observation with caution.

Observation: the proposed change (from .%9f to %.9f or even %.f) will make the subsequent .%6f and .%3f inaccessible for any exhaustive search that looks for the first match as it will match for any number of digits. To be confirmed. If true, dropping the 3f/6f patterns may present an optimization improvement, as long as some conditions are met (search logic, parsing only).

This is from looking into the performance overhead of datetime schema inference not too long ago, which is a very expensive effort in the current implementation. Something worth considering.

[MarcoGorelli]

Yup, thanks

The underlying issue is here: #22167 (comment)

looking into it to see what we can do

[MarcoGorelli]

is it ok to panic here? if we get here, it's a sign that something is seriously wrong, and it would've managed to avoid the bug pointed out in #22167 (comment)

In [21]: pl.Series(['2025-04-06T18:57:42.77756192Z']).str.to_datetime('%Y-%m-%dT%H:%M:%S.%9fcabbagebananapotato')
Out[21]:
shape: (1,)
Series: '' [datetime[ns]]
[
        2025-04-06 18:57:42.077756192
]

[ritchie46]

If it is an implementation error on our end, we can panic.

[MarcoGorelli]

looking into it to see what we can do

OK, resolved:

• StrpTimeState.parse was being reached when its preconditions didn't hold
• fixed fmt_len so that those StrpTimeState.parse wouldn't be reached in such an invalid state
• made StrpTimeState.parse panic if it's executed in an invalid state