Datetime parsing (PDEP-4): allow mixture of ISO formatted strings #50939

MarcoGorelli · 2023-01-23T11:30:23Z

closes Datetime parsing (PDEP-4): allow mixture of ISO formatted strings? #50411 (Replace xxxx with the GitHub issue number)
closes API allow for format='mixed' in to_datetime #50972
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

doc/source/user_guide/io.rst

pandas/_libs/tslibs/strptime.pyx

mroeschke

How does format="ISO8601" interact with exact=True/False?

MarcoGorelli · 2023-01-23T18:16:58Z

It's ignored - good point, I'll make sure that's in the docstring

jorisvandenbossche · 2023-01-24T09:06:40Z

How does format="ISO8601" interact with exact=True/False?

It's ignored - good point, I'll make sure that's in the docstring

Could we raise an error instead? (since the default is OK, it's only when the user explicitly passes exact=False that this can occur, so an error if you pass both format="ISO8601", exact=False seems fine?)

jorisvandenbossche · 2023-01-24T09:16:45Z

One question that is not directly clear to me from the doc update (without diving into the code): you can pass format="ISO8601" explicitly, but do we also "infer" this format? (i.e. if the inferred format is a subset of ISO8601, use the flexible one as inferred format)

I assume not (and we probably also wouldn't want to do that, I think? Although it would be more backwards compatible), but I was wondering if we can make this clearer for the cases that would benefit from this (like pd.to_datetime(["2022-01-01T09:00:00", "2022-01-02T09:00:00.123"])) that work fine on pandas < 2, but error now.
Right now, that case gives a "... doesn't match format ..." error message. Ideally we could point people to this option. In theory we could check if the inferred format is ISO-like and infer the format from the non-matching string, and if that is also ISO like, append the error message. But that's maybe making it a bit too complicated not worth this corner case ;)

jorisvandenbossche · 2023-01-24T09:22:12Z

pandas/core/tools/datetimes.py

@@ -759,13 +766,15 @@ def to_datetime(
        <https://docs.python.org/3/library/datetime.html
        #strftime-and-strptime-behavior>`_ for more information on choices, though
        note that :const:`"%f"` will parse all the way up to nanoseconds.
+        You can also pass "ISO8601" to parse any ISO8601 time string.


Add a link that explains ISO8601? (eg https://en.wikipedia.org/wiki/ISO_8601)

Although looking at that link makes me wonder if we then should be explicit about that this is only the calendar date + time formatting (and not week dates or ordinal dates)

MarcoGorelli · 2023-01-24T18:21:27Z

but do we also "infer" this format? (i.e. if the inferred format is a subset of ISO8601, use the flexible one as inferred format)
I assume not

I wasn't planning on doing that, as it would strike me as value-dependent behaviour

Good point on the error message though, probably worth putting extra effort in to making sure that's as clear and actionable as possible

mroeschke

LGTM

MarcoGorelli · 2023-02-06T21:34:04Z

LGTM

Thanks for your review!

@jorisvandenbossche any objections?

mroeschke · 2023-02-08T21:26:24Z

Does this close #50972 as well?

MarcoGorelli · 2023-02-08T21:28:54Z

yup, thanks!

jbrockmendel · 2023-02-09T16:11:54Z

pandas/_libs/tslibs/strptime.pyx

@@ -186,6 +186,7 @@ def array_strptime(
        bint iso_format = format_is_iso(fmt)
        NPY_DATETIMEUNIT out_bestunit
        int out_local = 0, out_tzoffset = 0
+        bint string_to_dts_succeeded = 0


any particular reason this is changed from failed? doesn't really matter to me, just curious

it just simplifies the logic

jbrockmendel · 2023-02-09T16:14:37Z

pandas/tests/tools/test_to_datetime.py

@@ -2086,7 +2108,7 @@ def test_dataframe_coerce(self, cache):

        msg = (
            r'^cannot assemble the datetimes: time data ".+" doesn\'t '
-            r'match format "%Y%m%d", at position 1$'
+            r'match format "%Y%m%d", at position 1.'


NBD but should the period be escaped here (and below)?

jbrockmendel · 2023-02-09T16:15:17Z

Couple of small questions, none of them blockers. LGTM.

mroeschke · 2023-02-09T22:55:45Z

Looks like a merge conflict then we can merge

MarcoGorelli · 2023-02-10T08:55:44Z

Looks like a merge conflict then we can merge

resolved, thanks for your reviews!

jorisvandenbossche · 2023-02-10T15:02:07Z

doc/source/whatsnew/v2.0.0.rst

-If you still need to parse dates with inconsistent formats, you'll need to apply :func:`to_datetime`
-to each element individually, e.g. ::
+If you still need to parse dates with inconsistent formats, you can use
+``format='mixed`` (preferably alongside ``dayfirst``) ::


Why is this "preferably", it depends on your data?

jorisvandenbossche · 2023-02-10T15:06:39Z

pandas/_libs/tslibs/strptime.pyx

+                f"{str(ex)}, at position {i}. You might want to try:\n"
+                "    - passing ``format='ISO8601'`` if your strings are "
+                "all ISO8601 but not necessarily in exactly the same format;\n"
+                "    - passing ``format='mixed'``, and the format will be "
+                "inferred for each element individually. "
+                "You might want to use ``dayfirst`` alongside this.",


We might still want to mention the suggestion to pass a specific format first?

For example, if an error occurs because the format is ambiguous based on the first value (dayfirst/daylast), and was inferred incorrectly:

pd.to_datetime(["12/01/2012", "13/01/2012"]) ... ValueError: time data "13/01/2012" doesn't match format "%m/%d/%Y", at position 1. You might want to try: - passing ``format='ISO8601'`` if your strings are all ISO8601 but not necessarily in exactly the same format; - passing ``format='mixed'``, and the format will be inferred for each element individually. You might want to use ``dayfirst`` alongside this.

Here, I think the best is still to provide a manual format="%d/%m/%Y" (or to specify dayfirst=True, then the inference would work for this data)

(also, a fomatting nit: since this is plain text (and not rst that will be rendered), no need for double backticks, I would use single backticks which is a bit less visually distracting)

good point, thanks!

jorisvandenbossche · 2023-02-10T15:09:07Z

pandas/core/tools/datetimes.py

+        - "ISO8601", to parse any `ISO8601 <https://en.wikipedia.org/wiki/ISO_8601>`_
+          time string (not necessarily in exactly the same format);
+        - "mixed", to infer the format for each element individually. This is risky,
+          and you should probably use it along with `dayfirst`.


It's true that in that case that is best, but the typicaly caveat of that this dayfirst=True/False is not strict still applies, right?

true, but it would solve the typical example at least

In [2]: pd.to_datetime(['12-01-2000 00:00:00', '13-01-2000 00:00:00'], format='mixed', dayfirst=True) Out[2]: DatetimeIndex(['2000-01-12', '2000-01-13'], dtype='datetime64[ns]', freq=None)

Yes, I was just wondering if it would be useful to still call it out here explicitly (but it's quite explicit in the dayfirst entry in the docstring)

jorisvandenbossche · 2023-02-10T15:11:26Z

web/pandas/pdeps/0004-consistent-to-datetime-parsing.md

@@ -64,6 +64,11 @@ Out[3]:
 1   2000-01-13
 dtype: datetime64[ns]
 ```
+or, if their dates are all ISO8601,


You can update the one above to use format="mixed" ?

I missed this one - thanks!

mroeschke · 2023-02-13T21:37:11Z

@MarcoGorelli merged in main once more and then I think we are good to merge. If there are any outstanding issues they can be addressed during the rc period

MarcoGorelli · 2023-02-13T21:38:30Z

sounds good, thanks!

MarcoGorelli · 2023-02-14T10:44:21Z

merging then so this doesn't block the RC, but if there's further requests please do let me know and I'll address them promptly

jorisvandenbossche · 2023-02-15T00:05:45Z

Thanks @MarcoGorelli !

MarcoGorelli added 4 commits January 23, 2023 11:08

allow format iso8601

044948f

fixup tests

f4e1392

🏷️ typing

9f06d80

remove duplicate code

d7f6056

MarcoGorelli marked this pull request as ready for review January 23, 2023 13:19

MarcoGorelli requested review from jbrockmendel, jorisvandenbossche and mroeschke January 23, 2023 13:20

MarcoGorelli added the Datetime Datetime data dtype label Jan 23, 2023

MarcoGorelli added 2 commits January 23, 2023 15:12

Merge remote-tracking branch 'upstream/main' into allow-mixed-iso

8952a0e

improve message, use if-statement

6e6d579

jbrockmendel reviewed Jan 23, 2023

View reviewed changes

doc/source/user_guide/io.rst Outdated Show resolved Hide resolved

jbrockmendel reviewed Jan 23, 2023

View reviewed changes

pandas/_libs/tslibs/strptime.pyx Outdated Show resolved Hide resolved

MarcoGorelli added 2 commits January 23, 2023 17:40

Merge remote-tracking branch 'upstream/main' into allow-mixed-iso

3d65dbf

note that exact has no effect if format=iso8601

b247bbd

mroeschke reviewed Jan 23, 2023

View reviewed changes

jorisvandenbossche reviewed Jan 24, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/main' into allow-mixed-iso

2f66f87

MarcoGorelli marked this pull request as draft January 24, 2023 18:21

MarcoGorelli added 2 commits January 25, 2023 10:12

Merge remote-tracking branch 'upstream/main' into allow-mixed-iso

eb36d8c

point to format=ISO8601 in error message

262be89

MarcoGorelli marked this pull request as ready for review January 25, 2023 10:45

MarcoGorelli marked this pull request as draft January 25, 2023 19:11

MarcoGorelli mentioned this pull request Jan 25, 2023

API allow for format='mixed' in to_datetime #50972

Closed

allow format="mixed"

e01b6ee

MarcoGorelli marked this pull request as ready for review January 25, 2023 21:06

MarcoGorelli and others added 2 commits February 2, 2023 09:15

Merge remote-tracking branch 'upstream/main' into allow-mixed-iso

531e0e8

Merge branch 'main' into allow-mixed-iso

5582882

MarcoGorelli requested a review from mroeschke February 6, 2023 20:50

mroeschke approved these changes Feb 6, 2023

View reviewed changes

mroeschke added this to the 2.0 milestone Feb 8, 2023

jbrockmendel reviewed Feb 9, 2023

View reviewed changes

jbrockmendel approved these changes Feb 9, 2023

View reviewed changes

Merge branch 'main' into allow-mixed-iso

57b922c

MarcoGorelli and others added 4 commits February 10, 2023 09:04

minor fixups

313003e

Merge branch 'main' into allow-mixed-iso

2ede506

double backticks -> single, suggest passing format

3b61e5b

use format=mixed instead of apply in example;

acd44ae

jorisvandenbossche reviewed Feb 10, 2023

View reviewed changes

mroeschke mentioned this pull request Feb 10, 2023

RLS: 2.0 #46776

Closed

1 task

MarcoGorelli requested a review from jorisvandenbossche February 13, 2023 13:24

Merge branch 'main' into allow-mixed-iso

ba6393f

MarcoGorelli merged commit d1095bc into pandas-dev:main Feb 14, 2023

Vincent-Maladiere mentioned this pull request Oct 13, 2023

MAIN Improve DatetimeEncoder skrub-data/skrub#784

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Datetime parsing (PDEP-4): allow mixture of ISO formatted strings #50939

Datetime parsing (PDEP-4): allow mixture of ISO formatted strings #50939

MarcoGorelli commented Jan 23, 2023 •

edited

Loading

mroeschke left a comment

MarcoGorelli commented Jan 23, 2023

jorisvandenbossche commented Jan 24, 2023

jorisvandenbossche commented Jan 24, 2023

jorisvandenbossche Jan 24, 2023

MarcoGorelli commented Jan 24, 2023

mroeschke left a comment

MarcoGorelli commented Feb 6, 2023

mroeschke commented Feb 8, 2023

MarcoGorelli commented Feb 8, 2023

jbrockmendel Feb 9, 2023

MarcoGorelli Feb 9, 2023

jbrockmendel Feb 9, 2023

jbrockmendel commented Feb 9, 2023

mroeschke commented Feb 9, 2023

MarcoGorelli commented Feb 10, 2023

jorisvandenbossche Feb 10, 2023

jorisvandenbossche Feb 10, 2023

jorisvandenbossche Feb 10, 2023

MarcoGorelli Feb 10, 2023

jorisvandenbossche Feb 10, 2023

MarcoGorelli Feb 10, 2023 •

edited

Loading

jorisvandenbossche Feb 10, 2023

jorisvandenbossche Feb 10, 2023

MarcoGorelli Feb 10, 2023

mroeschke commented Feb 13, 2023

MarcoGorelli commented Feb 13, 2023

MarcoGorelli commented Feb 14, 2023

jorisvandenbossche commented Feb 15, 2023

Datetime parsing (PDEP-4): allow mixture of ISO formatted strings #50939

Datetime parsing (PDEP-4): allow mixture of ISO formatted strings #50939

Conversation

MarcoGorelli commented Jan 23, 2023 • edited Loading

mroeschke left a comment

Choose a reason for hiding this comment

MarcoGorelli commented Jan 23, 2023

jorisvandenbossche commented Jan 24, 2023

jorisvandenbossche commented Jan 24, 2023

Choose a reason for hiding this comment

MarcoGorelli commented Jan 24, 2023

mroeschke left a comment

Choose a reason for hiding this comment

MarcoGorelli commented Feb 6, 2023

mroeschke commented Feb 8, 2023

MarcoGorelli commented Feb 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Feb 9, 2023

mroeschke commented Feb 9, 2023

MarcoGorelli commented Feb 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoGorelli Feb 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mroeschke commented Feb 13, 2023

MarcoGorelli commented Feb 13, 2023

MarcoGorelli commented Feb 14, 2023

jorisvandenbossche commented Feb 15, 2023

MarcoGorelli commented Jan 23, 2023 •

edited

Loading

MarcoGorelli Feb 10, 2023 •

edited

Loading