Fix incorrect exception raised by Series[datetime64] + int #19147

jbrockmendel · 2018-01-09T04:32:23Z

Series[datetime64] +/- int is currently raising a ValueError when it should be raising a TypeError. This was introduced in #19024.

In the process of fixing this bug, this also goes most of the way towards fixing #19123 by raising on add/sub with integer arrays.

Setup:

dti = pd.date_range('2016-01-01', periods=2)
ser = pd.Series(dti)

ATM:

>>> ser + 1
[...]
ValueError: Cannot shift with no freq

After this PR (and also in 0.22.0 and anything before #19024):

>>> ser + 1
[...]
TypeError: incompatible type for a datetime/timedelta operation [__add__]

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

…me64]+int

jbrockmendel · 2018-01-09T04:32:51Z

pandas/core/ops.py

@@ -703,6 +702,23 @@ def wrapper(left, right, name=name, na_op=na_op):
    return wrapper


+def _dispatch_to_index_op(op, left, right, index_class):


We're going to end up using this again for TimedeltaIndex a few bugfixes from now.

you can de-privatize. rename to 'dispatch_to_index_operation'. needs a doc-string.

jreback · 2018-01-09T12:40:53Z

pandas/core/indexes/datetimelike.py

@@ -692,6 +693,9 @@ def __add__(self, other):
                return self._add_datelike(other)
            elif isinstance(other, Index):
                return self._add_datelike(other)
+            elif is_integer_dtype(other) and self.freq is None:
+                # GH#19123
+                raise NullFrequencyError("Cannot shift with no freq")


just make this a ValueError, we don't want to have custom error messages normally.

We need to catch this specifically in core.ops. Want to catch by checking the error message?

don't want to have custom error messages normally

The error message here "Cannot shift with no freq" is the same as the error message raised if adding just an integer when self.freq is None.

jreback · 2018-01-09T12:41:40Z

pandas/core/ops.py

@@ -703,6 +702,23 @@ def wrapper(left, right, name=name, na_op=na_op):
    return wrapper


+def _dispatch_to_index_op(op, left, right, index_class):


you can de-privatize. rename to 'dispatch_to_index_operation'. needs a doc-string.

jreback · 2018-01-09T12:42:28Z

pandas/core/ops.py

+    checking and timezone handling.
+    """
+    left_idx = index_class(left)
+    left_idx.freq = None  # avoid accidentally allowing integer add/sub


this is a mutating operation, don't do it

This is necessary. If I change this to an assert left_idx.freq is None that assertion fails in a number of tests:

dti = pd.date_range('1999-09-30', periods=10, tz='US/Pacific') ser = pd.Series(dti) >>> pd.DatetimeIndex(ser).freq <Day>

It looks like this only occurs for tzaware cases.

It looks like this only occurs for tzaware cases.

Possibly related to caching of DatetimeIndexes and the fact that Series[datetime64tz]._data.blocks[0].values is itself a DatetimeIndex?

jreback · 2018-01-09T12:43:13Z

pandas/tseries/offsets.py

@@ -1226,7 +1226,8 @@ def _get_roll(self, i, before_day_of_month, after_day_of_month):
        return roll

    def _apply_index_days(self, i, roll):
-        i += (roll % 2) * Timedelta(days=self.day_of_month).value
+        nanos = (roll % 2) * Timedelta(days=self.day_of_month).value


leftover from another PR (and I had commented on that), remove from here.

You’re rIght that this is duplicated, but it is correct in both places, and this PR breaks without it.

see other PR, this needs a doc-string

codecov · 2018-01-09T16:35:17Z

Codecov Report

Merging #19147 into master will decrease coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #19147      +/-   ##
==========================================
- Coverage   91.56%   91.53%   -0.03%     
==========================================
  Files         148      148              
  Lines       48856    48870      +14     
==========================================
+ Hits        44733    44735       +2     
- Misses       4123     4135      +12

Flag	Coverage Δ
#multiple	`89.91% <100%> (-0.03%)`	⬇️
#single	`41.69% <55.55%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/ops.py	`92.17% <100%> (+0.1%)`	⬆️
pandas/core/indexes/datetimelike.py	`97.11% <100%> (+0.03%)`	⬆️
pandas/errors/__init__.py	`100% <100%> (ø)`	⬆️
pandas/plotting/_converter.py	`65.22% <0%> (-1.74%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c271d4d...2231505. Read the comment docs.

…llfreq

jreback · 2018-01-11T00:19:17Z

pandas/tests/series/test_operators.py

@@ -680,6 +680,25 @@ def test_timedelta_series_ops(self):
        assert_series_equal(ts - s, expected2)
        assert_series_equal(ts + (-s), expected2)

+    def test_td64series_add_intlike(self):
+        # GH#19123


don't conjoin words

test_td64_series_and_integer_like

jreback · 2018-01-11T00:19:26Z

pandas/tests/series/test_operators.py

@@ -1428,6 +1441,25 @@ def test_dt64series_arith_overflow(self):
        res = dt - ser
        tm.assert_series_equal(res, -expected)

+    def test_dt64series_add_intlike(self):


jreback · 2018-01-11T00:19:51Z

pandas/tseries/offsets.py

@@ -1226,7 +1226,8 @@ def _get_roll(self, i, before_day_of_month, after_day_of_month):
        return roll

    def _apply_index_days(self, i, roll):
-        i += (roll % 2) * Timedelta(days=self.day_of_month).value
+        nanos = (roll % 2) * Timedelta(days=self.day_of_month).value


see other PR, this needs a doc-string

jreback · 2018-01-11T00:20:53Z

pandas/core/ops.py

+    left_idx.freq = None
+    try:
+        result = op(left_idx, right)
+    except NullFrequencyError:


ok this is fine then. Please update tests where this should be caught (e.g. .shift)

jreback · 2018-01-11T00:22:02Z

pandas/core/ops.py

+    # avoid accidentally allowing integer add/sub.  For datetime64[tz] dtypes,
+    # left_idx may inherit a freq from a cached DatetimeIndex.
+    # See discussion in GH#19147.
+    left_idx.freq = None


actually I don't like this mutation here. why is it necessary? (e.g. if one of the end-points is None, e.g. an integer series this will still raise, yes?)

Because if left is tzaware then pd.DatetimeIndex(left, freq=None) may come back with non-None freq.

dti = pd.date_range('2016-01-01', periods=2, tz='US/Pacific') ser = pd.Series(dti) >>> pd.DatetimeIndex(ser, freq=None).freq <Day>

and why is this a problem?

Because in this scenario, without the left_idx.freq = None line, ser + 1 fails to raise.

Just parametrized test_dt64_series_add_intlike to make sure the tzaware case is covered.

so you didn't answer my question here. I don't want mutation.

I really, really did. Without setting the frequency to none, integer addition fails to raise.

then we need a better way to detect this, we cannot mutate things to just see if they raise

…llfreq

jbrockmendel · 2018-01-12T01:27:05Z

Re-push or let sit?

jreback · 2018-01-12T11:47:36Z

needs a rebase

…llfreq

jreback · 2018-01-13T18:23:22Z

pandas/tests/indexes/datetimes/test_ops.py

@@ -593,6 +594,12 @@ def test_nat_new(self):
        exp = np.array([tslib.iNaT] * 5, dtype=np.int64)
        tm.assert_numpy_array_equal(result, exp)



there are very likely tests that are raising ValueError that should now be NullFrequencyError, pls change them.

You can find them very easily, by temporarly changing the error where NullFrequencyInhertis to something else (e.g. KeyError) and then seeing what raises

Found and changed one.

…llfreq

jreback · 2018-01-14T22:15:27Z

pandas/core/ops.py

+    # avoid accidentally allowing integer add/sub.  For datetime64[tz] dtypes,
+    # left_idx may inherit a freq from a cached DatetimeIndex.
+    # See discussion in GH#19147.
+    left_idx.freq = None


then we need a better way to detect this, we cannot mutate things to just see if they raise

jreback · 2018-01-14T22:15:50Z

pandas/errors/__init__.py

+
+
+class NullFrequencyError(ValueError):
+    """


this needs a line in the api changes section

jreback · 2018-01-14T22:15:58Z

pandas/tests/indexes/datetimes/test_ops.py

@@ -593,6 +594,12 @@ def test_nat_new(self):
        exp = np.array([tslib.iNaT] * 5, dtype=np.int64)
        tm.assert_numpy_array_equal(result, exp)



…llfreq

jreback · 2018-01-16T00:04:00Z

needs a rebase, otherwise lgtm. ping on green.

jreback · 2018-01-16T00:04:50Z

doc/source/whatsnew/v0.23.0.txt

@@ -273,6 +273,7 @@ Other API Changes
 - The default ``Timedelta`` constructor now accepts an ``ISO 8601 Duration`` string as an argument (:issue:`19040`)
 - ``IntervalDtype`` now returns ``True`` when compared against ``'interval'`` regardless of subtype, and ``IntervalDtype.name`` now returns ``'interval'`` regardless of subtype (:issue:`18980`)
 - :func:`Series.to_csv` now accepts a ``compression`` argument that works in the same way as the ``compression`` argument in :func:`DataFrame.to_csv` (:issue:`18958`)
+- :func:`DatetimeIndex.shift` and :func:`TimedeltaIndex.shift` will now raise ``NullFrequencyError`` (which subclasses ``ValueError``) when the index object frequency is ``None`` (:issue:`19147`)


say something like: which subclasses ValueError, the currently raised exception

…llfreq

jbrockmendel · 2018-01-16T16:32:17Z

Ping

jbrockmendel · 2018-01-16T19:09:55Z

One more bugfix is ready to go once this goes in, plus another PR getting rid of _Op and _TimeOp.

jreback · 2018-01-17T00:14:08Z

thanks!

Implement NullFrequencyError to fix exception raised by Series[dateti…

1241994

…me64]+int

jbrockmendel commented Jan 9, 2018

View reviewed changes

jreback requested changes Jan 9, 2018

View reviewed changes

jreback added Datetime Datetime data dtype Compat pandas objects compatability with Numpy or Python functions labels Jan 9, 2018

jbrockmendel changed the title ~~Fix incorrect exception raises by Series[datetime64] + int~~ Fix incorrect exception raised by Series[datetime64] + int Jan 9, 2018

deprivatize, docstring, comment

1fe5732

jbrockmendel added 3 commits January 9, 2018 16:59

Merge branch 'master' of https://github.com/pandas-dev/pandas into nu…

0aee07a

…llfreq

Merge branch 'master' of https://github.com/pandas-dev/pandas into nu…

29a4931

…llfreq

Flake8 whitespace fixup

4fd68c1

jreback requested changes Jan 11, 2018

View reviewed changes

jbrockmendel added 3 commits January 10, 2018 17:42

Merge branch 'master' of https://github.com/pandas-dev/pandas into nu…

efcde8e

…llfreq

docstring, tests for NullFrequencyError in shift

df545a0

dt64_series +/- int test for tzaware

1a03a68

jbrockmendel added 2 commits January 12, 2018 08:47

Merge branch 'master' of https://github.com/pandas-dev/pandas into nu…

878e689

…llfreq

fix merge screwup

fd0ac99

jreback requested changes Jan 13, 2018

View reviewed changes

Merge branch 'master' of https://github.com/pandas-dev/pandas into nu…

a623d00

…llfreq

jreback requested changes Jan 14, 2018

View reviewed changes

jbrockmendel added 3 commits January 14, 2018 18:36

shallow_copy instead of changing inplace

aab851d

Merge branch 'master' of https://github.com/pandas-dev/pandas into nu…

66d6bce

…llfreq

whatsnew api note

15b5f08

jreback added this to the 0.23.0 milestone Jan 16, 2018

jreback approved these changes Jan 16, 2018

View reviewed changes

jreback reviewed Jan 16, 2018

View reviewed changes

jbrockmendel added 3 commits January 15, 2018 16:37

suggested edit to whatsnew note

63ae039

Merge branch 'master' of https://github.com/pandas-dev/pandas into nu…

edebbe2

…llfreq

Merge branch 'master' of https://github.com/pandas-dev/pandas into nu…

2231505

…llfreq

jbrockmendel mentioned this pull request Jan 16, 2018

CLN: Remove unused core.internals methods #19250

Merged

4 tasks

jreback merged commit 3db6f66 into pandas-dev:master Jan 17, 2018

jbrockmendel deleted the nullfreq branch February 11, 2018 21:59

		@@ -703,6 +702,23 @@ def wrapper(left, right, name=name, na_op=na_op):
		return wrapper


		def _dispatch_to_index_op(op, left, right, index_class):

		@@ -593,6 +594,12 @@ def test_nat_new(self):
		exp = np.array([tslib.iNaT] * 5, dtype=np.int64)
		tm.assert_numpy_array_equal(result, exp)

Fix incorrect exception raised by Series[datetime64] + int #19147

Fix incorrect exception raised by Series[datetime64] + int #19147

Conversation

jbrockmendel commented Jan 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 9, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Jan 12, 2018

jreback commented Jan 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jan 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Jan 16, 2018

jbrockmendel commented Jan 16, 2018

jreback commented Jan 17, 2018

codecov bot commented Jan 9, 2018 •

edited

Loading