Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: add datetimelike tests for tz-aware DatetimeIndex #17694

Closed
wants to merge 1 commit into from
Closed

TST: add datetimelike tests for tz-aware DatetimeIndex #17694

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion pandas/tests/indexes/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,12 @@ def verify_pickle(self, index):
unpickled = tm.round_trip_pickle(index)
assert index.equals(unpickled)

def _fix_tz(self, new_index, orig_index):
if hasattr(orig_index, 'tz'):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really don't spell things like this. rather change create_index() to work properly.

assert new_index.tz is None
new_index = new_index.tz_localize('UTC').tz_convert(orig_index.tz)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

create a new class DatetimeTZ in test_datetimelike.py which also inherits Datetimelike. Then just change its create_index

Copy link
Author

@azjps azjps Sep 28, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, this commit already introduces a new Datetimelike-derived class in test_datetimelikely.py with create_index() overriden to return a tz-aware DatetimeIndex: test_datetimelike.py#L159. This _fix_tz hack here is not to create the underlying index to be tested, but to "fix" various transformations of the index which don't preserve TZ, to get them to compare correctly back to the output of create_index(). But like you mentioned in the comment below, we can move more of this logic to class members instead.

return new_index

def test_pickle_compat_construction(self):
# this is testing for pickle compat
if self._holder is None:
Expand Down Expand Up @@ -278,6 +284,8 @@ def test_ensure_copied_data(self):

index_type = index.__class__
result = index_type(index.values, copy=True, **init_kwargs)
result = self._fix_tz(result, index)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need to touch this because _holder will also be different.

We shold prob refactor this init_kwargs bizness to instead be a class method, that way each class can construct things easily

IOW

class .....:

     # e.g. for PeriodIndex
     _init_kwargs = lambda index: return {'freq': index.freq'}

     # in the base class (no-op)
     _init_kwargs = lambda index: return {}

Copy link
Author

@azjps azjps Sep 28, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_holder here will also be DatetimeIndex, which isn't sufficient because DatetimeIndex constructor doesn't have TZ information. The problem is that the constructor DatetimeIndex(index.values, ...) is getting a numpy datetime64 array of tz-naive timestamps. There currently doesn't seem to be an argument to DatetimeIndex's constructor to tell it what timezone to convert to presuming that the input data is UTC (the tz= argument localizes to the specified timezone), so there's not currently a way to do this through init_kwargs. Hence why I currently call _fix_tz to first localize to UTC and then convert to the TZ of create_index().

To sum up:

  • index = create_index() has TZ = US/Eastern
  • result = index_type(index.values) has TZ = None ❌
  • result = index_type(index.values, tz=index.tz) has TZ = US/Eastern, but because of re-localization has been shifted again, and thus has differently underlying values. ❌
  • self._fix_tz(result, index) does have TZ = US/Eastern

I can do the suggested refactor though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes prob need to refactor a bit,


tm.assert_index_equal(index, result)
tm.assert_numpy_array_equal(index.values, result.values,
check_same='copy')
Expand Down Expand Up @@ -501,11 +509,13 @@ def test_repeat(self):
rep = 2
i = self.create_index()
expected = pd.Index(i.values.repeat(rep), name=i.name)
expected = self._fix_tz(expected, i)
tm.assert_index_equal(i.repeat(rep), expected)

i = self.create_index()
rep = np.arange(len(i))
expected = pd.Index(i.values.repeat(rep), name=i.name)
expected = self._fix_tz(expected, i)
tm.assert_index_equal(i.repeat(rep), expected)

def test_numpy_repeat(self):
Expand Down Expand Up @@ -726,7 +736,9 @@ def test_equals(self):
assert not idx.equals(np.array(idx))

# Cannot pass in non-int64 dtype to RangeIndex
if not isinstance(idx, RangeIndex):
# In tz-aware DatetimeIndex constructor,
# subarr.tz: ValueError: cannot localize from non-UTC data
if not isinstance(idx, RangeIndex) and not hasattr(idx, 'tz'):
same_values = Index(idx, dtype=object)
assert idx.equals(same_values)
assert same_values.equals(idx)
Expand Down
2 changes: 1 addition & 1 deletion pandas/tests/indexes/datetimelike.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ def test_str(self):

if hasattr(idx, 'tz'):
if idx.tz is not None:
assert idx.tz in str(idx)
assert str(idx.tz) in str(idx)
if hasattr(idx, 'freq'):
assert "freq='%s'" % idx.freqstr in str(idx)

Expand Down
82 changes: 82 additions & 0 deletions pandas/tests/indexes/datetimes/test_datetimelike.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,3 +76,85 @@ def test_union(self):
for case in cases:
result = first.union(case)
assert tm.equalContents(result, everything)


class TestDatetimeTZIndex(DatetimeLike):
_holder = DatetimeIndex

def setup_method(self, method):
# TODO: Consider adding another test instance that crosses
# DST boundary? Currently such fails a lot of tests.
tz = "US/Eastern"
self.indices = dict(index=tm.makeDateIndex(10).tz_localize(tz),
index_dec=date_range('20130101', periods=10,
freq='-1D').tz_localize(tz))
self.setup_indices()

def test_pickle_compat_construction(self):
pass

def test_repr_roundtrip(self):
# idx = self.create_index()
# tm.assert_index_equal(eval(repr(idx)), idx)

# The constructor is re-localizing to the dtype's TZ:
# Index values are different (100.0 %)
# [left]: DatetimeIndex(['2013-01-01 05:00:00-05:00',
# '2013-01-02 05:00:00-05:00',
# '2013-01-03 05:00:00-05:00',
# '2013-01-04 05:00:00-05:00',
# '2013-01-05 05:00:00-05:00'],
# dtype='datetime64[ns, US/Eastern]', freq='D')
# [right]: DatetimeIndex(['2013-01-01 00:00:00-05:00',
# '2013-01-02 00:00:00-05:00',
# '2013-01-03 00:00:00-05:00',
# '2013-01-04 00:00:00-05:00',
# '2013-01-05 00:00:00-05:00'],
# dtype='datetime64[ns, US/Eastern]', freq='D')
pass

# Disable following tests currently because they test comparing values
# with numpy arrays etc which are tz-naive, leading to issues.
def test_intersection_base(self):
# Subset of the full test
for name, idx in pd.compat.iteritems(self.indices):
first = idx[:5]
second = idx[:3]
intersect = first.intersection(second)

assert tm.equalContents(intersect, second)
# TODO: intersection with numpy array doesn't place nice
# because of the mixture of tz-naive and tz-aware timestamps

def test_union_base(self):
pass

def test_symmetric_difference(self):
pass

def test_shift(self):

# test shift for datetimeIndex and non datetimeIndex
# GH8083

drange = self.create_index()
result = drange.shift(1)
expected = DatetimeIndex(['2013-01-02', '2013-01-03', '2013-01-04',
'2013-01-05',
'2013-01-06'], freq='D', dtype=drange.dtype)
tm.assert_index_equal(result, expected)

result = drange.shift(-1)
expected = DatetimeIndex(['2012-12-31', '2013-01-01', '2013-01-02',
'2013-01-03', '2013-01-04'],
freq='D', dtype=drange.dtype)
tm.assert_index_equal(result, expected)

result = drange.shift(3, freq='2D')
expected = DatetimeIndex(['2013-01-07', '2013-01-08', '2013-01-09',
'2013-01-10',
'2013-01-11'], freq='D', dtype=drange.dtype)
tm.assert_index_equal(result, expected)

def create_index(self):
return date_range('20130101', periods=5).tz_localize("US/Eastern")