ENH: Add Index.fillna #11343

sinhrks · 2015-10-16T14:22:08Z

value can only accept scalar.
MultiIndex.fillna raises NotImplementedError because isnull is not defined for MI.
Moved hasnans and related properties to base Index.

jreback · 2015-10-16T16:28:39Z

pandas/tests/test_index.py

+            if len(index) == 0:
+                pass
+            elif isinstance(index, MultiIndex):
+                pass


assert NotImplementedError here

sinhrks · 2015-10-16T22:59:53Z

Thanks, did the suggested changes.

A test fails in Series.fillna which uses DatetimeIndex.putmask. Looks #11329 fixes the same part, I'm leaving it now.

sinhrks · 2015-10-18T01:16:48Z

Rebased, and now green.

jreback · 2015-10-18T13:41:06Z

pandas/core/index.py

+        try:
+            np.putmask(values, mask, value)
+            return self._shallow_copy(values)
+        except ValueError:


I think if this catches TypeError as well it can be much more generic

sinhrks · 2015-10-25T15:37:21Z

Thanks, simplified the logic based on the comments.

jreback · 2015-10-25T20:01:00Z

pandas/core/index.py

+            return np.array([], dtype=np.int64)
+
+    @cache_readonly
+    def hasnans(self):


this is override core/base.py, but do we use that anylonger? (I don't think Series uses this)

I haven't notice base.py. Should remove it from base.py because Series can be changed inplace?

jreback · 2015-10-28T15:22:13Z

pandas/tseries/index.py

+        --------
+        numpy.ndarray.putmask
+        """
+        try:


you don't need this any longer once you call _assert_can_do_op in Index.putmask

sinhrks · 2015-10-31T22:58:33Z

Thanks, updated based on the comments.

_convert_for_op converts value for scalar op
assert_can_do_op check whether the value is acceptable for scalar op
- timezone check is split to different method. Because it is used in list-likes such as equals.
Some changes related to output detailed error messages / keep current messages

jreback · 2015-10-31T23:01:55Z

pandas/tseries/index.py

+
+        if isinstance(item, np.datetime64):
+            # np.datetime64 doesn't have tz, once convert to Timestamp
+            item = Timestamp(item)


hmm, this seems to be not necessary?

e.g. you are checking if its np.datetime64 below so it will never be the case?

Ah, needs change. This is for current _has_same_tz cannot get tz from np.datetime64.

https://github.com/pydata/pandas/pull/11343/files#diff-23ecb29e7ceba52109a365e447400d2eR562

move the condition to _has_same_tz ? ( convert np.datetime64 to Timestamp).

If we keep 1594th line, 1598th line can be if isinstance(item, datetime):

Maybe option 1 looks better?

yeh 1 is better, this might work

vzone = tslib.get_timezone(getattr(Timestamp(other), 'tzinfo', '__no_tz__'))

actually if you always wrap other then you can simply return Timestamp(other).tz I think

(though may be overlooking something)

In _has_same_tz, other can be DTI or list-likes. Thus added if for np.datetime64.

jreback · 2015-10-31T23:03:40Z

@sinhrks looks really good!

just a minor comment / question

sinhrks · 2015-11-01T00:47:23Z

Reabsed, and now green.

jreback · 2015-11-01T00:56:53Z

lgtm

merge when ready

thanks!

might want to add s short example in the Index docs (iirc where we go over Index set ops)

sinhrks · 2015-11-01T12:06:33Z

Thanks. Added brief doc to indexing.rst. Merging within few days if there is no further comments.

CC: @jorisvandenbossche

jorisvandenbossche · 2015-11-01T20:47:34Z

doc/source/indexing.rst

+Missing values
+~~~~~~~~~~~~~~
+
+.. _indexing.missing:


Can you put this above the title?

jorisvandenbossche · 2015-11-01T20:52:18Z

@sinhrks Looks good! One more remark, doc-wise: I see there is a downcast keyword, but there is no example given of its usage in docstring or rst docs. That would be nice to include to clarify its usage I think.

ENH: Add Index.fillna

jreback · 2015-11-13T16:05:52Z

@sinhrks thanks!

always gr8 PR's from you!

jreback · 2015-11-13T16:06:38Z

I don't see an entry in API.rst can you add?

sinhrks · 2015-11-13T21:17:43Z

@jreback Ah, let me add it with #11537.

sinhrks added Enhancement Indexing Related to indexing on series/frames, not to indexes themselves Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate labels Oct 16, 2015

sinhrks added this to the 0.17.1 milestone Oct 16, 2015

jreback reviewed Oct 16, 2015
View reviewed changes

pandas/tests/test_index.py

if len(index) == 0:

pass

elif isinstance(index, MultiIndex):

pass

Copy link

Contributor

jreback Oct 16, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert NotImplementedError here

sinhrks force-pushed the idx_fillna branch from 902a723 to 4d09b80 Compare October 16, 2015 21:33

sinhrks changed the title ~~ENH: Add Index.fillna~~ (WIP) ENH: Add Index.fillna Oct 16, 2015

sinhrks force-pushed the idx_fillna branch 2 times, most recently from 163b6cc to c08ac92 Compare October 17, 2015 23:24

sinhrks changed the title ~~(WIP) ENH: Add Index.fillna~~ ENH: Add Index.fillna Oct 18, 2015

jreback reviewed Oct 18, 2015
View reviewed changes

sinhrks mentioned this pull request Oct 19, 2015

PERF/CLN: Improve datetime-like index ops perf #10277

Merged

sinhrks force-pushed the idx_fillna branch from c08ac92 to 5e1fff8 Compare October 25, 2015 04:25

jreback reviewed Oct 25, 2015
View reviewed changes

sinhrks force-pushed the idx_fillna branch from 5e1fff8 to bcdb59f Compare October 27, 2015 13:25

jreback reviewed Oct 28, 2015
View reviewed changes

jreback mentioned this pull request Oct 30, 2015

BUG: reset_index of level on a MultiIndex with NaT converts to np.nan #11479

Closed

sinhrks force-pushed the idx_fillna branch 4 times, most recently from 687ce6a to bf1f842 Compare October 31, 2015 01:14

sinhrks changed the title ~~ENH: Add Index.fillna~~ (WIP)ENH: Add Index.fillna Oct 31, 2015

sinhrks force-pushed the idx_fillna branch 2 times, most recently from 6ea192b to b20820a Compare October 31, 2015 03:16

sinhrks changed the title ~~(WIP)ENH: Add Index.fillna~~ ENH: Add Index.fillna Oct 31, 2015

jreback reviewed Oct 31, 2015
View reviewed changes

sinhrks force-pushed the idx_fillna branch from b20820a to 0c07cf5 Compare October 31, 2015 23:40

ENH: Add Index.fillna

bcdb3e9

sinhrks force-pushed the idx_fillna branch from 0c07cf5 to bcdb3e9 Compare November 1, 2015 11:50

jorisvandenbossche reviewed Nov 1, 2015
View reviewed changes

doc/source/indexing.rst

Missing values

~~~~~~~~~~~~~~

.. _indexing.missing:

Copy link

Member

jorisvandenbossche Nov 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you put this above the title?

jreback added a commit that referenced this pull request Nov 13, 2015

Merge pull request #11343 from sinhrks/idx_fillna

2b0886c

ENH: Add Index.fillna

jreback merged commit 2b0886c into pandas-dev:master Nov 13, 2015

sinhrks deleted the idx_fillna branch November 13, 2015 21:17

sinhrks mentioned this pull request Nov 13, 2015

BUG/API: Clarify the behaviour of fillna downcasting #11537

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Add Index.fillna #11343

ENH: Add Index.fillna #11343

sinhrks commented Oct 16, 2015

jreback Oct 16, 2015

sinhrks commented Oct 16, 2015

sinhrks commented Oct 18, 2015

jreback Oct 18, 2015

sinhrks commented Oct 25, 2015

jreback Oct 25, 2015

sinhrks Oct 31, 2015

jreback Oct 28, 2015

sinhrks commented Oct 31, 2015

jreback Oct 31, 2015

sinhrks Oct 31, 2015

jreback Oct 31, 2015

jreback Oct 31, 2015

sinhrks Oct 31, 2015

jreback commented Oct 31, 2015

sinhrks commented Nov 1, 2015

jreback commented Nov 1, 2015

sinhrks commented Nov 1, 2015

jorisvandenbossche Nov 1, 2015

jorisvandenbossche commented Nov 1, 2015

jreback commented Nov 13, 2015

jreback commented Nov 13, 2015

sinhrks commented Nov 13, 2015

ENH: Add Index.fillna #11343

ENH: Add Index.fillna #11343

Conversation

sinhrks commented Oct 16, 2015

Choose a reason for hiding this comment

sinhrks commented Oct 16, 2015

sinhrks commented Oct 18, 2015

Choose a reason for hiding this comment

sinhrks commented Oct 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sinhrks commented Oct 31, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Oct 31, 2015

sinhrks commented Nov 1, 2015

jreback commented Nov 1, 2015

sinhrks commented Nov 1, 2015

Choose a reason for hiding this comment

jorisvandenbossche commented Nov 1, 2015

jreback commented Nov 13, 2015

jreback commented Nov 13, 2015

sinhrks commented Nov 13, 2015