fix for TypeError: unorderable types" in when using set_index with multiple column names #22072

dickreuter · 2018-07-27T00:50:45Z

closes "TypeError: unorderable types" in Python3 when column for MultiIndex contains tuple and int #15457

gfyoung · 2018-07-27T04:39:32Z

Going to need tests for this as well.

toobaz · 2018-07-27T06:36:13Z

closes #16718

Thanks for your attempt! But is this really the bug you're after?! (feel free to edit the description)

codecov · 2018-07-27T09:28:45Z

Codecov Report

Merging #22072 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #22072   +/-   ##
=======================================
  Coverage   92.08%   92.08%           
=======================================
  Files         169      169           
  Lines       50694    50694           
=======================================
  Hits        46682    46682           
  Misses       4012     4012

Flag	Coverage Δ
#multiple	`90.49% <100%> (ø)`	⬆️
#single	`42.34% <100%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/arrays/categorical.py	`95.75% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0370740...213c4ea. Read the comment docs.

toobaz

The fix is good, some comments.

toobaz · 2018-07-27T08:49:00Z

doc/source/whatsnew/v0.24.0.txt

@@ -430,7 +430,7 @@ Bug Fixes
 Categorical
 ^^^^^^^^^^^

-
+- Fixes an issue with creating multiple indices with for example .set_index([a,b]) giving an error "TypeError: unorderable types" (:issue:'15457')


This is not telling the user what the problem really was (that example given works fine in 99% of cases). Moreover, the bug (in the sense of API misbehavior) is unrelated to categorical Consider e.g. Fix issue when creating :class:`MultiIndex` from mixed type arrays including tuples (:issue:'15457')

toobaz · 2018-07-27T10:22:22Z

pandas/core/arrays/categorical.py

@@ -2520,7 +2520,7 @@ def _factorize_from_iterable(values):
                                      ordered=values.ordered)
        codes = values.codes
    else:
-        cat = Categorical(values, ordered=True)
+        cat = Categorical(values, ordered=False)


Smart fix... but not really simple to understand. Please add a comment stating that the value of ordered is irrelevant since we don't use cat as such, but only the resulting categories, the order of which is independent from ordered.

jreback

always write tests first - how else can you tell if you fixed something?

pep8speaks · 2018-07-27T23:15:17Z

Hello @dickreuter! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on August 11, 2018 at 12:07 Hours UTC

dickreuter · 2018-07-27T23:15:54Z

Changed the description to point to the correct bug: "TypeError: unorderable types" in Python3 when column for MultiIndex contains tuple and int #15457
Added test
Better doc

dickreuter · 2018-07-28T00:14:40Z

Would like to check it locally but building pandas seems impossible (cl.exe failed with exit status 2). Is there no additional info what version of visual studio dev tools is needed exactly? https://stackoverflow.com/questions/51566895/unable-to-build-pandas-from-source-cl-exe-failed-with-exit-status-2

TomAugspurger · 2018-07-28T02:30:48Z

http://pandas-docs.github.io/pandas-docs-travis/contributing.html#installing-a-c-compiler And
https://devguide.python.org/setup/#build-dependencies may help

dickreuter · 2018-07-28T09:48:52Z

I don't see how the error on the ci is related to my change (especially also since the tests passed yesterday)

toobaz · 2018-07-28T09:55:08Z

I don't see how the error on the ci is related to my change (especially also since the tests passed yesterday)

At least one is explained by

pandas/tests/arrays/categorical/test_indexing.py:131:41: W292 no newline at end of file

(see also the automatic comment by @pep8speaks )

dickreuter · 2018-07-28T11:08:05Z

Still getting an error which I can't see how it's related to this: BufferError: could not get scalar buffer information. Running this test locally passes fine.

jreback · 2018-07-28T13:00:29Z

doc/source/whatsnew/v0.23.4.txt

@@ -51,7 +51,7 @@ Bug Fixes

 **Categorical**

-
+- Fix issue when creating :class:`MultiIndex` from mixed type arrays including tuples (:issue:'15457')


can you be even more explicit here, a user won't under stand this.

jreback · 2018-07-28T13:00:40Z

pandas/core/arrays/categorical.py

@@ -2520,7 +2520,10 @@ def _factorize_from_iterable(values):
                                      ordered=values.ordered)
        codes = values.codes
    else:
-        cat = Categorical(values, ordered=True)
+        # The value of ordered is irrelevant since we don't use cat as such,


can you add the issue reference

jreback · 2018-07-28T13:01:44Z

pandas/tests/arrays/categorical/test_indexing.py

@@ -121,3 +121,11 @@ def test_get_indexer_non_unique(self, idx_values, key_values, key_class):

            tm.assert_numpy_array_equal(expected, result)
            tm.assert_numpy_array_equal(exp_miss, res_miss)
+
+
+class TestSetIndex(object):


put in pandas/tests/frame/test_alter_axes.py.

put in a full comparison on the result table and use assert_fram_equal

df = DataFrame([[2, 1, 2], [4, (1, 2), 3]]).set_index([0, 1]) gives me Out[97]: 2 0 1 2 1 2 4 (1, 2) 3

How I construct this directly from the DataFrame constructor? The first fow seems empty?
df.to_dict() gives {2: {(2, 1): 2, (4, (1, 2)): 3}}

I tried:

DataFrame(['',2,3], columns=['2'], index=[['0','2','4'], ['1','1','(1,2)']])

but it's not equal (even though it looks equal), but
DataFrame shape mismatch

[left]: (2, 1) [right]: (3, 1)

Implemented with dict instead:
assert df.to_dict() == {2: {(2, 1): 2, (4, (1, 2)): 3}}

but it's not equal (even though it looks equal), but

Yes, they look equal, but you can see from how you built them they are not. [0, 1] are the index names, not index values.

pd.DataFrame([2,3], columns=['2'], index=pd.MultiIndex.from_arrays([['2','4'], ['1','(1,2)']], names=[0, 1]))

is what you're looking for.

df = DataFrame([[2, 1, 2], [4, (1, 2), 3]]).set_index([0, 1]) assert_frame_equal(df, pd.DataFrame([2, 3], columns=['2'], index=pd.MultiIndex. from_arrays([['2', '4'], ['1', '(1,2)']], names=[0, 1]))) MultiIndex level [0] classes are not equivalent [left]: Int64Index([2, 4], dtype='int64', name=0) [right]: Index(['2', '4'], dtype='object', name=0)

I also tried

df = DataFrame([[2, 1, 2], [4, (1, 2), 3]]).set_index([0, 1]) assert_frame_equal(df, pd.DataFrame([2, 3], columns=['2'], index=pd.MultiIndex. from_arrays([[2, 4], ['1', '(1,2)']], names=[0, 1]))) Attribute "inferred_type" are different [left]: mixed-integer [right]: string

Maybe the dictionary is a more feasible solution,.

I also tried

The original df has no strings at all, just integers. Why are you putting strings in the other (e.g. '2', '(1,2)'?!)

Removing the two strings I get ValueError: setting an array element with a sequence.

Removing the two strings I get ValueError: setting an array element with a sequence.

OK, pd.MultiIndex.from_tuples([(2, 1), (4, (1,2))], names=[0, 1])

jreback · 2018-07-28T16:01:34Z

doc/source/whatsnew/v0.23.4.txt

@@ -51,7 +51,7 @@ Bug Fixes

 **Categorical**

-
+- Fix issue when creating :class:`MultiIndex` from mixed type arrays including tuples (:issue:'15457'). In some cases df.set_index(['A','B']) raised a TypeError unorderable types when trying to set multiple columns as indices.


move to 0.24

jreback · 2018-07-28T16:02:07Z

pandas/tests/frame/test_alter_axes.py

@@ -1186,3 +1186,11 @@ def test_set_axis_prior_to_deprecation_signature(self):
            with tm.assert_produces_warning(FutureWarning):
                result = df.set_axis(axis, list('abc'), inplace=False)
            tm.assert_frame_equal(result, expected[axis])
+


don't add a new class, rather put this test next to the other set_index tests

jreback · 2018-07-29T15:38:49Z

pandas/tests/frame/test_alter_axes.py

+    def test_set_index_tuple(self):
+        # test for fix of issue #15457, where the following raised a TypeError
+        df = DataFrame([[2, 1, 2], [4, (1, 2), 3]]).set_index([0, 1])
+        assert df.to_dict() == {2: {(2, 1): 2, (4, (1, 2)): 3}}


use assert_frame_equal, IOW you must construct the expected frame. use result= and expected= here

@jreback @dickreuter since the bug is in MultiIndex.from_arrays, I think that only reconstructing the index (as I suggested above, comparing from_arrays with from_tuples, which already works) rather than the full DataFrame - and moving the test to pandas/tests/indexes/multi/test_constructor.py - is OK (actually even better)

@toobaz Comparing from_arrays with from_tuples? Not sure I understand. As you said, this already works.

@dickreuter from_tuples already works, from_arrays doesn't. With your patch, both do, so you can compare one to the other. Something like (untested):

result = pd.MultiIndex.from_arrays([[2, 4], [1, (1, 2)]]) expected = pd.MultiIndex.from_tuples([(2, 1), (4, (1,2))]) ...

@toobaz I reverted to the test via dict as otherwise line 908 of cast.py causes an error when trying to create an array from [1, (1, 2)], raising {ValueError}setting an array element with a sequence. This may be an unrelated issue but not sure the test can be constructed in that way at the moment.

@dickreuter this has to be an assert on a frame, otherwise this test is not testing what we need here

{ValueError}setting an array element with a sequence

@dickreuter You are right, sorry. This should work instead:

expected = pd.DataFrame([[2, 1], [4, (1, 2)]]).set_index([0, 1]).index result = pd.MultiIndex.from_tuples([(2, 1), (4, (1,2))], names=(0, 1))

@jreback see my comment above, the bug is unrelated to DataFrame

jreback · 2018-07-31T12:20:03Z

doc/source/whatsnew/v0.24.0.txt

@@ -430,7 +430,7 @@ Bug Fixes
 Categorical
 ^^^^^^^^^^^

-
+- Fix issue when creating :class:`MultiIndex` from mixed type arrays including tuples (:issue:'15457'). In some cases df.set_index(['A','B']) raised a TypeError unorderable types when trying to set multiple columns as indices.


use double backticks around TypeError, list the issue number here as well. Move this to MultiIndex section.

You want thesame issue number mentioned twice? I'm not aware of any second issue number.

dickreuter · 2018-08-01T04:16:31Z

@toobaz: I'm getting ValueError: setting an array element with a sequence when I test that. Not sure my fix is related to this bug?

dickreuter · 2018-08-04T12:53:14Z

@toobaz I reverted to the test via dict as otherwise line 908 of cast.py causes an error when trying to create an array from [1, (1, 2)], raising {ValueError}setting an array element with a sequence.

dickreuter · 2018-08-08T08:11:52Z

@toobaz @jreback What's the next step? Can this be merged? tx

gfyoung · 2018-08-08T09:15:50Z

https://travis-ci.org/pandas-dev/pandas/jobs/413344917#L3557

@dickreuter : You have linter failures. I would fix those first before pinging.

jreback · 2018-08-09T10:50:23Z

rebased

@toobaz merge when satisfied

toobaz · 2018-08-10T10:42:41Z

doc/source/whatsnew/v0.24.0.txt

@@ -626,7 +626,7 @@ MultiIndex

 - Removed compatibility for :class:`MultiIndex` pickles prior to version 0.8.0; compatibility with :class:`MultiIndex` pickles from version 0.13 forward is maintained (:issue:`21654`)
 - :meth:`MultiIndex.get_loc_level` (and as a consequence, ``.loc`` on a :class:``MultiIndex``ed object) will now raise a ``KeyError``, rather than returning an empty ``slice``, if asked a label which is present in the ``levels`` but is unused (:issue:`22221`)
-
+- Fix `TypeError` in Python 3 when creating `MultiIndex` in which some columns are of object type, e.g. when labels are tuples  (:issue:'15457')


Apart from my comment on "object type" ( #22072 ), you have lost the ":class:" before "MultiIndex", and you still have quotes rather than backticks around the issue number

toobaz · 2018-08-10T11:45:10Z

doc/source/whatsnew/v0.24.0.txt

@@ -626,7 +626,7 @@ MultiIndex

 - Removed compatibility for :class:`MultiIndex` pickles prior to version 0.8.0; compatibility with :class:`MultiIndex` pickles from version 0.13 forward is maintained (:issue:`21654`)
 - :meth:`MultiIndex.get_loc_level` (and as a consequence, ``.loc`` on a :class:``MultiIndex``ed object) will now raise a ``KeyError``, rather than returning an empty ``slice``, if asked a label which is present in the ``levels`` but is unused (:issue:`22221`)
-
+- Fix `TypeError` in Python 3 when creating :class: `MultiIndex` in which some columns are of object type, e.g. when labels are tuples  (:issue:`15457`)


@dickreuter OK, I think I see your point. What I'm saying is "not all object" dtype give problems, what you are saying is "other object dtypes than tuples give problems". I think a more precise description then is "in which some levels have mixed types", as I guess happens in your case (no error comes if all labels are ctype items, right?).

Then i think we're ready.

@toobaz I believe in the example I first hit the error there where 3 columns. All of them were dtype and all the rows were homogenous of the same type as well.

@toobaz When you say mixed type, is the column name also relevant for that for each for? If yes, then I assume it’s a mixed type because the column names were strings in my case.

Your new version is almost ready (see my last comment), but for the records:

I believe in the example I first hit the error there where 3 columns. All of them were dtype and all the rows were homogenous of the same type as well.

The error only comes out when two elements (in a same column) cannot be ordered, which in Python 3 happens when they are of non-comparable types. If you got an error with columns which homogeneous types, I doubt it was the same error.
Notice that I'm referring to (Python) types, not dtypes. If you put mixed types in a single column, then its dtype will be just object, but it is still a mixed type column.

When you say mixed type, is the column name also relevant for that for each for?

No, column names are stored separately and should be unrelated to this bug and to any issue with mixed type columns/levels.

I assume in my case the elements in the same column were unorderable not because they had multiple non-comparable types but because they contained just references to memory of python ctype objects.

toobaz · 2018-08-10T13:09:40Z

doc/source/whatsnew/v0.24.0.txt

@@ -626,7 +626,7 @@ MultiIndex

 - Removed compatibility for :class:`MultiIndex` pickles prior to version 0.8.0; compatibility with :class:`MultiIndex` pickles from version 0.13 forward is maintained (:issue:`21654`)
 - :meth:`MultiIndex.get_loc_level` (and as a consequence, ``.loc`` on a :class:``MultiIndex``ed object) will now raise a ``KeyError``, rather than returning an empty ``slice``, if asked a label which is present in the ``levels`` but is unused (:issue:`22221`)
-
+- Fix `TypeError` in Python 3 when creating :class: `MultiIndex` in which some levels have mixed types, e.g. when labels are tuples (:issue:`15457`)


Almost there... "when some labels are tuples". If (for instance) all labels are tuples of ints, there is no problem.

toobaz · 2018-08-10T16:54:41Z

doc/source/whatsnew/v0.24.0.txt

@@ -626,7 +626,7 @@ MultiIndex

 - Removed compatibility for :class:`MultiIndex` pickles prior to version 0.8.0; compatibility with :class:`MultiIndex` pickles from version 0.13 forward is maintained (:issue:`21654`)
 - :meth:`MultiIndex.get_loc_level` (and as a consequence, ``.loc`` on a :class:``MultiIndex``ed object) will now raise a ``KeyError``, rather than returning an empty ``slice``, if asked a label which is present in the ``levels`` but is unused (:issue:`22221`)
-
+- Fix `TypeError` in Python 3 when creating :class: `MultiIndex` in which some levels have mixed types. If (for instance) all labels are tuples of ints, there is no problem (:issue:`15457`)


@dickreuter the previous one was almost OK, this one is not... in the whatsnew, we don't need examples in which "there is no problem" :-)

Please revert to the previous one, just adding "some", that is

Fix `TypeError` in Python 3 when creating :class: `MultiIndex` in which some levels have mixed types, e.g. when some labels are tuples (:issue:`15457`)

This reverts commit b645286

toobaz · 2018-08-11T11:15:58Z

doc/source/whatsnew/v0.24.0.txt

@@ -626,7 +626,7 @@ MultiIndex

 - Removed compatibility for :class:`MultiIndex` pickles prior to version 0.8.0; compatibility with :class:`MultiIndex` pickles from version 0.13 forward is maintained (:issue:`21654`)
 - :meth:`MultiIndex.get_loc_level` (and as a consequence, ``.loc`` on a :class:``MultiIndex``ed object) will now raise a ``KeyError``, rather than returning an empty ``slice``, if asked a label which is present in the ``levels`` but is unused (:issue:`22221`)
-
+- Fix `TypeError` in Python 3 when creating :class: `MultiIndex` in which some levels have mixed types, e.g. when some labels are tuples (:issue:`15457`)


Double backticks around TypeError, no space after :class:

dickreuter · 2018-08-11T19:51:22Z

@toobaz Why does travis fail with "Unable to download 3.5 archive. The archive may not exist. Please consider a different version"?

toobaz · 2018-08-12T08:17:21Z

Thanks @dickreuter !

gfyoung · 2018-08-12T08:35:15Z

@dickreuter : FYI, those Travis failures that you saw earlier are spurious. I restarted the builds to get them passing and this PR merged 🙂

…with mixed dtypes (pandas-dev#22072) closes pandas-dev#15457

dickreuter mentioned this pull request Jul 27, 2018

"TypeError: unorderable types" in Python3 when column for MultiIndex contains tuple and int #15457

Closed

dickreuter changed the title ~~fix for issue #15457~~ for for TypeError: unorderable types" in when using set_index with list Jul 27, 2018

dickreuter changed the title ~~for for TypeError: unorderable types" in when using set_index with list~~ for for TypeError: unorderable types" in when using set_index with multiple column names Jul 27, 2018

dickreuter changed the title ~~for for TypeError: unorderable types" in when using set_index with multiple column names~~ fix for TypeError: unorderable types" in when using set_index with multiple column names Jul 27, 2018

gfyoung added Bug IO LaTeX to_latex MultiIndex labels Jul 27, 2018

gfyoung requested review from jreback and toobaz July 27, 2018 04:39

toobaz suggested changes Jul 27, 2018

View reviewed changes

jreback requested changes Jul 27, 2018

View reviewed changes

jreback requested changes Jul 28, 2018

View reviewed changes

jreback requested changes Jul 29, 2018

View reviewed changes

jreback requested changes Jul 31, 2018

View reviewed changes

jreback approved these changes Aug 9, 2018

View reviewed changes

toobaz reviewed Aug 10, 2018

View reviewed changes

dickreuter and others added 12 commits August 11, 2018 10:58

BUG: Fix for TypeError: unorderable types" for multi index (#15457)

af2ab19

revert to test with dictionary

9bf7262

pylint fixes, new test

bcea58b

text correction

42f093d

changes as per comments

bba1e9c

revert changes

1b3959e

better description

10ed01a

more fixes

801803a

change in doc as per comments

8daf9e6

text change

4cd3c0c

additional text changes

c45705c

Revert "additional text changes"

927f24f

This reverts commit b645286

toobaz reviewed Aug 11, 2018

View reviewed changes

text fix

213c4ea

toobaz approved these changes Aug 12, 2018

View reviewed changes

toobaz merged commit d09db1f into pandas-dev:master Aug 12, 2018

dickreuter deleted the feature/fix-#15457 branch August 12, 2018 08:29

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

BUG: fix for "TypeError: unorderable types" when creating MultiIndex …

dded930

…with mixed dtypes (pandas-dev#22072) closes pandas-dev#15457

toobaz mentioned this pull request Nov 5, 2018

Can't combine DatetimeIndex MultiIndex with numeric MultiIndex #23478

Closed

benediamond mentioned this pull request Nov 22, 2018

Support for use of Enums in MultiIndex #21348

Closed

4 tasks

WillAyd mentioned this pull request Nov 26, 2018

Support for Enums in MultiIndex #21298

Closed

fix for TypeError: unorderable types" in when using set_index with multiple column names #22072

fix for TypeError: unorderable types" in when using set_index with multiple column names #22072

Conversation

dickreuter commented Jul 27, 2018 • edited Loading

gfyoung commented Jul 27, 2018

toobaz commented Jul 27, 2018

codecov bot commented Jul 27, 2018 • edited Loading

Codecov Report

toobaz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

pep8speaks commented Jul 27, 2018 • edited Loading

Comment last updated on August 11, 2018 at 12:07 Hours UTC

dickreuter commented Jul 27, 2018 • edited Loading

dickreuter commented Jul 28, 2018 • edited Loading

TomAugspurger commented Jul 28, 2018

dickreuter commented Jul 28, 2018

toobaz commented Jul 28, 2018

dickreuter commented Jul 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dickreuter Jul 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dickreuter Jul 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dickreuter Jul 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toobaz Aug 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dickreuter commented Aug 1, 2018 • edited Loading

dickreuter commented Aug 4, 2018

dickreuter commented Aug 8, 2018

gfyoung commented Aug 8, 2018

jreback commented Aug 9, 2018

toobaz Aug 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dickreuter Aug 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dickreuter commented Aug 11, 2018

toobaz commented Aug 12, 2018

gfyoung commented Aug 12, 2018 • edited Loading

dickreuter commented Jul 27, 2018 •

edited

Loading

codecov bot commented Jul 27, 2018 •

edited

Loading

pep8speaks commented Jul 27, 2018 •

edited

Loading

dickreuter commented Jul 27, 2018 •

edited

Loading

dickreuter commented Jul 28, 2018 •

edited

Loading

dickreuter commented Jul 28, 2018 •

edited

Loading

dickreuter Jul 28, 2018 •

edited

Loading

dickreuter Jul 28, 2018 •

edited

Loading

dickreuter Jul 30, 2018 •

edited

Loading

toobaz Aug 8, 2018 •

edited

Loading

dickreuter commented Aug 1, 2018 •

edited

Loading

toobaz Aug 10, 2018 •

edited

Loading

dickreuter Aug 10, 2018 •

edited

Loading

gfyoung commented Aug 12, 2018 •

edited

Loading