[Bug] v0.23.4 is not able to handle tuples as keys of pd.Series like v0.22.0 | TypeError: 'values' is not ordered...

[jolespin]

Code Sample, a copy-pastable example if possible

I've posted this bug on issue 15457 and the issue was closed but it still occurs in the current version (0.23.4)

# =====
# v0.23.4
# =====

>>> import sys; sys.version
'3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 12:04:33) \n[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]'
>>> import pandas as pd; pd.__version__
'0.23.4'
>>> import numpy as np; from numpy import array; from collections import *
>>> param_index = OrderedDict([((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 1)), array([  0,  40,  80, 120, 160, 200])), ((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 2)), array([  1,  41,  81, 121, 161, 201])), ((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 3)), array([  2,  42,  82, 122, 162, 202])), ((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 5)), array([  3,  43,  83, 123, 163, 203])), ((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 8)), array([  4,  44,  84, 124, 164, 204])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 1)), array([  5,  45,  85, 125, 165, 205])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 2)), array([  6,  46,  86, 126, 166, 206])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 3)), array([  7,  47,  87, 127, 167, 207])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 5)), array([  8,  48,  88, 128, 168, 208])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 8)), array([  9,  49,  89, 129, 169, 209])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 1)), array([ 10,  50,  90, 130, 170, 210])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 2)), array([ 11,  51,  91, 131, 171, 211])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 3)), array([ 12,  52,  92, 132, 172, 212])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 5)), array([ 13,  53,  93, 133, 173, 213])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 8)), array([ 14,  54,  94, 134, 174, 214])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 1)), array([ 15,  55,  95, 135, 175, 215])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 2)), array([ 16,  56,  96, 136, 176, 216])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 3)), array([ 17,  57,  97, 137, 177, 217])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 5)), array([ 18,  58,  98, 138, 178, 218])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 8)), array([ 19,  59,  99, 139, 179, 219])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 1)), array([ 20,  60, 100, 140, 180, 220])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 2)), array([ 21,  61, 101, 141, 181, 221])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 3)), array([ 22,  62, 102, 142, 182, 222])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 5)), array([ 23,  63, 103, 143, 183, 223])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 8)), array([ 24,  64, 104, 144, 184, 224])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 1)), array([ 25,  65, 105, 145, 185, 225])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 2)), array([ 26,  66, 106, 146, 186, 226])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 3)), array([ 27,  67, 107, 147, 187, 227])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 5)), array([ 28,  68, 108, 148, 188, 228])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 8)), array([ 29,  69, 109, 149, 189, 229])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 1)), array([ 30,  70, 110, 150, 190, 230])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 2)), array([ 31,  71, 111, 151, 191, 231])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 3)), array([ 32,  72, 112, 152, 192, 232])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 5)), array([ 33,  73, 113, 153, 193, 233])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 8)), array([ 34,  74, 114, 154, 194, 234])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 1)), array([ 35,  75, 115, 155, 195, 235])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 2)), array([ 36,  76, 116, 156, 196, 236])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 3)), array([ 37,  77, 117, 157, 197, 237])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 5)), array([ 38,  78, 118, 158, 198, 238])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 8)), array([ 39,  79, 119, 159, 199, 239]))])
>>> pd.Series(list(param_index.values()), index=param_index.keys())
Traceback (most recent call last):
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/algorithms.py", line 635, in factorize
    order = uniques.argsort()
TypeError: '<' not supported between instances of 'NoneType' and 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/sorting.py", line 451, in safe_sort
    sorter = values.argsort()
TypeError: '<' not supported between instances of 'NoneType' and 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/arrays/categorical.py", line 345, in __init__
    codes, categories = factorize(values, sort=True)
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/util/_decorators.py", line 178, in wrapper
    return func(*args, **kwargs)
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/algorithms.py", line 643, in factorize
    assume_unique=True)
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/sorting.py", line 455, in safe_sort
    ordered = sort_mixed(values)
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/sorting.py", line 441, in sort_mixed
    nums = np.sort(values[~str_pos])
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 847, in sort
    a.sort(axis=axis, kind=kind, order=order)
TypeError: '<' not supported between instances of 'NoneType' and 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/series.py", line 183, in __init__
    index = _ensure_index(index)
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 4974, in _ensure_index
    return Index(index_like)
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 449, in __new__
    data, names=name or kwargs.get('names'))
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/indexes/multi.py", line 1330, in from_tuples
    return MultiIndex.from_arrays(arrays, sortorder=sortorder, names=names)
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/indexes/multi.py", line 1274, in from_arrays
    labels, levels = _factorize_from_iterables(arrays)
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/arrays/categorical.py", line 2543, in _factorize_from_iterables
    return map(list, lzip(*[_factorize_from_iterable(it) for it in iterables]))
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/arrays/categorical.py", line 2543, in <listcomp>
    return map(list, lzip(*[_factorize_from_iterable(it) for it in iterables]))
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/arrays/categorical.py", line 2515, in _factorize_from_iterable
    cat = Categorical(values, ordered=True)
  File "/Users/jespinoz/anaconda/envs/py3_testing/lib/python3.6/site-packages/pandas/core/arrays/categorical.py", line 351, in __init__
    raise TypeError("'values' is not ordered, please "
TypeError: 'values' is not ordered, please explicitly specify the categories order by passing in a categories argument.
>>>

Problem description

I'm unable to use tuples as index keys in the newest version 0.23.4 like I was able to in 0.22.0. I have some software that is dependent on this functionality. Is there any way to add this functionality back into the coming versions?

Expected Output

I was expecting to be able to create a pd.Series object from my param_index dictionary object.

# =====
# v0.22.0
# =====
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.version
'3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51) \n[GCC 7.2.0]'
>>> import pandas as pd; pd.__version__
'0.22.0'
>>> import numpy as np; from numpy import array; from collections import *
>>> param_index = OrderedDict([((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 1)), array([  0,  40,  80, 120, 160, 200])), ((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 2)), array([  1,  41,  81, 121, 161, 201])), ((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 3)), array([  2,  42,  82, 122, 162, 202])), ((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 5)), array([  3,  43,  83, 123, 163, 203])), ((('criterion', 'gini'), ('max_features', 'log2'), ('min_samples_leaf', 8)), array([  4,  44,  84, 124, 164, 204])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 1)), array([  5,  45,  85, 125, 165, 205])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 2)), array([  6,  46,  86, 126, 166, 206])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 3)), array([  7,  47,  87, 127, 167, 207])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 5)), array([  8,  48,  88, 128, 168, 208])), ((('criterion', 'gini'), ('max_features', 'sqrt'), ('min_samples_leaf', 8)), array([  9,  49,  89, 129, 169, 209])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 1)), array([ 10,  50,  90, 130, 170, 210])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 2)), array([ 11,  51,  91, 131, 171, 211])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 3)), array([ 12,  52,  92, 132, 172, 212])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 5)), array([ 13,  53,  93, 133, 173, 213])), ((('criterion', 'gini'), ('max_features', None), ('min_samples_leaf', 8)), array([ 14,  54,  94, 134, 174, 214])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 1)), array([ 15,  55,  95, 135, 175, 215])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 2)), array([ 16,  56,  96, 136, 176, 216])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 3)), array([ 17,  57,  97, 137, 177, 217])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 5)), array([ 18,  58,  98, 138, 178, 218])), ((('criterion', 'gini'), ('max_features', 0.382), ('min_samples_leaf', 8)), array([ 19,  59,  99, 139, 179, 219])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 1)), array([ 20,  60, 100, 140, 180, 220])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 2)), array([ 21,  61, 101, 141, 181, 221])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 3)), array([ 22,  62, 102, 142, 182, 222])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 5)), array([ 23,  63, 103, 143, 183, 223])), ((('criterion', 'entropy'), ('max_features', 'log2'), ('min_samples_leaf', 8)), array([ 24,  64, 104, 144, 184, 224])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 1)), array([ 25,  65, 105, 145, 185, 225])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 2)), array([ 26,  66, 106, 146, 186, 226])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 3)), array([ 27,  67, 107, 147, 187, 227])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 5)), array([ 28,  68, 108, 148, 188, 228])), ((('criterion', 'entropy'), ('max_features', 'sqrt'), ('min_samples_leaf', 8)), array([ 29,  69, 109, 149, 189, 229])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 1)), array([ 30,  70, 110, 150, 190, 230])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 2)), array([ 31,  71, 111, 151, 191, 231])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 3)), array([ 32,  72, 112, 152, 192, 232])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 5)), array([ 33,  73, 113, 153, 193, 233])), ((('criterion', 'entropy'), ('max_features', None), ('min_samples_leaf', 8)), array([ 34,  74, 114, 154, 194, 234])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 1)), array([ 35,  75, 115, 155, 195, 235])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 2)), array([ 36,  76, 116, 156, 196, 236])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 3)), array([ 37,  77, 117, 157, 197, 237])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 5)), array([ 38,  78, 118, 158, 198, 238])), ((('criterion', 'entropy'), ('max_features', 0.382), ('min_samples_leaf', 8)), array([ 39,  79, 119, 159, 199, 239]))])
>>> pd.Series(list(param_index.values()), index=param_index.keys())
((criterion, gini), (max_features, log2), (min_samples_leaf, 1))          [0, 40, 80, 120, 160, 200]
((criterion, gini), (max_features, log2), (min_samples_leaf, 2))          [1, 41, 81, 121, 161, 201]
((criterion, gini), (max_features, log2), (min_samples_leaf, 3))          [2, 42, 82, 122, 162, 202]
((criterion, gini), (max_features, log2), (min_samples_leaf, 5))          [3, 43, 83, 123, 163, 203]
((criterion, gini), (max_features, log2), (min_samples_leaf, 8))          [4, 44, 84, 124, 164, 204]
((criterion, gini), (max_features, sqrt), (min_samples_leaf, 1))          [5, 45, 85, 125, 165, 205]
((criterion, gini), (max_features, sqrt), (min_samples_leaf, 2))          [6, 46, 86, 126, 166, 206]
((criterion, gini), (max_features, sqrt), (min_samples_leaf, 3))          [7, 47, 87, 127, 167, 207]
((criterion, gini), (max_features, sqrt), (min_samples_leaf, 5))          [8, 48, 88, 128, 168, 208]
((criterion, gini), (max_features, sqrt), (min_samples_leaf, 8))          [9, 49, 89, 129, 169, 209]
((criterion, gini), (max_features, None), (min_samples_leaf, 1))         [10, 50, 90, 130, 170, 210]
((criterion, gini), (max_features, None), (min_samples_leaf, 2))         [11, 51, 91, 131, 171, 211]
((criterion, gini), (max_features, None), (min_samples_leaf, 3))         [12, 52, 92, 132, 172, 212]
((criterion, gini), (max_features, None), (min_samples_leaf, 5))         [13, 53, 93, 133, 173, 213]
((criterion, gini), (max_features, None), (min_samples_leaf, 8))         [14, 54, 94, 134, 174, 214]
((criterion, gini), (max_features, 0.382), (min_samples_leaf, 1))        [15, 55, 95, 135, 175, 215]
((criterion, gini), (max_features, 0.382), (min_samples_leaf, 2))        [16, 56, 96, 136, 176, 216]
((criterion, gini), (max_features, 0.382), (min_samples_leaf, 3))        [17, 57, 97, 137, 177, 217]
((criterion, gini), (max_features, 0.382), (min_samples_leaf, 5))        [18, 58, 98, 138, 178, 218]
((criterion, gini), (max_features, 0.382), (min_samples_leaf, 8))        [19, 59, 99, 139, 179, 219]
((criterion, entropy), (max_features, log2), (min_samples_leaf, 1))     [20, 60, 100, 140, 180, 220]
((criterion, entropy), (max_features, log2), (min_samples_leaf, 2))     [21, 61, 101, 141, 181, 221]
((criterion, entropy), (max_features, log2), (min_samples_leaf, 3))     [22, 62, 102, 142, 182, 222]
((criterion, entropy), (max_features, log2), (min_samples_leaf, 5))     [23, 63, 103, 143, 183, 223]
((criterion, entropy), (max_features, log2), (min_samples_leaf, 8))     [24, 64, 104, 144, 184, 224]
((criterion, entropy), (max_features, sqrt), (min_samples_leaf, 1))     [25, 65, 105, 145, 185, 225]
((criterion, entropy), (max_features, sqrt), (min_samples_leaf, 2))     [26, 66, 106, 146, 186, 226]
((criterion, entropy), (max_features, sqrt), (min_samples_leaf, 3))     [27, 67, 107, 147, 187, 227]
((criterion, entropy), (max_features, sqrt), (min_samples_leaf, 5))     [28, 68, 108, 148, 188, 228]
((criterion, entropy), (max_features, sqrt), (min_samples_leaf, 8))     [29, 69, 109, 149, 189, 229]
((criterion, entropy), (max_features, None), (min_samples_leaf, 1))     [30, 70, 110, 150, 190, 230]
((criterion, entropy), (max_features, None), (min_samples_leaf, 2))     [31, 71, 111, 151, 191, 231]
((criterion, entropy), (max_features, None), (min_samples_leaf, 3))     [32, 72, 112, 152, 192, 232]
((criterion, entropy), (max_features, None), (min_samples_leaf, 5))     [33, 73, 113, 153, 193, 233]
((criterion, entropy), (max_features, None), (min_samples_leaf, 8))     [34, 74, 114, 154, 194, 234]
((criterion, entropy), (max_features, 0.382), (min_samples_leaf, 1))    [35, 75, 115, 155, 195, 235]
((criterion, entropy), (max_features, 0.382), (min_samples_leaf, 2))    [36, 76, 116, 156, 196, 236]
((criterion, entropy), (max_features, 0.382), (min_samples_leaf, 3))    [37, 77, 117, 157, 197, 237]
((criterion, entropy), (max_features, 0.382), (min_samples_leaf, 5))    [38, 78, 118, 158, 198, 238]
((criterion, entropy), (max_features, 0.382), (min_samples_leaf, 8))    [39, 79, 119, 159, 199, 239]
dtype: object

Output of pd.show_versions()

pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 15.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 3.4.0
pip: 9.0.1
setuptools: 39.0.1
Cython: 0.27.3
numpy: 1.14.3
scipy: 1.0.0
pyarrow: 0.8.0
xarray: 0.10.3
IPython: 6.4.0
sphinx: 1.7.4
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: 0.4.0
matplotlib: 2.2.2
openpyxl: 2.5.0
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[jolespin]

I ended up doing this a quick fix for version stability. I wasn't sure what the problem was at first from the error message. I ended up doing this which is compatible with both versions:

pandas v0.23.4

 >>> pd.__version__
'0.23.4'
>>> index_categorical = pd.Index([*map(frozenset, index)], dtype="category")
>>> dict(index_categorical[0])
{'criterion': 'gini', 'max_features': 'log2', 'min_samples_leaf': 1}

pandas v0.22.0

 >>> pd.__version__
'0.22.0'
>>> index_categorical = pd.Index([*map(frozenset, index)], dtype="category")
>>> dict(index_categorical[0])
{'min_samples_leaf': 1, 'criterion': 'gini', 'max_features': 'log2'}

[mroeschke]

I'm getting this error on master:

ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

Not sure if this is expected, but a simplified example would be helpful. Happy to reopen if we could get a minimal example. https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports