AttributeError: 'IntBlock'/'FloatBlock'/etc. object has no attribute 'sp_index'

[Bthompso8784]

Code Sample, a copy-pastable example if possible

import pandas as pd
# AttributeError: 'IntBlock' object has no attribute 'sp_index'
s = pd.SparseSeries([1, 2])
print(s.where(s >= 2, 0))
# AttributeError: 'FloatBlock' object has no attribute 'sp_index'
df = pd.SparseDataFrame([[1, 2], [3, 4]])
print(df.quantile())

Problem description

On the surface, SparseSeries and SparseDataFrame should produce the same results as Series and DataFrame, respectively.

Expected Output

0 0
1 2
dtype: int64
0 2.0
1 3.0
Name: 0.5, dtype: float64

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None

pandas: 0.20.1
pytest: 3.0.7
pip: 9.0.1
setuptools: 36.0.1
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
xarray: None
IPython: 5.3.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.2.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

[jorisvandenbossche]

@Bthompso8784 Thanks for the report.

For the first case (s.where(s >= 2, 0)) I actually get a segfault (core dumped) on master.
For the second case (df.quantile()), it is seems to be the repr that produces this error, because df.quantile() seems to work only partially (it creates an invalid SparseSeries). Not sure if it should actually return a SparseSeries rather than a plain Series (which is what eg df.mean() is doing).

[kernc]

Can't reproduce this on 0.22.0.dev0+116, Python 3.5. The output is as expected.

[jreback]

@kernc can you do a PR with validation tests?

@Licht-T in addition to you have been improving sparse lately

[jorisvandenbossche]

The where case still doesn't work for me on master (I currently get a RecursionError, while got a segfault in august).
But the df.quantile() seems to work now.

[kernc]

Ah, right, can confirm RecursionError. I was on a branch where I appear to have inadvertently fixed it.

[ColinSuess]

Confirmed both issues repro in Pandas 0.23.4 with Python 3.7 amd64 on Win7SP1x64.

[mroeschke]

This looks to work on master with SparseDtype. Could use a test:

In [8]: s = pd.Series(pd.SparseArray([1, 2]))

In [9]: s
Out[9]:
0    1
1    2
dtype: Sparse[int64, 0]

In [10]: s.where(s >= 2, 0)
Out[10]:
0    0
1    2
dtype: Sparse[int64, 0]

In [11]: s1 = pd.Series(pd.SparseArray([3, 4]))

In [12]: df = pd.DataFrame({0: s, 1: s1})

In [13]: df
Out[13]:
   0  1
0  1  3
1  2  4

In [14]: df.dtypes
Out[14]:
0    Sparse[int64, 0]
1    Sparse[int64, 0]
dtype: object

In [15]: df.quantile()
Out[15]:
0    1.5
1    3.5
Name: 0.5, dtype: float64

In [16]: pd.__version__
Out[16]: '0.26.0.dev0+555.gf7d162b18'