Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SeriesGroupby transform unique not working in Pandas 1.0.1 #31849

Closed
Shawe82 opened this issue Feb 10, 2020 · 7 comments
Closed

SeriesGroupby transform unique not working in Pandas 1.0.1 #31849

Shawe82 opened this issue Feb 10, 2020 · 7 comments
Labels
Regression Functionality that used to work in a prior pandas version

Comments

@Shawe82
Copy link

Shawe82 commented Feb 10, 2020

Code Sample, a copy-pastable example if possible

# Your code here
import pandas as pd
df = pd.DataFrame({"title":['a','b','c','c','a'], "asset_id":['a1','b1','c1','c2','c1']})
df.groupby("title").asset_id.transform("unique")

Problem description

Since pandas 1.0.1, unique does not seem to be a whitelisted transform function for SeriesGroupby objects anymore. The above code works with pandas 0.25.3 but not with 1.0.1 It raises a ValueError exception:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-ccb0ce21c35f> in <module>
      1 import pandas as pd
      2 df = pd.DataFrame({"title":['a','b','c','c','a'], "asset_id":['a1','b1','c1','c2','c1']})
----> 3 df.groupby("title").asset_id.transform("unique")

~/sandbox/pandas/pandas/core/groupby/generic.py in transform(self, func, *args, **kwargs)
    470         elif func not in base.transform_kernel_whitelist:
    471             msg = f"'{func}' is not a valid function name for transform(name)"
--> 472             raise ValueError(msg)
    473         elif func in base.cythonized_kernels:
    474             # cythonized transform or canned "agg+broadcast"

ValueError: 'unique' is not a valid function name for transform(name)

I tried to find a documentation on that changed but was not succesful. If that is the intended behaviour, I think it would be helpful to document it.

Expected Output

0 [a1, c1]
1 [b1]
2 [c1, c2]
3 [c1, c2]
4 [a1, c1]
Name: asset_id, dtype: object

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit : None
python : 3.8.1.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8
pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 19.0.3
setuptools : 40.8.0
Cython : None
pytest : 5.3.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.3.13
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

@MarcoGorelli MarcoGorelli added the Regression Functionality that used to work in a prior pandas version label Feb 10, 2020
@TomAugspurger
Copy link
Contributor

Thanks. In the future, can you include the full traceback? I've added it

@TomAugspurger
Copy link
Contributor

Last touched in #29672 (cc @jbrockmendel) but I haven't checked if it was working prior to that commit.

@jorisvandenbossche jorisvandenbossche added this to the 1.0.2 milestone Feb 10, 2020
@jbrockmendel
Copy link
Member

IIRC #29672 was intended to be logic-neutral.

@MarcoGorelli
Copy link
Member

If I've done git bisect correctly, it stopped working after this PR: #27467

@WillAyd
Copy link
Member

WillAyd commented Feb 29, 2020

Hmm this seems a little hacky as nothing else in our built-in transform machinery returns a non-scalar (at least that I can think of off the top of my head)

Note that df.groupby("title").asset_id.transform("unique") would also return a different value than df.groupby("title").asset_id.transform(lambda x: x.unique()) which can't be said of other functions

Given that, I'd be -1 on this being considered a regression or even a bug

@MarcoGorelli
Copy link
Member

Makes sense. Currently the whitelisted kernels either return a single numeric value (reduction_kernels), or produce a result which has the same shape as the group (transformation_kernels). unique returns a single list, which doesn't fit into either category.

Should this be closed then?

@jreback jreback modified the milestones: 1.0.2, No action Mar 3, 2020
@jreback
Copy link
Contributor

jreback commented Mar 3, 2020

this is not a transformation kernel, you can do this if you want

In [13]: df.groupby('title').asset_id.apply(lambda x: x.unique())                                                                                                                                                                                               
Out[13]: 
title
a    [a1, c1]
b        [b1]
c    [c1, c2]
Name: asset_id, dtype: object

@jreback jreback closed this as completed Mar 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Regression Functionality that used to work in a prior pandas version
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants