DOC clarify inplace operation section in 1.5 whats_new #47433

lesteve · 2022-06-20T15:39:46Z

Coming from scikit-learn tests failing on scikit-learn with the pandas development version, I found the whats_new entry not very helpful at all. This does a few things:

use an example with different dtypes so that the behaviour is as advertised for old behaviour (i.e. ser is not updated in place). This was not the case before, since when dtype matches, ser is updated in place so the code snippet was not showing the right behaviour.
make it clearer that there is no behaviour change in pandas 1.5

Some of the wording may be not completely accurate, as I don't have a very good grasp of the pandas internals, feel free to suggest improvements!

jreback

this needs some editing

i

jreback

i don't know if this is an improvement. can you try to make a minimal change instead.

jreback · 2022-06-21T01:24:43Z

doc/source/whatsnew/v1.5.0.rst


 .. code-block:: ipython

-    In [3]: df.iloc[:, 0] = np.array([10, 11])
+    In [3]: df.iloc[:, 0] = np.array([10, 11], dtype=np.int32)


why did you change this?

When the dtypes match the old (pandas < 1.5) behaviour is already to update the underlying array in place.

import numpy as np import pandas as pd print(f"{pd.__version__=}") values = np.arange(4).reshape(2, 2) df = pd.DataFrame(values) ser = df[0] print(f"before setting column\n{ser}") df.iloc[:, 0] = np.array([10, 11]) print(f"after setting column\n{ser}")

Output:

pd.__version__='1.4.2' before setting column 0 0 1 2 Name: 0, dtype: int64 after setting column 0 10 1 11 Name: 0, dtype: int64

This snippet was incorrect in pretending that ser was not updated in place.

I used a different dtype for the assignment rhs term to be in the case, where:

the old (pandas < 1.5) behaviour is not to update in place

the new (pandas 1.5) behaviour is the same with an additional warning

the future behaviour will be to update in place

jreback · 2022-06-21T01:25:03Z

doc/source/whatsnew/v1.5.0.rst

    In [4]: ser
    Out[4]:
    0    0
    1    2
    Name: 0, dtype: int64

-This behavior is deprecated. In a future version, setting an entire column with
-iloc will attempt to operate inplace.
+*Behavior with pandas 1.5* is the same but you get a ``FutureWarning``:


leave the original note, this is not inline with our styling.

jreback · 2022-06-21T01:25:18Z

doc/source/whatsnew/v1.5.0.rst


 *Future behavior*:

+In a future version, setting an entire column with ``iloc`` will attempt to


this is redundant

lesteve · 2022-06-21T10:22:12Z

doc/source/whatsnew/v1.5.0.rst

@@ -562,14 +562,15 @@ As ``group_keys=True`` is the default value of :meth:`DataFrame.groupby` and
 raise a ``FutureWarning``. This can be silenced and the previous behavior
 retained by specifying ``group_keys=False``.

-.. _whatsnew_150.notable_bug_fixes.setitem_column_try_inplace:
+.. _whatsnew_150.deprecations.setitem_column_try_inplace:
   _ see also _whatsnew_130.notable_bug_fixes.setitem_column_try_inplace


side-comment: the _ see also ... line is not rendered in the HTML, not sure whether it was supposed to be a link to https://pandas.pydata.org/docs/dev/whatsnew/v1.3.0.html#try-operating-inplace-when-setting-values-with-loc-and-iloc

Yeah it looks like it's supposed to reference that section. Mind correcting it? Can be in the body of the paragraph too like Please reference `this section <whatsnew_130.notable_bug_fixes.setitem_column_try_inplace>` of the 1.3 whatsnew file.

lesteve · 2022-06-21T10:22:49Z

I have tried to make the diff smaller as advised

lesteve · 2022-06-21T10:23:30Z

doc/source/whatsnew/v1.5.0.rst

   _ see also _whatsnew_130.notable_bug_fixes.setitem_column_try_inplace

-Try operating inplace when setting values with ``loc`` and ``iloc``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Inplace operation when setting values with ``iloc``


Not 100% sure but it seems the changes are only about .iloc so I remove the .loc mention

Actually this happens with .loc too when setting the entire column so reverting this change. In this case the warning is a bit misleading as it mentions .iloc

datapythonista

lgtm, added couple of small things, but this makes sense.

If you want to spend the time, I think it'd help read this section is the example had some more "interesting" data. I think it's more difficult to understand the code with the original arbitrary code, and even more now, that one needs to understand what's going on with the types.

Maybe something like:

pandas.DataFrame({'price': [10, 15]}, index=['book1', 'boo2'])
new_prices = [11.50, 18.20]

Does this make sense?

datapythonista · 2022-06-22T13:16:26Z

doc/source/whatsnew/v1.5.0.rst

@@ -595,7 +596,7 @@ iloc will attempt to operate inplace.



Maybe worth adding ...an entire column with different type with iloc... to the comment above?

datapythonista · 2022-06-22T13:18:12Z

doc/source/whatsnew/v1.5.0.rst


-*Future behavior*:
+*New behavior*


For what we're saying, this is not new behavior, but future after the deprecation, right? Or I'm missing something? Also, there is another Future behavior block above, that needs to be updated if I'm wrong. And the colon is there in the other behavior titles.

My understanding is that DataFrame.isetitem has been added in https://github.com/pandas-dev/pandas/pull/45333/files#diff-421998af5fe0fbc54b35773ce9b6289cae3e8ae607f81783af04ebf1fbcaf077R3690 so it only exists in pandas 1.5

lesteve · 2022-06-22T15:41:28Z

If you want to spend the time, I think it'd help read this section is the example had some more "interesting" data.

I did this in my last commit, I also showed df.iloc[:, 0] which I think helps in understanding what is happening (whether the column keeps its dtype or not).

datapythonista

This seems super clear to me now, thanks a lot @lesteve

mroeschke · 2022-06-30T20:11:44Z

Thanks @lesteve

) * DOC make inplace operation section in whats_new clearer * tweak * Can happen with both loc and iloc when setting entire column * Use more meaningful data Co-authored-by: Matthew Roeschke <[email protected]>

DOC make inplace operation section in whats_new clearer

5334e6a

lesteve mentioned this pull request Jun 20, 2022

New behaviour regarding inplace values setting with iloc #47381

Closed

jreback requested changes Jun 20, 2022

View reviewed changes

jreback added the Docs label Jun 21, 2022

jreback requested changes Jun 21, 2022

View reviewed changes

tweak

fae74ac

lesteve commented Jun 21, 2022

View reviewed changes

lesteve mentioned this pull request Jun 22, 2022

MAINT make explicit casting when using iloc scikit-learn/scikit-learn#23649

Merged

Can happen with both loc and iloc when setting entire column

23cca41

datapythonista reviewed Jun 22, 2022

View reviewed changes

Use more meaningful data

d189395

datapythonista approved these changes Jun 22, 2022

View reviewed changes

simonjayhawkins added this to the 1.5 milestone Jun 25, 2022

Merge branch 'main' into enh-inplace-iloc-whats-new

fbe6202

jreback approved these changes Jun 30, 2022

View reviewed changes

mroeschke merged commit 9a3231f into pandas-dev:main Jun 30, 2022

lesteve deleted the enh-inplace-iloc-whats-new branch July 1, 2022 06:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DOC clarify inplace operation section in 1.5 whats_new #47433

DOC clarify inplace operation section in 1.5 whats_new #47433

lesteve commented Jun 20, 2022 •

edited

Loading

jreback left a comment

jreback left a comment

jreback Jun 21, 2022

lesteve Jun 21, 2022 •

edited

Loading

jreback Jun 21, 2022

jreback Jun 21, 2022

lesteve Jun 21, 2022 •

edited

Loading

mroeschke Jun 23, 2022

lesteve commented Jun 21, 2022

lesteve Jun 21, 2022

lesteve Jun 22, 2022

datapythonista left a comment

datapythonista Jun 22, 2022

datapythonista Jun 22, 2022

lesteve Jun 22, 2022

lesteve commented Jun 22, 2022

datapythonista left a comment

mroeschke commented Jun 30, 2022


		Future behavior:

		In a future version, setting an entire column with ``iloc`` will attempt to

DOC clarify inplace operation section in 1.5 whats_new #47433

DOC clarify inplace operation section in 1.5 whats_new #47433

Conversation

lesteve commented Jun 20, 2022 • edited Loading

jreback left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lesteve Jun 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lesteve Jun 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lesteve commented Jun 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datapythonista left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lesteve commented Jun 22, 2022

datapythonista left a comment

Choose a reason for hiding this comment

mroeschke commented Jun 30, 2022

lesteve commented Jun 20, 2022 •

edited

Loading

lesteve Jun 21, 2022 •

edited

Loading

lesteve Jun 21, 2022 •

edited

Loading