whatsnew 1.3.0 #41747

rhshadrach · 2021-05-31T18:21:33Z

Ensure all linting tests pass, see here for how to run them

Minor fixups. After I get feedback here, I plan to make a PR with detailed instructions about adding to the whatsnew.

cc @attack68 - the Styler sections had a major revisions, wanted to get any feedback here.

…atsnew1.3 � Conflicts: � doc/source/whatsnew/v1.3.0.rst

jreback · 2021-05-31T19:12:59Z

doc/source/whatsnew/v1.3.0.rst


-:class:`.Styler` has also been compatible with non-unique index or columns, at least for as many features as are fully compatible, others made only partially compatible (:issue:`41269`).
-One also has greater control of the display through separate sparsification of the index or columns, using the new 'styler' options context (:issue:`41142`).
+Many features of the :class:`.Styler` class are now either partially or fully usable on a DataFrame with a non-unique indexes or columns (:issue:`41269`).


can use bullet points to improve readability

Agreed - while reworking this as bullet points I noticed that much is a duplicate of other lines in the whatsnew. We should either have a single section in Enhancements or single lines in the Other Enhancements (which will be broken into sections, and hence include a Styler section) and Bugfixes.

I lean toward breaking it up (since they aren't all enhancements) into the appropriate sections. cc @attack68 for thoughts.

Marking this PR as a draft until this is resolved so it won't be merged.

jreback · 2021-05-31T19:13:51Z

doc/source/whatsnew/v1.3.0.rst

+- Add support for dict-like names in :meth:`Index.set_names` and :meth:`.Index.rename` in the case of a :class:`MultiIndex` (:issue:`20421`)
+- :func:`read_excel` can now auto-detect .xlsb files and older .xls files (:issue:`35416`, :issue:`41225`)
+- :class:`ExcelWriter` now accepts an ``if_sheet_exists`` parameter to control the behaviour of append mode when writing to existing sheets (:issue:`40230`)
+- :meth:`.Rolling.sum`, :meth:`.Expanding.sum`, :meth:`.Rolling.mean`, :meth:`.Expanding.mean`, :meth:`.ExponentialMovingWindow.mean`, :meth:`.Rolling.median`, :meth:`.Expanding.median`, :meth:`.Rolling.max`, :meth:`.Expanding.max`, :meth:`.Rolling.min`, and :meth:`.Expanding.min` now support `Numba <http://numba.pydata.org/>`_ execution with the ``engine`` keyword (:issue:`38895`, :issue:`41267`)


might be worth it to:

group the other enhancements by category / type (groupby / rolling / styler etc). and create-sub headers

jreback · 2021-05-31T19:15:55Z

doc/source/whatsnew/v1.3.0.rst

- Bug in :meth:`DataFrame.loc.__setitem__` when setting-with-expansion incorrectly raising when the index in the expanding axis contains duplicates (:issue:`40096`)
- Bug in :meth:`DataFrame.loc.__getitem__` with :class:`MultiIndex` casting to float when at least one column is from has float dtype and we retrieve a scalar (:issue:`41369`)
- Bug in :meth:`DataFrame.loc` incorrectly matching non-boolean index elements (:issue:`20432`)
+- Bug in :meth:`DataFrame.loc` not raising ``KeyError`` when the key was not found in :class:`MultiIndex` and the levels were not fully specified (:issue:`41170`)


I think we have a MultiIndex section? if so can you move relevant notes

i think this belongs here in the Indexing section, since the user-facing bug is in .loc

jreback · 2021-05-31T19:16:34Z

doc/source/whatsnew/v1.3.0.rst

- Bug in :func:`read_sas` raising ``ValueError`` when ``datetimes`` were null (:issue:`39725`)
+- Bug in :func:`read_csv` raising a ``TypeError`` when ``names`` and ``parse_dates`` is specified for ``engine="c"`` (:issue:`33699`)
+- Bug in :func:`read_clipboard` and :func:`DataFrame.to_clipboard` not working in WSL (:issue:`38527`)
+- Allow custom error values for the ``parse_dates`` argument of :func:`read_sql`, :func:`read_sql_query` and :func:`read_sql_table` (:issue:`35185`)


ideally can group notes e.g. for read_csv, read_excel etc. sub-sections prob not needed)

jreback · 2021-05-31T19:17:16Z

doc/source/whatsnew/v1.3.0.rst



 Other
 ^^^^^
 - Bug in :class:`Index` constructor sometimes silently ignoring a specified ``dtype`` (:issue:`38879`)
- Bug in :func:`pandas.api.types.infer_dtype` not recognizing Series, Index or array with a period dtype (:issue:`23553`)
- Bug in :func:`pandas.api.types.infer_dtype` raising an error for general :class:`.ExtensionArray` objects. It will now return ``"unknown-array"`` instead of raising (:issue:`37367`)
+- Bug in :func:`.infer_dtype` not recognizing Series, Index, or array with a Period dtype (:issue:`23553`)


any of these that can be in other more specific sections would be good.

doc/source/whatsnew/v1.3.0.rst

rhshadrach · 2021-06-01T03:34:12Z

@jreback - agreed with all your suggestions above but would like to hold off and do in a followup to keep the diff here reasonable. I think it's okay for the changes to happen during the RC period (but won't be waiting for the RC period either)

rhshadrach · 2021-06-01T03:38:13Z

@attack68 - The performance improvement of the HTML rendering referenced #37792 but this is unrelated, not sure what issue/PR was meant. Also from the line

The method :meth:.Styler.format has been improved to easily format missing data, precision, and perform HTML escaping (:issue:40437, :issue:40134)

I think the "format missing data" is referring to #40134, but I can't find changes to na_rep there. Can you help me understand what changed?

attack68

lgtm.

after many commits it became a bit messy. the bullet point format seems better. Also there were some Styler additions in relevant sections below this one, so some things might have been duplicated, but maybe thats ok for readers?

attack68 · 2021-06-01T16:57:01Z

@attack68 - The performance improvement of the HTML rendering referenced #37792 but this is unrelated, not sure what issue/PR was meant. Also from the line

The method :meth:.Styler.format has been improved to easily format missing data, precision, and perform HTML escaping (:issue:40437, :issue:40134)

I think the "format missing data" is referring to #40134, but I can't find changes to na_rep there. Can you help me understand what changed?

there was a function called Styler.set_na_rep(), which set a property on self and the format method with an na_rep arg which conflicted at times. So, yes, it was there previously but it didn't always do what you thought. Now the missing data in Styler is handled exclusively and consistently by Styler.format, and Styler.set_na_rep is deprecated.

attack68 · 2021-06-01T16:59:45Z

@attack68 - The performance improvement of the HTML rendering referenced #37792 but this is unrelated, not sure what issue/PR was meant. Also from the line

#39972 and #39952

jbrockmendel · 2021-06-01T18:44:11Z

doc/source/whatsnew/v1.3.0.rst

@@ -693,10 +686,10 @@ Deprecations
 - Deprecated passing arguments as positional in :meth:`DataFrame.set_index` (other than ``"keys"``) (:issue:`41485`)
 - Deprecated passing arguments as positional (except for ``"levels"``) in :meth:`MultiIndex.set_levels` (:issue:`41485`)
 - Deprecated passing arguments as positional in :meth:`DataFrame.sort_index` and :meth:`Series.sort_index` (:issue:`41485`)
- Deprecated passing arguments as positional in :meth:`DataFrame.drop_duplicates` (except for ``subset``), :meth:`Series.drop_duplicates`, :meth:`Index.drop_duplicates` and :meth:`MultiIndex.drop_duplicates`(:issue:`41485`)
+- Deprecated passing arguments as positional in :meth:`DataFrame.drop_duplicates` (except for ``subset``), :meth:`Series.drop_duplicates`, :meth:`Index.drop_duplicates` and :meth:`MultiIndex.drop_duplicates` (:issue:`41485`)


Not necessarily for this PR, but there are enough deprecations i think it might be worth subsections for e.g. positional arguments (or at least collect them all together)

@MarcoGorelli is planning to tidy that up. #41485 (comment)

yup - those deprecations are all done now so I'll do this week

jbrockmendel · 2021-06-01T18:45:39Z

@rhshadrach there's a lot here. id suggest breaking off the non-controversial bits into a precursor PR

simonjayhawkins · 2021-06-01T18:50:59Z

@rhshadrach there's a lot here. id suggest breaking off the non-controversial bits into a precursor PR

maybe start with the sphinx directive syntax errors, then another PR for the sphinx directives with broken links, then the grammer and then finally move stuff.

simonjayhawkins · 2021-06-01T18:52:09Z

The first one (maybe 2) would be good for the rc

rhshadrach · 2021-06-02T00:08:34Z

Going back on what I said in #41747 (comment), I only plan to finish up this what's in this PR. I'll break it up as @simonjayhawkins recommended.

doc/source/whatsnew/v1.3.0.rst

jbrockmendel · 2021-06-04T19:37:11Z

@rhshadrach is more of this getting broken off or can you mark as Ready For Review?

…atsnew1.3 � Conflicts: � doc/source/whatsnew/v1.3.0.rst

rhshadrach · 2021-06-05T04:00:15Z

@jbrockmendel - one last one (#41822).

The main reason for draft status is #41747 (comment). Currently the top section of styler duplicates many of the other notes in the whatsnew. Need to deduplicate and decide whether enhancements for the Styler should appear in the top enhancement section or as line items in the Other Enhancements (grouped together). Any thoughts here would be appreciated.

I think it is clear that bugfixes should go in the Styler bugfixes section and not in the top enhancement section.

rhshadrach · 2021-06-05T20:42:15Z

I've deduplicated the Styler notes and reorganized so that all enhancements are in Enhancements in their own section, while Performance, Deprecation, and Bugfix notes appear below in the appropriate sections.

doc/source/whatsnew/v1.3.0.rst

…atsnew1.3 � Conflicts: � doc/source/whatsnew/v1.3.0.rst

rhshadrach · 2021-06-07T20:47:19Z

I can siphon off the Styler relevant pieces into a separate PR as well if that will make this easier to review. Let me know if that's desirable @simonjayhawkins, @jbrockmendel (or anyone else)

doc/source/whatsnew/v1.3.0.rst

attack68 · 2021-06-07T21:34:36Z

doc/source/whatsnew/v1.3.0.rst

+ - Added the method :meth:`.Styler.highlight_between` (:issue:`39821`)
+ - Added the method :meth:`.Styler.highlight_quantile` (:issue:`40926`)
+ - Added the method :meth:`.Styler.text_gradient` (:issue:`41098`)
+ - Added the method :meth:`.Styler.set_tooltips` allowing hover tooltips; this can be used enhance interactive displays (:issue:`21266`)


this is duplicated with line 123

doc/source/whatsnew/v1.3.0.rst

attack68

see a few inline comments

doc/source/whatsnew/v1.3.0.rst

simonjayhawkins

I would say merge this now. and can do another pass to address any outstanding comments, but web and docs ci is failing. can you fix that up.

simonjayhawkins · 2021-06-08T13:43:48Z

@rhshadrach merge conflicts

…atsnew1.3 � Conflicts: � doc/source/whatsnew/v1.3.0.rst

rhshadrach · 2021-06-09T11:41:07Z

Thanks @attack68 and @simonjayhawkins for the comments/requests, adjustments made and conflicts resolved.

simonjayhawkins · 2021-06-09T11:57:41Z

Thanks @rhshadrach I think just @jreback comments about moving stuff to do in follow-up.

rhshadrach added 3 commits May 31, 2021 13:57

Fixups for whatsnew 1.3.0

20aa4ad

Merge branch 'master' of https://github.com/pandas-dev/pandas into wh…

5044390

…atsnew1.3 � Conflicts: � doc/source/whatsnew/v1.3.0.rst

Few more fixups

41f87d9

rhshadrach added the Docs label May 31, 2021

jreback requested changes May 31, 2021

View reviewed changes

rhshadrach commented May 31, 2021

View reviewed changes

doc/source/whatsnew/v1.3.0.rst Show resolved Hide resolved

Reworked the Styler section

468aaac

rhshadrach marked this pull request as draft June 1, 2021 03:32

attack68 approved these changes Jun 1, 2021

View reviewed changes

jbrockmendel reviewed Jun 1, 2021

View reviewed changes

jreback added this to the 1.3 milestone Jun 1, 2021

jbrockmendel reviewed Jun 2, 2021

View reviewed changes

doc/source/whatsnew/v1.3.0.rst Outdated Show resolved Hide resolved

jbrockmendel reviewed Jun 2, 2021

View reviewed changes

doc/source/whatsnew/v1.3.0.rst Outdated Show resolved Hide resolved

jbrockmendel reviewed Jun 2, 2021

View reviewed changes

doc/source/whatsnew/v1.3.0.rst Outdated Show resolved Hide resolved

This was referenced Jun 3, 2021

DOC: whatsnew 1.3.0 syntax errors #41794

Merged

Whatsnew1.3.0 links #41795

Merged

Merge branch 'master' of https://github.com/pandas-dev/pandas into wh…

05d4873

…atsnew1.3 � Conflicts: � doc/source/whatsnew/v1.3.0.rst

rhshadrach mentioned this pull request Jun 5, 2021

DOC: Whatsnew1.3 grammar #41822

Merged

1 task

Re-organize and dedupe Styler notes

d3a7bb3

rhshadrach commented Jun 5, 2021

View reviewed changes

doc/source/whatsnew/v1.3.0.rst Outdated Show resolved Hide resolved

rhshadrach added 2 commits June 7, 2021 16:31

Merge branch 'master' of https://github.com/pandas-dev/pandas into wh…

2f8ce07

…atsnew1.3 � Conflicts: � doc/source/whatsnew/v1.3.0.rst

Some reverts and fixups

6adf94f

rhshadrach marked this pull request as ready for review June 7, 2021 20:41

caused -> causing

0d01c9e

attack68 reviewed Jun 7, 2021

View reviewed changes

doc/source/whatsnew/v1.3.0.rst Outdated Show resolved Hide resolved

attack68 reviewed Jun 7, 2021

View reviewed changes

doc/source/whatsnew/v1.3.0.rst Outdated Show resolved Hide resolved

attack68 reviewed Jun 7, 2021

View reviewed changes

doc/source/whatsnew/v1.3.0.rst Outdated Show resolved Hide resolved

attack68 requested changes Jun 7, 2021

View reviewed changes

simonjayhawkins reviewed Jun 8, 2021

View reviewed changes

doc/source/whatsnew/v1.3.0.rst Show resolved Hide resolved

simonjayhawkins requested changes Jun 8, 2021

View reviewed changes

rhshadrach added 4 commits June 8, 2021 23:15

Fixups

d87b59d

Merge branch 'master' of https://github.com/pandas-dev/pandas into wh…

7801118

…atsnew1.3 � Conflicts: � doc/source/whatsnew/v1.3.0.rst

Fix indentation

9202b74

More indents

c510bd8

simonjayhawkins merged commit cecc3a1 into pandas-dev:master Jun 9, 2021

rhshadrach deleted the whatsnew1.3 branch June 9, 2021 15:50

JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021

whatsnew 1.3.0 (pandas-dev#41747)

092c66f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

whatsnew 1.3.0 #41747

whatsnew 1.3.0 #41747

rhshadrach commented May 31, 2021 •

edited

Loading

jreback May 31, 2021

rhshadrach Jun 1, 2021

rhshadrach Jun 1, 2021

jreback May 31, 2021

jreback May 31, 2021

jbrockmendel Jun 1, 2021

jreback May 31, 2021

jreback May 31, 2021

rhshadrach commented Jun 1, 2021

rhshadrach commented Jun 1, 2021

attack68 left a comment

attack68 commented Jun 1, 2021 •

edited

Loading

attack68 commented Jun 1, 2021

jbrockmendel Jun 1, 2021

simonjayhawkins Jun 9, 2021

MarcoGorelli Jun 9, 2021

jbrockmendel commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021 •

edited

Loading

simonjayhawkins commented Jun 1, 2021

rhshadrach commented Jun 2, 2021

jbrockmendel commented Jun 4, 2021

rhshadrach commented Jun 5, 2021

rhshadrach commented Jun 5, 2021 •

edited

Loading

rhshadrach commented Jun 7, 2021

attack68 Jun 7, 2021

attack68 left a comment

simonjayhawkins left a comment

simonjayhawkins commented Jun 8, 2021

rhshadrach commented Jun 9, 2021

simonjayhawkins commented Jun 9, 2021

whatsnew 1.3.0 #41747

whatsnew 1.3.0 #41747

Conversation

rhshadrach commented May 31, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhshadrach commented Jun 1, 2021

rhshadrach commented Jun 1, 2021

attack68 left a comment

Choose a reason for hiding this comment

attack68 commented Jun 1, 2021 • edited Loading

attack68 commented Jun 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Jun 1, 2021

simonjayhawkins commented Jun 1, 2021 • edited Loading

simonjayhawkins commented Jun 1, 2021

rhshadrach commented Jun 2, 2021

jbrockmendel commented Jun 4, 2021

rhshadrach commented Jun 5, 2021

rhshadrach commented Jun 5, 2021 • edited Loading

rhshadrach commented Jun 7, 2021

Choose a reason for hiding this comment

attack68 left a comment

Choose a reason for hiding this comment

simonjayhawkins left a comment

Choose a reason for hiding this comment

simonjayhawkins commented Jun 8, 2021

rhshadrach commented Jun 9, 2021

simonjayhawkins commented Jun 9, 2021

rhshadrach commented May 31, 2021 •

edited

Loading

attack68 commented Jun 1, 2021 •

edited

Loading

simonjayhawkins commented Jun 1, 2021 •

edited

Loading

rhshadrach commented Jun 5, 2021 •

edited

Loading