API/COMPAT: support axis=None for logical reduction (reduce over all axes) #21486

TomAugspurger · 2018-06-14T19:56:00Z

closes CI: NumPy logical reductions (any, all) fail on DataFrame with NumPy master #19976
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

This is the minimal fix, just to get np.all / np.any working again.

Some followup items:

Expand to all aggregations, not just logical ones
Do logical reductions blockwiwse: DataFrame.any(axis={0, 1}) returns inconsistent values for non-zero timedelta #17667. Currently, we do DataFrame.values, which isn't necessary for logical reductions.

Accepts axis=None as reduce all dims

jorisvandenbossche · 2018-06-15T06:56:52Z

doc/source/whatsnew/v0.23.2.txt

+
+
+This also provides compatibility with NumPy 1.15, which now dispatches to ``DataFrame.all``.
+With NumPy 1.15 and pandas 0.23.1 or earlier, :func:`numpy.all` will not reduce over every axis:


not -> no longer ?

jorisvandenbossche · 2018-06-15T06:57:34Z

doc/source/whatsnew/v0.23.2.txt

+   B    False
+   dtype: bool
+
+With pandas 0.23.2, that will correctly return False.


maybe add ", as it did before with numpy < 1.15" ?

jorisvandenbossche · 2018-06-15T07:05:28Z

pandas/core/generic.py

@@ -9055,8 +9057,16 @@ def _doc_parms(cls):

 Parameters
 ----------
-axis : int, default 0
-    Select the axis which can be 0 for indices and 1 for columns.
+axis : {None, 0 or 'index', 1 or 'columns'}, default None


I don't think default None is correct?
The default for our method is still 0? It's only when doing np.all(..) that we respect the axis=None ?

jorisvandenbossche · 2018-06-15T07:07:02Z

pandas/core/series.py

@@ -3238,6 +3238,8 @@ def _reduce(self, op, name, axis=0, skipna=True, numeric_only=None,

        """
        delegate = self._values
+        if axis is None:
+            axis = self._stat_axis_number


Why is this still needed?

Can you have a look at this one?

Sorry, missed this earlier. We still call self._get_axis_number(axis) to validate that the user passed axis is correct. I'll modify things to be clearer.

jorisvandenbossche · 2018-06-15T07:08:51Z

pandas/core/generic.py

-        if skipna is None:
-            skipna = True
-        if axis is None:
-            axis = self._stat_axis_number


Shouldn't there be an override of this in case of Panel? (which is the only case where the stat_axis differs from 0)

jreback · 2018-06-15T16:39:36Z

pandas/core/frame.py

@@ -6859,6 +6864,13 @@ def f(x):
            try:
                values = self.values
                result = f(values)
+
+                if (filter_type == 'bool' and values.dtype.kind == 'O' and


use is_object_dtype

jreback · 2018-06-15T16:41:20Z

pandas/tests/frame/test_analytics.py

@@ -1159,11 +1159,34 @@ def test_any_all(self):
        self._check_bool_op('any', np.any, has_skipna=True, has_bool_only=True)
        self._check_bool_op('all', np.all, has_skipna=True, has_bool_only=True)

-        df = DataFrame(randn(10, 4)) > 0


I would make a new test function here

jorisvandenbossche · 2018-06-21T09:57:53Z

Tests are still failing: np.any/np.all don't seem to work on timedelta data (at least for certain versions of numpy, locally for me it is failing on 1.13)

codecov · 2018-06-21T21:37:29Z

Codecov Report

Merging #21486 into master will decrease coverage by 0.02%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master   #21486      +/-   ##
==========================================
- Coverage   91.92%    91.9%   -0.03%     
==========================================
  Files         153      153              
  Lines       49563    49562       -1     
==========================================
- Hits        45559    45548      -11     
- Misses       4004     4014      +10

Flag	Coverage Δ
#multiple	`90.3% <85.71%> (-0.03%)`	⬇️
#single	`41.77% <53.57%> (-0.04%)`	⬇️

Impacted Files	Coverage Δ
pandas/util/_test_decorators.py	`92.68% <100%> (+0.18%)`	⬆️
pandas/core/generic.py	`96.21% <100%> (+0.08%)`	⬆️
pandas/core/series.py	`94.19% <100%> (ø)`	⬆️
pandas/core/panel.py	`97.43% <62.5%> (-0.44%)`	⬇️
pandas/core/frame.py	`97.19% <90.9%> (-0.05%)`	⬇️
pandas/util/testing.py	`85.27% <0%> (-0.7%)`	⬇️
pandas/core/indexing.py	`93.37% <0%> (-0.05%)`	⬇️
pandas/tseries/offsets.py	`97.16% <0%> (-0.04%)`	⬇️
pandas/core/dtypes/common.py	`94.63% <0%> (-0.02%)`	⬇️
pandas/core/indexes/multi.py	`94.97% <0%> (-0.01%)`	⬇️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b36b451...ae759bd. Read the comment docs.

jreback · 2018-06-22T23:28:44Z

I think you need to change

    @pytest.mark.xfail(
        not _np_version_under1p15,
        reason="failing under numpy-dev gh-19976")
    @pytest.mark.parametrize("axis", [0, 1, None])
    def test_clip_against_frame(self, axis):
        df = DataFrame(np.random.randn(1000, 2))

in pandas/tests/frame/test_analytics.py which was the original failing test, to use your new decorator?

otherwise lgtm.

TomAugspurger · 2018-06-23T03:05:15Z

Good catch. I've removed the xfail entirely, since I think it will pass on all versions now.

…axes) (pandas-dev#21486) * Compat with NumPy 1.15 logical func * Accepts axis=None as reduce all dims (cherry picked from commit f7ed7f8)

…axes) (#21486) * Compat with NumPy 1.15 logical func * Accepts axis=None as reduce all dims (cherry picked from commit f7ed7f8)

…axes) (pandas-dev#21486) * Compat with NumPy 1.15 logical func * Accepts axis=None as reduce all dims

TomAugspurger added 2 commits June 14, 2018 14:44

Compat with NumPy 1.15 logical func

a6114b4

Accepts axis=None as reduce all dims

whatsnew

4bb6e53

TomAugspurger added Numeric Operations Arithmetic, Comparison, and Logical operations Compat pandas objects compatability with Numpy or Python functions labels Jun 14, 2018

TomAugspurger added this to the 0.23.2 milestone Jun 14, 2018

TomAugspurger mentioned this pull request Jun 14, 2018

CI: NumPy logical reductions (any, all) fail on DataFrame with NumPy master #19976

Closed

remove old test

9636d54

jorisvandenbossche reviewed Jun 15, 2018

View reviewed changes

jreback requested changes Jun 15, 2018

View reviewed changes

TomAugspurger added 3 commits June 20, 2018 08:01

updated docs

18c1b11

Merge remote-tracking branch 'upstream/master' into logical-agg

3c7f4e5

Handle panel

1c32b2b

jorisvandenbossche changed the title ~~Logical agg~~ API/COMPAT: support axis=None for logical reduction (reduce over all axes) Jun 21, 2018

Skip for np 114

1f94469

jorisvandenbossche approved these changes Jun 22, 2018

View reviewed changes

TomAugspurger mentioned this pull request Jun 22, 2018

Support axis=None in all reductions #21597

Closed

TomAugspurger added 2 commits June 22, 2018 22:01

reuse numpy compat

50db719

remove xfail

9fd9740

Linting

ae759bd

jorisvandenbossche merged commit f7ed7f8 into pandas-dev:master Jun 26, 2018

jorisvandenbossche added Needs Backport and removed Needs Backport labels Jun 26, 2018

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

API/COMPAT: support axis=None for logical reduction (reduce over all …

78e1a67

…axes) (pandas-dev#21486) * Compat with NumPy 1.15 logical func * Accepts axis=None as reduce all dims

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API/COMPAT: support axis=None for logical reduction (reduce over all axes) #21486

API/COMPAT: support axis=None for logical reduction (reduce over all axes) #21486

TomAugspurger commented Jun 14, 2018

jorisvandenbossche Jun 15, 2018

jorisvandenbossche Jun 15, 2018

jorisvandenbossche Jun 15, 2018

jorisvandenbossche Jun 15, 2018

jorisvandenbossche Jun 20, 2018

TomAugspurger Jun 21, 2018

jorisvandenbossche Jun 15, 2018

jreback Jun 15, 2018

jreback Jun 15, 2018

jorisvandenbossche commented Jun 21, 2018

codecov bot commented Jun 21, 2018 •

edited

Loading

jreback commented Jun 22, 2018

TomAugspurger commented Jun 23, 2018



		This also provides compatibility with NumPy 1.15, which now dispatches to ``DataFrame.all``.
		With NumPy 1.15 and pandas 0.23.1 or earlier, :func:`numpy.all` will not reduce over every axis:

API/COMPAT: support axis=None for logical reduction (reduce over all axes) #21486

API/COMPAT: support axis=None for logical reduction (reduce over all axes) #21486

Conversation

TomAugspurger commented Jun 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Jun 21, 2018

codecov bot commented Jun 21, 2018 • edited Loading

Codecov Report

jreback commented Jun 22, 2018

TomAugspurger commented Jun 23, 2018

codecov bot commented Jun 21, 2018 •

edited

Loading