Support axis=None in all reductions

[TomAugspurger]

#21486 is implementing support for axis=None for compatibility with NumPy 1.15's all / any.

The basic idea is to have df.op(axis=None) be the same as np.op(df, axis=None) for reductions like sum, mean, etc.

Getting there poses a backwards-compatibility challenge. Currently for some reduction functions like np.sum we interpret axis=None as

In [11]: df = pd.DataFrame({"A": [1, 2]})

In [12]: np.sum(df)
Out[12]:
A    3
dtype: int64

The best I option I see is to just warn that the behavior will change in the future, without providing a way to achieve the behavior today. Then users can update things like np.sum(df) to np.sum(df, axis=0). In a later version we'll make the actual change to interpret axis=None like NumPy.

[uds5501]

@TomAugspurger @gfyoung , I am interested in this issue and would like to help. But could you guys guide me on how to display these deprecate messages?

[gfyoung]

@uds5501 : Perhaps this will help?

import pandas.util.testing as tm
import warnings

def f():
   warnings.warn("This is a deprecation warning", DeprecationWarning)

def test_f():
   # Might need to also pass in `check_stacklevel=False`.
   with tm.assert_produces_warning(DeprecationWarning):
      f()

[uds5501]

@gfyoung Thanks for the help!
Now regarding this issue, will the warning in the functions like df.sum and df.mean etc : "The behavior 'df.op(axis = None)` is going to change in the future versions " suffice for this issue?

[gfyoung]

@uds5501 : Describe (briefly) how the behavior would change in the message.

[uds5501]

@gfyoung Alright sure, would submit a PR soon 😄

[uds5501]

@gfyoung I had one more query regarding this. If I am not wrong, we call Dataframe.sum from generic.py at core . So, am I supposed to place this error message in the function given below

pandas/pandas/core/generic.py

Lines 9551 to 9571 in 1a23779

	def _make_min_count_stat_function(cls, name, name1, name2, axis_descr, desc,
	f, examples):
	@Substitution(outname=name, desc=desc, name1=name1, name2=name2,
	axis_descr=axis_descr, min_count=_min_count_stub,
	examples=examples)
	@Appender(_num_doc)
	def stat_func(self, axis=None, skipna=None, level=None, numeric_only=None,
	min_count=0,
	**kwargs):
	nv.validate_stat_func(tuple(), kwargs, fname=name)
	if skipna is None:
	skipna = True
	if axis is None:
	axis = self._stat_axis_number
	if level is not None:
	return self._agg_by_level(name, axis=axis, level=level,
	skipna=skipna, min_count=min_count)
	return self._reduce(f, name, axis=axis, skipna=skipna,
	numeric_only=numeric_only, min_count=min_count)
	return set_function_name(stat_func, name, cls)

or make separate files for the same like these?

pandas/pandas/core/categorical.py

Lines 1 to 8 in 83e95e0

	import warnings
	# TODO: Remove after 0.23.x
	warnings.warn("'pandas.core' is private. Use 'pandas.Categorical'",
	FutureWarning, stacklevel=2)
	from pandas.core.arrays import Categorical # noqa
	from pandas.core.dtypes.dtypes import CategoricalDtype # noqa

A little guidance would be appreciated!

[gfyoung]

IINM, you should add it to generic.py.

cc @jreback

[TomAugspurger]

FYI @uds5501, this is a rather large change that needs discussion first. If you're interested in working on it, then by all means, but you might want to wait for feedback from other maintainers before diving into it.

[uds5501]

@TomAugspurger Sure sir! I have not yet begun either and conclusive discussions from maintainers who have better context regarding this are always welcome!

[jreback]

deprecation PR is #45072

[jreback]

@jbrockmendel if you can add this to the do-it-in-2.0 list

[rhshadrach]

Since the deprecation portion of this issue has been handled, I'm removing the deprecate label as it is now just an enhancement. I believe the enhancement portion still needs to be implemented.

[mroeschke]

After #50593, axis=None support is still needed for sum, prod, var, std, sem

[arv-77]

Hi @mroeschke , I would love to contribute to this issue. Can I work on implementing the remaining ops?

[jbrockmendel]

We suggest contributors start with an issue with the "good first issue" tag

[arv-77]

Ok, I'll start with some good first issue then. Thanks!

[jbrockmendel]

It looks like the remaining reductions didn't have the axis=None behavior deprecated, so we'll need to do a deprecation cycle and do the change in 3.0