BUG: DataFrame reductions dtypes on object input

[rhshadrach]

• closes BUG: DataFrame reductions with object dtype and axis=1 #49603 (Replace xxxx with the GitHub issue number)
• closes BUG: DataFrame.min(axis=1) raises FutureWarning for timezone aware datetimes and returns wrong results #51242
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[jbrockmendel]

does NA here refer to pd.NaT?

[rhshadrach]

Thanks - will fix.

[jbrockmendel]

hmm .values can be expensive, might be better to reduce twice? (... which can also be expensive. darn). Is punting on this viable?

[rhshadrach]

Using .values is the current behavior on main if numeric_only is False and axis is not 0.

pandas/pandas/core/frame.py

Lines 10494 to 10496 in c7fa611

	data = self
	values = data.values
	result = func(values)

In fact, main is currently broken when numeric_only is True and axis is None:

df = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5], 'c': list('xyz')})
result = df.mean(axis=None, numeric_only=True)
print(result)
# a    1.333333
# b    4.000000
# dtype: float64

result2 = df[['a', 'b']].mean(axis=None)
print(result2)
# 2.6666666666666665

Assuming this is okay for now, I can add a test for when numeric_only is False and axis is None.

[rhshadrach]

@jbrockmendel - friendly ping

[jbrockmendel]

result_type.type("nan")?

[jbrockmendel]

except for the ArrayManager thing, this LGTM

[jbrockmendel]

LGTM

[mroeschke]

Thanks @rhshadrach

[jbrockmendel]

@rhshadrach looks like this caused a regression in https://asv-runner.github.io/asv-collection/pandas/#stat_ops.FrameOps.time_op?p-op='mean'&p-dtype='Int64'&p-axis=1&commits=b836a88f for nullable cases with axis=1. Can you take a look?

[rhshadrach]

1.5.3 is dropping down to the underlying NumPy array and calling the op with axis=1. main is computing the transpose of the DataFrame (which is inefficient for EAs), then computing the sum via the Block manager (which I think is inefficient for EAs with wide data).

I'm still looking for a better alternatively currently, but likely I expect to need to largely restore the previous implementation and defer a better solution for 2.1.

[jbrockmendel]

Is it a correctness vs perf tradeoff? If so correctness wins every time.

Might be able to do something similar to _reduce_axis1 to avoid .values/transpose

[rhshadrach]

Is it a correctness vs perf tradeoff? If so correctness wins every time.

I think I can get every test to pass by using a combination of 1.5.x's _reduce and that in this PR except for test_minmax_tzaware_skipna_axis_1. For tzaware with NaT, dropping down to NumPy arrays is insufficient to compute the op. I don't see a way to fix this in any easy manner and avoid the perf regression. I think (but haven't confirmed yet) that this was also broken in 1.5.x; is it worth the perf hit for this?

Longer term (for 2.1), it does seem to me doing something like _reduce_axis1 is the way to go. For example, here is the ASV that identified this regression (using 1.5.x, so before the perf regression occurred; I'm leaving out the setup here) against a direct implementation of mean for axis=1:

%timeit df.mean(axis=1)
18.5 ms ± 40.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit sum(df[c] * (1/df.shape[1]) for c in df)
778 µs ± 1.67 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

[jbrockmendel]

Any interest in trying to get this improved for 2.0?

[jorisvandenbossche]

For the case of masked arrays (the specific case from the asv regression), the transpose spends most of the time in reconstructing an EA for each row, which spends a lot of time in _coerce_to_data_and_mask. There is a lot of room for improvement in that function, if we would want to do that (eg it's doing a lot of dtype checks, and we know that those are slow if you do those repeatedly many times, in some other (critical) paths we switched to dtype.kind == .. checks, IIRC)

Improving the constructor might be worth anyway, but I agree that for the specific case of the numeric reduction with axis=1, doing a special case like _reduce_axis1 already does for any/all seems quite easy to do without being very complex.
Because however we improve the constructor, constructing 10,000x a small EA in case of 10,000 rows DataFrame is just always going to be slower.

[rhshadrach]

Might be able to do something similar to _reduce_axis1 to avoid .values/transpose

I looked into this - I think there are some common ops / dtypes we can do this for, but we can't do it for all ops (e.g. median) and all dtypes (e.g. 3rd party EAs since I think the implementation will depend on knowing the value of the neutral element for an op like sum or prod).

A straightforward way to fix this regression is to mostly revert this PR, where we still return object dtype on object input but otherwise the results are unchanged. I can put up a PR for this approach if #51923 isn't the right way forward or is too much to try to get it into 2.0.