DOC: Different behaviour with respect to Series.all docs when NA values are present

[galipremsagar]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

In[29]: s = pd.Series([None, None, None], dtype=pd.BooleanDtype())
In[30]: s
Out[30]: 
0    <NA>
1    <NA>
2    <NA>
dtype: boolean
In[31]: s.all(skipna=True)
Out[31]: True
In[32]: s.all(skipna=False)
Out[32]: <NA>

Problem description

The documentation says:

If skipna is False, then NA are treated as True, because these are not equal to zero.

But looks like a pd.NA value is being returned instead.

Expected Output

The expected behavior is the result of all should be True if we go by the docs or if this was a conscious design decision to return NA as scalar we should be updating the docs to reflect the same.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2a7d332
python : 3.7.8.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-48-generic
Version : #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.1.2
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 49.6.0.post20200917
Cython : 0.29.21
pytest : 6.0.2
hypothesis : 5.28.0
sphinx : 3.2.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 0.8.2
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 1.0.1
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.51.2

[dsaxton]

To me it seems like the output is correct here and the docs should be updated

[jorisvandenbossche]

Indeed, the behaviour for nullable dtypes is as expected, so the docs need an update. The complicating factor, however, is that the current explanation is correct if you have NaN missing data, but the behaviour is thus different with NA missing data.

[junjunjunk]

It seems to me that we need to explicitly describe the differences in behavior due to missing values.
I discovered that "None" as object also behave differently as below.

import pandas as pd
s = pd.Series([None, None, None], dtype='object')
s.all(skipna=False) # no output

[jorisvandenbossche]

I discovered that "None" as object also behave differently as below.

Yeah, any/all are kind of broken with object dtype .. That's due to how numpy works with object dtype, see eg #27709 and #12863

[jaraqueffdc]

Hi! I was going to open an issue for this but I just noticed this one already existed although is marked as a doc update. I am not sure this is only an issue with documentation since this shows undefined behaviour depending on the case. I noticed this issue when testing that a model produces a column with a given value. When the model registers the result it converts a given column to StringDType. This is a toy example doing something similar to the test.

dd = pd.DataFrame({"a": [pd.NA, pd.NA]}, dtype=pd.StringDtype())
dd["b"] = dd["a"] == "hi"
dd["b"]

# 0    <NA>
# 1    <NA>
# Name: b, dtype: boolean

At this point I was surprised because pd.NA == "hi" does not return False but pd.NA for the BooleanDtype type extension. I can understand why one would say that if the value is not defined the comparison is not defined but I would argue this is an unexpected behaviour. I would expect that if the are not equal the equality comparison returns False. This is how None behaves for instance, if we do None == "hi" we don't get None in return.

In any case, I was thinking that ok, if it returns pd.NA it should be ok as those seemed falsy and the all method would return False. But this is not the case, as this issue is reporting.

We could pass skipna=False to avoid all() returning True but then the all() method would return pd.NAand since there is no boolean conversion forpd.NA` we cannot assert over it. This means that there is no clean way to write an assertion on this column

dd = pd.DataFrame({"a": [pd.NA, pd.NA]}, dtype=pd.StringDtype())
dd["b"] = dd["a"] == "hi"
# This will be executed but it will pass even though there is no single value set to "hi"
assert dd["b"].all() 
# This will raise an exception because we cannot assert over `pd.NA`
assert dd["b"].all(skipna=False)  

I would say this is a very unexpected behaviour of the pd.NA values. At least when comparing with scalar it should return False instead of pd.NA since they are not the same.

This behaviour goes away of there is at least one value that is not NA.

dd = pd.DataFrame({"a": [pd.NA, pd.NA, "Value"]}, dtype=pd.StringDtype())
dd["b"] = dd["a"] == "hi"
dd["b"].all()
# False