Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Order Dependency between parquet tests and internals #19585

Closed
TomAugspurger opened this issue Feb 7, 2018 · 1 comment · Fixed by #19520
Closed

Test Order Dependency between parquet tests and internals #19585

TomAugspurger opened this issue Feb 7, 2018 · 1 comment · Fixed by #19520
Labels
Compat pandas objects compatability with Numpy or Python functions Testing pandas testing functions or related to the test suite Unreliable Test Unit tests that occasionally fail

Comments

@TomAugspurger
Copy link
Contributor

I observe this locally (py2 only)

pytest pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_basic  pandas/tests/internals/test_internals.py::test_deprecated_fastpath --tb=no -v
=========================================================== test session starts ============================================================
platform darwin -- Python 2.7.14, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /Users/taugspurger/miniconda3/envs/pandas-ip2/bin/python
cachedir: .cache
rootdir: /Users/taugspurger/sandbox/pandas-ip-2/pandas, inifile: setup.cfg
plugins: xdist-1.22.0, forked-0.2, hypothesis-3.38.5
collected 2 items

pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_basic PASSED                                                               [ 50%]
pandas/tests/internals/test_internals.py::test_deprecated_fastpath FAILED                                                            [100%]

==================================================== 1 failed, 1 passed in 0.35 seconds ====================================================

We apparently have an environment variable for controlling deprecation warnings.

PANDAS_TESTING_MODE=deprecate pytest pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_basic  panda
s/tests/internals/test_internals.py::test_deprecated_fastpath --tb=no -v
=========================================================== test session starts ============================================================
platform darwin -- Python 2.7.14, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /Users/taugspurger/miniconda3/envs/pandas-ip2/bin/python
cachedir: .cache
rootdir: /Users/taugspurger/sandbox/pandas-ip-2/pandas, inifile: setup.cfg
plugins: xdist-1.22.0, forked-0.2, hypothesis-3.38.5
collected 2 items

pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_basic PASSED                                                               [ 50%]
pandas/tests/internals/test_internals.py::test_deprecated_fastpath PASSED                                                            [100%]

========================================================= 2 passed in 0.13 seconds =========================================================

It'd be nice if the tests didn't rely on this.

@TomAugspurger TomAugspurger added Testing pandas testing functions or related to the test suite Compat pandas objects compatability with Numpy or Python functions Unreliable Test Unit tests that occasionally fail Difficulty Intermediate labels Feb 7, 2018
@TomAugspurger TomAugspurger added this to the Next Major Release milestone Feb 7, 2018
@TomAugspurger
Copy link
Contributor Author

TomAugspurger commented Feb 17, 2018

Something like this may work, but it'd be nice to clean things up quite a bit.

diff --git a/pandas/tests/io/test_parquet.py b/pandas/tests/io/test_parquet.py
index 11cbea8ce..69b651839 100644
--- a/pandas/tests/io/test_parquet.py
+++ b/pandas/tests/io/test_parquet.py
@@ -154,10 +154,46 @@ def check_round_trip(df, engine=None, path=None,
         write_kwargs['engine'] = engine
         read_kwargs['engine'] = engine
 
+    fastparquet_make_block_dtype = (
+        # Use of deprecated `dtype` in `make_block` that's hit only for
+        # bool dtypes with no Nones.
+        engine == 'fastparquet' and
+        LooseVersion(fastparquet.__version__) == LooseVersion("0.1.4") and
+        any(pd.api.types.is_bool_dtype(df[col]) for col in df.columns)
+    )
+
+    if (engine == 'pyarrow' and
+            LooseVersion(pyarrow.__version__) <= LooseVersion("0.8.0") and
+            any(pd.api.types.is_datetime64tz_dtype(dtype)
+                for dtype in df.dtypes)):
+        # Use of deprecated fastpath in make_block
+        # Deprecated in pandas 0.23 and removed in pyarrow 0.9
+        # Remove this when all pyarrow builds >= 0.9
+        warning_type = DeprecationWarning
+    # elif (engine == 'fastparquet' and
+    #         LooseVersion(fastparquet.__version__) == LooseVersion('0.1.3')):
+    #     warning_type = DeprecationWarning
+    elif (engine == 'fastparquet' and
+          LooseVersion(fastparquet.__version__) <= LooseVersion("0.1.4") and
+          LooseVersion(np.__version__) >= LooseVersion("1.14.0") and
+          df.select_dtypes(['bool', 'object'])
+            .isin([True, False]).any().any()):
+        # use of deprecated np.fromstring for boolean columns
+        # Deprecated in numpy 1.14
+        # Used in fastparquet <= 0.1.4
+        # Remove when all fastparquet builds >= 0.1.5
+        # https://github.com/dask/fastparquet/issues/302
+        warning_type = DeprecationWarning
+    elif fastparquet_make_block_dtype:
+        warning_type = DeprecationWarning
+    else:
+        warning_type = None
+
     def compare(repeat):
         for _ in range(repeat):
             df.to_parquet(path, **write_kwargs)
-            with catch_warnings(record=True):
+            with tm.assert_produces_warning(warning_type,
+                                            check_stacklevel=False):
                 actual = read_parquet(path, **read_kwargs)
             tm.assert_frame_equal(expected, actual,
                                   check_names=check_names)
@@ -224,7 +260,17 @@ def test_cross_engine_pa_fp(df_cross_compat, pa, fp):
     with tm.ensure_clean() as path:
         df.to_parquet(path, engine=pa, compression=None)
 
-        result = read_parquet(path, engine=fp)
+        if (LooseVersion(fastparquet.__version__) <= LooseVersion('0.1.4') and
+                LooseVersion(np.__version__) >= LooseVersion('1.14.0')):
+            # fastparquet used np.fromstring, deprecated in numpy 1.14.0
+            expected_warning = DeprecationWarning
+        else:
+            expected_warning = None
+
+        with tm.assert_produces_warning(expected_warning,
+                                        check_stacklevel=False):
+            result = read_parquet(path, engine=fp)
+
         tm.assert_frame_equal(result, df)
 
         result = read_parquet(path, engine=fp, columns=['a', 'd'])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Compat pandas objects compatability with Numpy or Python functions Testing pandas testing functions or related to the test suite Unreliable Test Unit tests that occasionally fail
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant