Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST/CI: Unify CI test scripts (single and multi) and simplify the system #23924

Merged
merged 8 commits into from
Nov 30, 2018
Merged

TST/CI: Unify CI test scripts (single and multi) and simplify the system #23924

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
252729c
WIP: unifying script_single and script_multi
datapythonista Nov 26, 2018
be9fb9b
Merging from master
datapythonista Nov 26, 2018
12bb059
Unified scripts that run the tests (single and multi)
datapythonista Nov 26, 2018
9fac328
pending fixes
datapythonista Nov 26, 2018
93c41db
logging when we upload the coverage
datapythonista Nov 26, 2018
18446b2
Merge remote-tracking branch 'upstream/master' into unify_test_scripts
datapythonista Nov 29, 2018
a17a9ce
Restoring pytest --skip-slow... options
datapythonista Nov 29, 2018
8c9b557
Not running tests on doc build
datapythonista Nov 29, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ jobs:
command: |
export PATH="$MINICONDA_DIR/bin:$PATH"
source activate pandas-dev
echo "pytest --strict --durations=10 --color=no --junitxml=$CIRCLE_TEST_REPORTS/reports/junit.xml --skip-slow --skip-network pandas"
pytest --strict --durations=10 --color=no --junitxml=$CIRCLE_TEST_REPORTS/reports/junit.xml --skip-slow --skip-network pandas
echo "pytest -m "not slow and not network" --strict --durations=10 --color=no --junitxml=$CIRCLE_TEST_REPORTS/reports/junit.xml pandas"
pytest -m "not slow and not network" --strict --durations=10 --color=no --junitxml=$CIRCLE_TEST_REPORTS/reports/junit.xml pandas
24 changes: 10 additions & 14 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,28 +34,28 @@ matrix:
include:
- dist: trusty
env:
- JOB="3.7" ENV_FILE="ci/deps/travis-37.yaml" TEST_ARGS="--skip-slow --skip-network"
- JOB="3.7" ENV_FILE="ci/deps/travis-37.yaml" PATTERN="not slow and not network"

- dist: trusty
env:
- JOB="2.7, locale, slow, old NumPy" ENV_FILE="ci/deps/travis-27-locale.yaml" LOCALE_OVERRIDE="zh_CN.UTF-8" SLOW=true
- JOB="2.7, locale, slow, old NumPy" ENV_FILE="ci/deps/travis-27-locale.yaml" LOCALE_OVERRIDE="zh_CN.UTF-8" PATTERN="slow"
addons:
apt:
packages:
- language-pack-zh-hans
- dist: trusty
env:
- JOB="2.7" ENV_FILE="ci/deps/travis-27.yaml" TEST_ARGS="--skip-slow"
- JOB="2.7" ENV_FILE="ci/deps/travis-27.yaml" PATTERN="not slow"
addons:
apt:
packages:
- python-gtk2
- dist: trusty
env:
- JOB="3.6, lint, coverage" ENV_FILE="ci/deps/travis-36.yaml" TEST_ARGS="--skip-slow --skip-network" PANDAS_TESTING_MODE="deprecate" COVERAGE=true LINT=true
- JOB="3.6, lint, coverage" ENV_FILE="ci/deps/travis-36.yaml" PATTERN="not slow and not network" PANDAS_TESTING_MODE="deprecate" COVERAGE=true LINT=true
- dist: trusty
env:
- JOB="3.7, NumPy dev" ENV_FILE="ci/deps/travis-37-numpydev.yaml" TEST_ARGS="--skip-slow --skip-network -W error" PANDAS_TESTING_MODE="deprecate"
- JOB="3.7, NumPy dev" ENV_FILE="ci/deps/travis-37-numpydev.yaml" PATTERN="not slow and not network" TEST_ARGS="-W error" PANDAS_TESTING_MODE="deprecate"
addons:
apt:
packages:
Expand All @@ -64,7 +64,7 @@ matrix:
# In allow_failures
- dist: trusty
env:
- JOB="3.6, slow" ENV_FILE="ci/deps/travis-36-slow.yaml" SLOW=true
- JOB="3.6, slow" ENV_FILE="ci/deps/travis-36-slow.yaml" PATTERN="slow"

# In allow_failures
- dist: trusty
Expand All @@ -73,7 +73,7 @@ matrix:
allow_failures:
- dist: trusty
env:
- JOB="3.6, slow" ENV_FILE="ci/deps/travis-36-slow.yaml" SLOW=true
- JOB="3.6, slow" ENV_FILE="ci/deps/travis-36-slow.yaml" PATTERN="slow"
- dist: trusty
env:
- JOB="3.6, doc" ENV_FILE="ci/deps/travis-36-doc.yaml" DOC=true
Expand Down Expand Up @@ -107,20 +107,16 @@ script:
- echo "script start"
- source activate pandas-dev
- ci/run_build_docs.sh
- ci/script_single.sh
- ci/script_multi.sh
- ci/run_tests.sh
- ci/code_checks.sh

after_success:
- ci/upload_coverage.sh

after_script:
- echo "after_script start"
- source activate pandas-dev && pushd /tmp && python -c "import pandas; pandas.show_versions();" && popd
- if [ -e test-data-single.xml ]; then
ci/print_skipped.py test-data-single.xml;
ci/print_skipped.py test-data-single.xml;
fi
- if [ -e test-data-multiple.xml ]; then
ci/print_skipped.py test-data-multiple.xml;
ci/print_skipped.py test-data-multiple.xml;
fi
- echo "after_script done"
10 changes: 4 additions & 6 deletions ci/azure/linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,18 @@ jobs:
py27_np_120:
ENV_FILE: ci/deps/azure-27-compat.yaml
CONDA_PY: "27"
TEST_ARGS: "--skip-slow --skip-network"
PATTERN: "not slow and not network"

py37_locale:
ENV_FILE: ci/deps/azure-37-locale.yaml
CONDA_PY: "37"
TEST_ARGS: "--skip-slow --skip-network"
PATTERN: "not slow and not network"
LOCALE_OVERRIDE: "zh_CN.UTF-8"

py36_locale_slow:
ENV_FILE: ci/deps/azure-36-locale_slow.yaml
CONDA_PY: "36"
TEST_ARGS: "--only-slow --skip-network"
PATTERN: "not slow and not network"
LOCALE_OVERRIDE: "it_IT.UTF-8"

steps:
Expand All @@ -43,9 +43,7 @@ jobs:
- script: |
export PATH=$HOME/miniconda3/bin:$PATH
source activate pandas-dev
ci/script_single.sh
ci/script_multi.sh
echo "[Test done]"
ci/run_tests.sh
displayName: 'Test'
- script: |
export PATH=$HOME/miniconda3/bin:$PATH
Expand Down
6 changes: 2 additions & 4 deletions ci/azure/macos.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
py35_np_120:
ENV_FILE: ci/deps/azure-macos-35.yaml
CONDA_PY: "35"
TEST_ARGS: "--skip-slow --skip-network"
PATTERN: "not slow and not network"

steps:
- script: |
Expand All @@ -31,9 +31,7 @@ jobs:
- script: |
export PATH=$HOME/miniconda3/bin:$PATH
source activate pandas-dev
ci/script_single.sh
ci/script_multi.sh
echo "[Test done]"
ci/run_tests.sh
displayName: 'Test'
- script: |
export PATH=$HOME/miniconda3/bin:$PATH
Expand Down
2 changes: 1 addition & 1 deletion ci/azure/windows-py27.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
displayName: 'Build'
- script: |
call activate pandas-dev
pytest --junitxml=test-data.xml --skip-slow --skip-network pandas -n 2 -r sxX --strict --durations=10 %*
pytest -m "not slow and not network" --junitxml=test-data.xml pandas -n 2 -r sxX --strict --durations=10 %*
displayName: 'Test'
- task: PublishTestResults@2
inputs:
Expand Down
2 changes: 1 addition & 1 deletion ci/azure/windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
displayName: 'Build'
- script: |
call activate pandas-dev
pytest --junitxml=test-data.xml --skip-slow --skip-network pandas -n 2 -r sxX --strict --durations=10 %*
pytest -m "not slow and not network" --junitxml=test-data.xml pandas -n 2 -r sxX --strict --durations=10 %*
displayName: 'Test'
- task: PublishTestResults@2
inputs:
Expand Down
47 changes: 47 additions & 0 deletions ci/run_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#!/bin/bash

# Workaround for pytest-xdist flaky collection order
# https://github.com/pytest-dev/pytest/issues/920
# https://github.com/pytest-dev/pytest/issues/1075
export PYTHONHASHSEED=$(python -c 'import random; print(random.randint(1, 4294967295))')

if [ -n "$LOCALE_OVERRIDE" ]; then
export LC_ALL="$LOCALE_OVERRIDE"
export LANG="$LOCALE_OVERRIDE"
PANDAS_LOCALE=`python -c 'import pandas; pandas.get_option("display.encoding")'`
if [[ "$LOCALE_OVERIDE" != "$PANDAS_LOCALE" ]]; then
echo "pandas could not detect the locale. System locale: $LOCALE_OVERRIDE, pandas detected: $PANDAS_LOCALE"
# TODO Not really aborting the tests until https://github.com/pandas-dev/pandas/issues/23923 is fixed
# exit 1
fi
fi
if [[ "not network" == *"$PATTERN"* ]]; then
export http_proxy=http://1.2.3.4 https_proxy=http://1.2.3.4;
fi


if [ -n "$PATTERN" ]; then
PATTERN=" and $PATTERN"
fi

for TYPE in single multiple
do
if [ "$COVERAGE" ]; then
COVERAGE_FNAME="/tmp/coc-$TYPE.xml"
COVERAGE="-s --cov=pandas --cov-report=xml:$COVERAGE_FNAME"
fi

TYPE_PATTERN=$TYPE
NUM_JOBS=1
if [[ "$TYPE_PATTERN" == "multiple" ]]; then
TYPE_PATTERN="not single"
NUM_JOBS=2
fi

pytest -m "$TYPE_PATTERN$PATTERN" -n $NUM_JOBS -s --strict --durations=10 --junitxml=test-data-$TYPE.xml $TEST_ARGS $COVERAGE pandas
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you have TEST_ARGS here, though you removed it elsewhere

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to get rid of it, but it's needed in the numpydev build for the -W error

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any other way to do that. this makes it pretty clunky.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to take a look at pytest concurrency in a different PR.

I think it should be a simpler way to avoid running the simple tests in parallel, without having to call pytest twice. I think that would be cleaner, and we can simplify all this even more.

Also, I want to see how many processes we have, and not call always for 2 (which I assume it's the number of cores in Travis), but instead use all the available (I think that should be as easy as -n auto, but I want to take a look at how many cores are detected in azure, and how many makes sense to use).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the single tests must remain though
we got failures because these are using single system resources


if [[ "$COVERAGE" && $? == 0 ]]; then
echo "uploading coverage for $TYPE tests"
bash <(curl -s https://codecov.io/bash) -Z -c -F $TYPE -f $COVERAGE_FNAME
fi
done
49 changes: 0 additions & 49 deletions ci/script_multi.sh
View file
Open in desktop

This file was deleted.

39 changes: 0 additions & 39 deletions ci/script_single.sh
View file
Open in desktop

This file was deleted.

11 changes: 0 additions & 11 deletions ci/upload_coverage.sh
View file
Open in desktop

This file was deleted.