Add the steps to setup gbq integration testing to the contributing docs #14144

parthea · 2016-09-03T13:43:30Z

Pull requests from forked repositories cannot access the existing secure Google BigQuery credentials which are on Travis at pydata/pandas. In order for contributors to run Google BigQuery integration tests on Travis, they need to create a pull request on their forked repository of pandas with their own secure Google BigQuery credentials.

I've updated the contributing documentation to include the steps required to run Google BigQuery integration tests on a forked repository of pandas.

jreback · 2016-09-03T14:02:14Z

any easier way to do this?

codecov-io · 2016-09-03T14:10:17Z

Current coverage is 85.25% (diff: 100%)

Merging #14144 into master will increase coverage by <.01%

@@             master     #14144   diff @@
==========================================
  Files           139        139          
  Lines         50496      50502     +6   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43049      43058     +9   
+ Misses         7447       7444     -3   
  Partials          0          0

Powered by Codecov. Last update 8023029...6383bba

parthea · 2016-09-03T14:11:34Z

any easier way to do this?

The easiest way is to use the existing encrypted credentials that are already on Travis, but we would need to create a branch from master each time. If we don't use existing credentials, then we need to ask contributors to create their own encrypted credentials. Should I consolidate text and reduce the number of steps? (For example, assume that users already know how to create BigQuery credentials in JSON format?)

jreback · 2016-09-03T14:17:57Z

The easiest way is to use the existing encrypted credentials that are already on Travis, but we would need to create a branch from master each time.

what does this mean? most branches are forks of master

parthea · 2016-09-03T14:25:09Z

Sorry that I wasn't clear. I meant to say that we would need to create a branch from pydata/pandas which is not on a fork. The credentials can only be encrypted/decrypted from the same repo (not fork).

For example:
The credentials can be decrypted when the PR shows:
parthea wants to merge 2 commits into master from enable-gbq-unit-tests
but cannot be decrypted when the PR shows:
parthea wants to merge 2 commits into pydata:master from parthea:enable-gbq-unit-tests

jreback · 2016-09-03T14:42:38Z

ok I c. another (maybe easier option), is to have the travis secure key embedded in a file, then we could have a script which a user could run to take there credentials and update the encoded credentials file and the key very easily. (and of course make sure to exclude them from being added to the repo).

jreback · 2016-09-03T14:44:07Z

is this the 'name' of the secure key that we have? "$encrypted_1d9d7b1f171b_iv"

parthea · 2016-09-03T14:51:05Z

Yes, that is the name of the environment variables encrypted_1d9d7b1f171b_iv and encrypted_1d9d7b1f171b_key which contain the private iv and key, which is only available on the travis repo that encrypted the file. Each repo has a different environment variable.

parthea · 2016-09-03T14:53:48Z

You're right. It makes sense to automate these steps. I'll put together a script in order to reduce the steps.

jorisvandenbossche · 2016-09-04T11:13:39Z

@parthea Good to document this!

Some general comments:

can you clarify that we only expect to do this from contributors if they are doing a PR related to the gbq module
I would make it a separate section (and maybe put it at the end), and then refer to this section from the sections on running tests

jreback · 2016-09-04T11:16:11Z

this is not mergeable ATM / see above comments

jreback · 2016-09-05T16:08:05Z

ci/encrypt_gbq.sh

+read -d "\n" TRAVIS_KEY TRAVIS_IV <<<$(travis encrypt-file $GBQ_JSON_FILE \
+travis_gbq.json.enc -f | grep -o "\w*_iv\|\w*_key");
+
+echo "Adding your secure key to .travis.yml ..."


It would be better to move this check OUT of .travis.yml and actually put in a separate (yaml or txt) type of file, then you don't have to do all of this crazy sub/replace stuff, just have a file like

TRAVIS_IV=.... GBQ_PROJECT_ID=...

then simply re-write the file

then .travis.yml section is:

ci\travis_process_gbq_encryption.sh

ci\travis_process_gbq_encryption.sh

# read the variables / project id from the file - if [ -n '$encrypted_f44557bb53ea_iv' ]; then openssl aes-256-cbc -K $encrypted_f44557bb53ea_key -iv $encrypted_f44557bb53ea_iv -in ci/travis_gbq.json.enc -out ci/travis_gbq.json -d; export GBQ_PROJECT_ID='pandas-140401';

parthea · 2016-09-06T03:39:42Z

@jreback @jorisvandenbossche All gbq tests passed at parthea#3. I've moved gbq integration testing to a separate section in the contributing docs. Please take a look.

jreback · 2016-09-06T10:49:03Z

lgtm.

jorisvandenbossche · 2016-09-06T13:55:08Z

doc/source/contributing.rst

+tests are `encrypted <https://docs.travis-ci.com/user/encrypting-files/>`__
+on Travis-CI and are only accessible from the pydata/pandas repository. In
+addition to creating a pull request on pydata/pandas, contributors should also
+create a pull request on their forked repository and add encrypted credentials


Is it needed to create a PR on your fork?

@jorisvandenbossche No, it is not needed. Thanks for catching that! I've updated the instructions. Please take another look.

jorisvandenbossche · 2016-09-07T07:27:21Z

doc/source/contributing.rst

+
+#. Create a new branch from the branch used in your pull request. Commit the
+   encrypted file called ``travis_gbq.json.enc`` as well as the file
+   ``travis_gbq_config.txt``, in an otherwise empty commit. DO NOT commit the


Isn't this file already committed? Or will it change due to the script you call?

You are correct. The file is already committed. Both travis_gbq.json.enc and travis_gbq_config.txt will change when users run ``/travis_encrypt_gbq.sh...` so we need to tell users to commit those files (or changes) in a new branch.

jorisvandenbossche · 2016-09-07T07:37:10Z

Small further question, otherwise looks good!

parthea · 2016-09-07T13:21:48Z

Here is a link to the build from my branch which has my encrypted credentials so that gbq test are not skipped. https://travis-ci.org/parthea/pandas/jobs/158060278

jorisvandenbossche · 2016-09-07T13:23:03Z

@parthea Nice, thanks a lot!

…eter * github.com:pydata/pandas: (554 commits) BUG: compat with Stata ver 111 Fix: F999 dictionary key '2000q4' repeated with different values (pandas-dev#14198) BLD: Test for Python 3.5 with C locale BUG: DatetimeTZBlock can't assign values near dst boundary BUG: union_categorical with Series and cat idx BUG: fix str.contains for series containing only nan values BUG: Categorical constructor not idempotent with ext dtype TST: Make encoded sep check more locale sensitive (pandas-dev#14161) DOC: minor typo in 0.19.0 whatsnew file (pandas-dev#14185) BUG: fix tz-aware datetime convert to DatetimeIndex (GH 14088) BUG : bug in setting a slice of a Series with a np.timedelta64 RLS: v0.19.0rc1 DOC: clean-up 0.19.0 whatsnew file (pandas-dev#14176) DOC: cleanup build warnings (pandas-dev#14172) Add steps to run gbq integration testing to the contributing docs (pandas-dev#14144) ENH: concat and append now can handle unordered categories (pandas-dev#13767) DEPR: Deprecate pandas.core.datetools (pandas-dev#14105) API/DEPR: Remove +/- as setops for DatetimeIndex/PeriodIndex (GH9630) (pandas-dev#14164) Fix trivial typo in comment (pandas-dev#14174) API/DEPR: Remove +/- as setops for Index (GH8227) (pandas-dev#14127) ...

* github.com:pydata/pandas: (554 commits) BUG: compat with Stata ver 111 Fix: F999 dictionary key '2000q4' repeated with different values (pandas-dev#14198) BLD: Test for Python 3.5 with C locale BUG: DatetimeTZBlock can't assign values near dst boundary BUG: union_categorical with Series and cat idx BUG: fix str.contains for series containing only nan values BUG: Categorical constructor not idempotent with ext dtype TST: Make encoded sep check more locale sensitive (pandas-dev#14161) DOC: minor typo in 0.19.0 whatsnew file (pandas-dev#14185) BUG: fix tz-aware datetime convert to DatetimeIndex (GH 14088) BUG : bug in setting a slice of a Series with a np.timedelta64 RLS: v0.19.0rc1 DOC: clean-up 0.19.0 whatsnew file (pandas-dev#14176) DOC: cleanup build warnings (pandas-dev#14172) Add steps to run gbq integration testing to the contributing docs (pandas-dev#14144) ENH: concat and append now can handle unordered categories (pandas-dev#13767) DEPR: Deprecate pandas.core.datetools (pandas-dev#14105) API/DEPR: Remove +/- as setops for DatetimeIndex/PeriodIndex (GH9630) (pandas-dev#14164) Fix trivial typo in comment (pandas-dev#14174) API/DEPR: Remove +/- as setops for Index (GH8227) (pandas-dev#14127) ...

parthea force-pushed the add-gbq-integration-test-setup-to-docs branch 2 times, most recently from d202284 to 2628574 Compare September 3, 2016 13:52

parthea mentioned this pull request Sep 3, 2016

TST: Enable Google BigQuery (pandas.io.gbq) integration testing #11089 #14111

Closed

parthea force-pushed the add-gbq-integration-test-setup-to-docs branch 3 times, most recently from 063c454 to a99dd67 Compare September 3, 2016 14:01

parthea force-pushed the add-gbq-integration-test-setup-to-docs branch from a99dd67 to f5d4c0f Compare September 3, 2016 14:07

parthea force-pushed the add-gbq-integration-test-setup-to-docs branch from f5d4c0f to 5c5541e Compare September 3, 2016 14:13

jreback added the IO Google label Sep 3, 2016

parthea force-pushed the add-gbq-integration-test-setup-to-docs branch 4 times, most recently from 3fdaa6d to df1eb02 Compare September 4, 2016 17:08

parthea mentioned this pull request Sep 4, 2016

Add gbq integration test setup to docs parthea/pandas#3

Closed

parthea force-pushed the add-gbq-integration-test-setup-to-docs branch 3 times, most recently from a6448c3 to 2a9a830 Compare September 5, 2016 16:00

jreback reviewed Sep 5, 2016
View reviewed changes

parthea force-pushed the add-gbq-integration-test-setup-to-docs branch from 2a9a830 to 731bad3 Compare September 5, 2016 18:49

parthea force-pushed the add-gbq-integration-test-setup-to-docs branch 5 times, most recently from 6383bba to bb2e2b6 Compare September 6, 2016 01:42

jreback added this to the 0.19.0 milestone Sep 6, 2016

jorisvandenbossche reviewed Sep 6, 2016
View reviewed changes

Add steps to run gbq integration testing to the contributing docs

79f790a

parthea force-pushed the add-gbq-integration-test-setup-to-docs branch from bb2e2b6 to 79f790a Compare September 7, 2016 02:51

jorisvandenbossche reviewed Sep 7, 2016
View reviewed changes

jorisvandenbossche merged commit 9b7efd6 into pandas-dev:master Sep 7, 2016

parthea deleted the add-gbq-integration-test-setup-to-docs branch September 7, 2016 13:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the steps to setup gbq integration testing to the contributing docs #14144

Add the steps to setup gbq integration testing to the contributing docs #14144

parthea commented Sep 3, 2016

jreback commented Sep 3, 2016

codecov-io commented Sep 3, 2016 •

edited

Loading

parthea commented Sep 3, 2016 •

edited

Loading

jreback commented Sep 3, 2016

parthea commented Sep 3, 2016 •

edited

Loading

jreback commented Sep 3, 2016

jreback commented Sep 3, 2016

parthea commented Sep 3, 2016 •

edited

Loading

parthea commented Sep 3, 2016

jorisvandenbossche commented Sep 4, 2016

jreback commented Sep 4, 2016

jreback Sep 5, 2016

parthea Sep 5, 2016

parthea commented Sep 6, 2016

jreback commented Sep 6, 2016

jorisvandenbossche Sep 6, 2016

parthea Sep 7, 2016 •

edited

Loading

jorisvandenbossche Sep 7, 2016

parthea Sep 7, 2016

jorisvandenbossche Sep 7, 2016

jorisvandenbossche commented Sep 7, 2016

parthea commented Sep 7, 2016

jorisvandenbossche commented Sep 7, 2016

Add the steps to setup gbq integration testing to the contributing docs #14144

Add the steps to setup gbq integration testing to the contributing docs #14144

Conversation

parthea commented Sep 3, 2016

jreback commented Sep 3, 2016

codecov-io commented Sep 3, 2016 • edited Loading

Current coverage is 85.25% (diff: 100%)

parthea commented Sep 3, 2016 • edited Loading

jreback commented Sep 3, 2016

parthea commented Sep 3, 2016 • edited Loading

jreback commented Sep 3, 2016

jreback commented Sep 3, 2016

parthea commented Sep 3, 2016 • edited Loading

parthea commented Sep 3, 2016

jorisvandenbossche commented Sep 4, 2016

jreback commented Sep 4, 2016

jreback Sep 5, 2016

Choose a reason for hiding this comment

parthea Sep 5, 2016

Choose a reason for hiding this comment

parthea commented Sep 6, 2016

jreback commented Sep 6, 2016

jorisvandenbossche Sep 6, 2016

Choose a reason for hiding this comment

parthea Sep 7, 2016 • edited Loading

Choose a reason for hiding this comment

jorisvandenbossche Sep 7, 2016

Choose a reason for hiding this comment

parthea Sep 7, 2016

Choose a reason for hiding this comment

jorisvandenbossche Sep 7, 2016

Choose a reason for hiding this comment

jorisvandenbossche commented Sep 7, 2016

parthea commented Sep 7, 2016

jorisvandenbossche commented Sep 7, 2016

codecov-io commented Sep 3, 2016 •

edited

Loading

parthea commented Sep 3, 2016 •

edited

Loading

parthea commented Sep 3, 2016 •

edited

Loading

parthea commented Sep 3, 2016 •

edited

Loading

parthea Sep 7, 2016 •

edited

Loading