ENH: Add table_schema parameter for user-defined BigQuery schema #46

ghost · 2017-06-03T11:32:40Z

Closes #44

codecov-io · 2017-06-03T11:57:22Z

Codecov Report

Merging #46 into master will decrease coverage by 0.95%.
The diff coverage is 16%.

@@            Coverage Diff             @@
##           master      #46      +/-   ##
==========================================
- Coverage   28.49%   27.54%   -0.96%     
==========================================
  Files           4        4              
  Lines        1537     1561      +24     
==========================================
- Hits          438      430       -8     
- Misses       1099     1131      +32

Impacted Files	Coverage Δ
pandas_gbq/gbq.py	`18.61% <0%> (-0.08%)`	⬇️
pandas_gbq/tests/test_gbq.py	`26.5% <17.39%> (-1.78%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 183daf1...4cfc00b. Read the comment docs.

jreback

lgtm. can you add an entry in doc/source/changelog.rst for 0.2.0

jreback · 2017-06-04T21:07:00Z

pandas_gbq/gbq.py

+        List of BigQuery table fields to which according DataFrame columns
+        conform to, e.g. `[{'name': 'col1', 'type': 'STRING'},...]`. If
+        schema is not provided, it will be generated according to dtypes
+        of DataFrame columns.


add a versionadded tag. are there required names in the schema (e.g. name, type)? list what is required.

jreback · 2017-06-04T21:07:23Z

pandas_gbq/gbq.py

+    if not table_schema:
+        table_schema = _generate_bq_schema(dataframe)
+    else:
+        table_schema = dict(fields=table_schema)


assume validation is now up to BQ. can you test this though?

Current implementation will throw StreamingInsetError after a chunk is done (see tests) along printing error trace from the BQ API. Which is OK.

jreback · 2017-06-04T21:07:33Z

pandas_gbq/tests/test_gbq.py

@@ -1258,6 +1258,34 @@ def test_verify_schema_ignores_field_mode(self):
        assert self.sut.verify_schema(
            self.dataset_prefix + "1", TABLE_ID + test_id, test_schema_2)

+    def test_upload_data_with_valid_user_schema(self):
+        df = tm.makeMixedDataFrame()


add the issue number as a comment

jreback · 2017-06-04T21:08:24Z

pandas_gbq/tests/test_gbq.py

+                                        dict(fields=test_schema))
+
+    def test_upload_data_with_invalid_user_schema_raises_error(self):
+        df = tm.makeMixedDataFrame()


can you also tests with missing keys in the schema

Test added.

jreback · 2017-06-04T21:08:59Z

cc @parthea

jreback · 2017-06-04T21:10:18Z

once this is merged, will need a PR to pandas itself to modify the signature of to_gbq (for 0.21.0). (or maybe will just add **kwargs to it).

ghost · 2017-06-11T14:24:48Z

This can be re-reviewed now.

jreback

pls add an entry in changelog (0.2.0). minor comments. ping when pushed / green.

jreback · 2017-06-11T14:30:43Z

pandas_gbq/gbq.py

@@ -815,6 +816,13 @@ def to_gbq(dataframe, destination_table, project_id, chunksize=10000,
        Service account private key in JSON format. Can be file path
        or string contents. This is useful for remote server
        authentication (eg. jupyter iPython notebook on remote host)
+    table_schema : list of dicts
+    .. versionadded:: 0.2.0


the versionadded should go after the expl (with a blank line before ), otherwise doesn't render.

also needs to line up with the indent for the expl.

jreback · 2017-06-11T14:31:20Z

pandas_gbq/tests/test_gbq.py

@@ -1258,6 +1258,49 @@ def test_verify_schema_ignores_field_mode(self):
        assert self.sut.verify_schema(
            self.dataset_prefix + "1", TABLE_ID + test_id, test_schema_2)

+    # Issue #46; tests test scenarios with user-provided


put the comment inside the test

jreback · 2017-06-11T14:32:04Z

cc @parthea

ghost · 2017-06-11T14:45:34Z

@jreback this is green now.

jreback · 2017-07-01T10:07:03Z

can you rebase

…gleapis#46)

ghost · 2017-07-03T16:48:03Z

@jreback done

max-sixty · 2017-07-03T17:03:23Z

@mremes you need to rebase on master so that there are no conflicts

jreback

tiny comment. ping on green.

jreback · 2017-07-05T10:08:08Z

docs/source/changelog.rst

@@ -6,6 +6,7 @@ Changelog

 - Resolve issue where the optional ``--noauth_local_webserver`` command line argument would not be propagated during the authentication process. (:issue:`35`)
 - Drop support for Python 3.4 (:issue:`40`)
+- Add support for users to provide a table schema in ``to_gbq`` call instead of letting module to infer the schema from ``DataFrame.dtypes`` (:issue:`46`)


Add support for a passed schema in :func:to_gbq instead inferring the schema from the passed DataFrame with DataFrame.dtypes

…gleapis#46)

ghost · 2017-07-05T11:44:10Z

@jreback green. let's merge this after #47

jreback · 2017-07-06T22:25:52Z

pls rebase

jreback · 2017-07-06T22:26:16Z

sorry...just saw your comment about after #47

max-sixty · 2018-01-24T02:33:52Z

Ghost - did you delete your account? This needs a small rebase and then I am happy to quarterback it to get it approved and merged

…gleapis#46)

mremes · 2018-01-24T09:55:28Z

Ghost here 👻 This is rebased now.

…gleapis#46)

max-sixty · 2018-01-24T16:40:04Z

@tswast @jreback please could you take a look?

tswast

Could you setup Travis CI on your fork according to https://pandas-gbq.readthedocs.io/en/latest/contributing.html#running-google-bigquery-integration-tests and link to the build logs?

tswast · 2018-01-24T22:54:28Z

pandas_gbq/tests/test_gbq.py

+                       {'name': 'C', 'type': 'FLOAT'},
+                       {'name': 'D', 'type': 'FLOAT'}]
+        destination_table = self.destination_table + test_id
+        with tm.assertRaises(gbq.StreamingInsertError):


StreamingInsertError was removed in 0.3.0. to_gbq now creates a load job.

Replaced with the generic error. Errors need a better hierarchy in this module though.

tswast · 2018-01-24T22:54:56Z

pandas_gbq/tests/test_gbq.py

+        # schemas
+        df = tm.makeMixedDataFrame()
+        test_id = "15"
+        test_schema = [{'name': 'A', 'type': 'FLOAT'},


There's a chance this might fail with version 0.29.0 of google-cloud-bigquery due to googleapis/google-cloud-python#4456

Generated schemas do not include the mode property in the fields either. So this should be fine.

mremes · 2018-01-25T20:28:48Z

@tswast Comments addressed / responded. I cleaned BigQuery for any pandas_gbq_ datasets so I hope this will now pass ✌️ :
https://travis-ci.org/mremes/pandas-gbq/builds/333441821

tswast

Thanks @mremes ! Most of the test failures look like flakes (BigQuery is having some troubles right now) but there's one real one. I've left a comment.

tswast · 2018-01-25T21:45:51Z

pandas_gbq/tests/test_gbq.py

+                       {'name': 'C', 'type': 'FLOAT'},
+                       {'name': 'D', 'type': 'FLOAT'}]
+        destination_table = self.destination_table + test_id
+        with tm.assertRaises(gbq.GenericGBQException):


tm.assertRaises() doesn't exist: https://travis-ci.org/mremes/pandas-gbq/jobs/333441824#L2850

Since we are using PyTest as our test runner, I think you can use pytest.raises(...) https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions

tswast · 2018-01-28T04:43:24Z

Thanks for fixing it to use pytest.raises(). One last thing, looks like flake8 is failing with pandas_gbq/tests/test_gbq.py:4:1: F401 'warnings' imported but unused

mremes · 2018-01-28T14:35:49Z

Done.

max-sixty · 2018-01-28T22:34:59Z

Thanks a lot @mremes !

serafdev · 2018-02-06T16:00:57Z

Is this integrated in the latest pandas-gbq version?

tswast · 2018-02-06T17:37:48Z

@faresbessrour It hasn't been released yet.

max-sixty · 2018-02-06T23:38:24Z

But you can use with pip install git+git://github.com/pydata/pandas-gbq --upgrade

tswast · 2018-02-13T21:25:22Z

@faresbessrour Just released!

ghost force-pushed the to-gbq-with-schema branch from c7341ee to 1c89ef2 Compare June 3, 2017 11:57

jreback approved these changes Jun 4, 2017

View reviewed changes

jreback added the type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. label Jun 4, 2017

jreback added this to the 0.2.0 milestone Jun 4, 2017

ghost force-pushed the to-gbq-with-schema branch from 1c89ef2 to fc948bb Compare June 11, 2017 14:22

jreback approved these changes Jun 11, 2017

View reviewed changes

ghost force-pushed the to-gbq-with-schema branch 2 times, most recently from c0e3cd8 to 4e9a6cd Compare June 11, 2017 14:43

parthea changed the title ~~Add table_schema parameter for user-defined BigQuery schema~~ ENH: Add table_schema parameter for user-defined BigQuery schema Jun 13, 2017

ghost force-pushed the to-gbq-with-schema branch from 4e9a6cd to 71d4de1 Compare July 3, 2017 16:42

ghost pushed a commit to mremes/pandas-gbq that referenced this pull request Jul 3, 2017

ENH: Add table_schema parameter for user-defined BigQuery schema (goo…

71d4de1

…gleapis#46)

jreback reviewed Jul 5, 2017

View reviewed changes

ghost force-pushed the to-gbq-with-schema branch from 71d4de1 to 8ec63a4 Compare July 5, 2017 11:41

ghost pushed a commit to mremes/pandas-gbq that referenced this pull request Jul 5, 2017

ENH: Add table_schema parameter for user-defined BigQuery schema (goo…

8ec63a4

…gleapis#46)

ghost mentioned this pull request Jul 5, 2017

BUG: Add support to replace partitions in date-partitioned tables #47

Closed

max-sixty mentioned this pull request Jul 10, 2017

Add support for DATETIME BigQuery type #69

Closed

jreback modified the milestones: 0.2.0, 0.3.0 Jul 22, 2017

jreback added this to the 0.4.0 milestone Nov 25, 2017

mremes mentioned this pull request Jan 24, 2018

User-provided schema as to_gbq parameter #44

Closed

mremes force-pushed the to-gbq-with-schema branch from 7345159 to 6792076 Compare January 24, 2018 09:34

mremes added a commit to mremes/pandas-gbq that referenced this pull request Jan 24, 2018

ENH: Add table_schema parameter for user-defined BigQuery schema (goo…

6792076

…gleapis#46)

ENH: Add table_schema parameter for user-defined BigQuery schema (goo…

8791c49

…gleapis#46)

mremes force-pushed the to-gbq-with-schema branch from 6792076 to 8791c49 Compare January 24, 2018 09:56

tswast requested changes Jan 24, 2018

View reviewed changes

Matti Remes added 3 commits January 25, 2018 21:09

remove unsupported gbq exception and replace with a generic one

f6f7dc9

fix versionadded for to_gbq table_schema parameter

2783f7c

fix test id numbering

6740634

mremes force-pushed the to-gbq-with-schema branch from 088038e to 6740634 Compare January 25, 2018 20:18

tswast reviewed Jan 25, 2018

View reviewed changes

fix tests by using pytest raise asserts

4cfc00b

mremes force-pushed the to-gbq-with-schema branch from 4fc475d to 4cfc00b Compare January 28, 2018 14:35

tswast approved these changes Jan 28, 2018

View reviewed changes

tswast merged commit 296f15a into googleapis:master Jan 28, 2018

This was referenced Feb 13, 2018

Pin pandas-gbq to latest version 0.3.1 tnir/pandas#48

Closed

Pin pandas_gbq to latest version 0.3.1 tnir/pandas#49

Closed

mremes deleted the to-gbq-with-schema branch February 19, 2018 09:09

ENH: Add table_schema parameter for user-defined BigQuery schema #46

ENH: Add table_schema parameter for user-defined BigQuery schema #46

Conversation

ghost commented Jun 3, 2017

codecov-io commented Jun 3, 2017 • edited Loading

Codecov Report

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 4, 2017

jreback commented Jun 4, 2017

ghost commented Jun 11, 2017

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 11, 2017

ghost commented Jun 11, 2017

jreback commented Jul 1, 2017

ghost commented Jul 3, 2017

max-sixty commented Jul 3, 2017

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Jul 5, 2017

jreback commented Jul 6, 2017

jreback commented Jul 6, 2017

max-sixty commented Jan 24, 2018

mremes commented Jan 24, 2018

max-sixty commented Jan 24, 2018

tswast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mremes commented Jan 25, 2018

tswast left a comment

Choose a reason for hiding this comment

tswast Jan 25, 2018 • edited Loading

Choose a reason for hiding this comment

tswast commented Jan 28, 2018

mremes commented Jan 28, 2018

max-sixty commented Jan 28, 2018

serafdev commented Feb 6, 2018

tswast commented Feb 6, 2018

max-sixty commented Feb 6, 2018

tswast commented Feb 13, 2018

codecov-io commented Jun 3, 2017 •

edited

Loading

tswast Jan 25, 2018 •

edited

Loading