Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added tox config with flake8 and nosetests. #6

Merged
merged 9 commits into from
May 14, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[run]
include =
datastore/*
localtesting/*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging #2 So please add bigquery/

bigquery/*
[report]
exclude_lines =
pragma: NO COVER
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@

*.pyc
.coverage
.tox
coverage.xml
nosetests.xml
python-docs-samples.json
14 changes: 8 additions & 6 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
sudo: false
#add language, etc. here
language: python

cache:
directories:
- $HOME/gcloud/
env:
- PATH=$PATH:$HOME/gcloud/google-cloud-sdk/bin GOOGLE_APPLICATION_CREDENTIALS=$TRAVIS_BUILD_DIR/client_secrets.json #Other environment variables on same line
- PATH=$PATH:$HOME/gcloud/google-cloud-sdk/bin GOOGLE_APPLICATION_CREDENTIALS=$TRAVIS_BUILD_DIR/python-docs-samples.json PYTHONPATH=${HOME}/gcloud/google-cloud-sdk/platform/google_appengine #Other environment variables on same line

before_install:
#ENCRYPT YOUR PRIVATE KEY (If you need authentication)
Expand All @@ -27,13 +28,14 @@ before_install:
printf '\ny\n\ny\ny\n' | ./google-cloud-sdk/install.sh &&
cd $TRAVIS_BUILD_DIR;
fi
- printf 'y\n' | gcloud components update
- if [ -a client_secrets.json ]; then
gcloud auth activate-service-account --key-file client_secrets.json;
- gcloud -q components update gae-python
- openssl aes-256-cbc -K $encrypted_4fda24e244ca_key -iv $encrypted_4fda24e244ca_iv -in python-docs-samples.json.enc -out python-docs-samples.json -d
- if [ -a python-docs-samples.json ]; then
gcloud auth activate-service-account --key-file python-docs-samples.json;
fi

install:
#Add app specific setup here
- pip install tox

script:
#Test and/or deploy your app with gcloud commands here!
- tox
20 changes: 20 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,26 @@ For more detailed introduction to a product, check the README in the correspondi

* See [CONTRIBUTING.md](CONTRIBUTING.md)

### How to run the test

To run the tests, please install App Engine Python SDK and tox and run
tox with the environment variable PYTHONPATH to the App Engine Python
SDK.

You can install App Engine Python SDK with
[Google Cloud SDK](https://cloud.google.com/sdk/) with the following
command:

$ gcloud components update gae-python

Here is instructions to run the tests with virtualenv, $GCLOUD is your
Google Cloud SDK installation path.

$ virtualenv -p python2.7 --no-site-packages /some/where
$ source /some/where/bin/activate
$ pip install tox
$ env PYTHONPATH=${GCLOUD}/platform/google_appengine tox

## Licensing

* See [LICENSE](LICENSE)
2 changes: 0 additions & 2 deletions datastore/ndb/app.yaml → app.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# dummy app.yaml for nosegae

application: ndb-snippets
version: 1
api_version: 1
runtime: python27
threadsafe: true
Expand Down
Empty file added bigquery/__init__.py
Empty file.
13 changes: 0 additions & 13 deletions bigquery/samples/__init__.py
Original file line number Diff line number Diff line change
@@ -1,13 +0,0 @@
# Copyright 2015, Google, Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
64 changes: 31 additions & 33 deletions bigquery/samples/async_query.py
Original file line number Diff line number Diff line change
@@ -1,20 +1,24 @@
# Copyright 2015, Google, Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Copyright 2015, Google, Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from __future__ import print_function # For python 2/3 interoperability
from samples.utils import get_service, paging, poll_job
import uuid

import json
import uuid

from bigquery.samples.utils import get_service
from bigquery.samples.utils import paging
from bigquery.samples.utils import poll_job


# [START async_query]
Expand All @@ -23,19 +27,19 @@ def async_query(service, project_id, query, batch=False, num_retries=5):
# don't accidentally duplicate query
job_data = {
'jobReference': {
'projectId': project_id,
'job_id': str(uuid.uuid4())
},
'projectId': project_id,
'job_id': str(uuid.uuid4())
},
'configuration': {
'query': {
'query': query,
'priority': 'BATCH' if batch else 'INTERACTIVE',
},
}
'query': {
'query': query,
'priority': 'BATCH' if batch else 'INTERACTIVE'
}
}
}
return service.jobs().insert(
projectId=project_id,
body=job_data).execute(num_retries=num_retries)
projectId=project_id,
body=job_data).execute(num_retries=num_retries)
# [END async_query]


Expand All @@ -55,7 +59,6 @@ def run(project_id, query_string, batch, num_retries, interval):
interval,
num_retries)


for page in paging(service,
service.jobs().getQueryResults,
num_retries=num_retries,
Expand All @@ -69,18 +72,13 @@ def run(project_id, query_string, batch, num_retries, interval):
def main():
project_id = raw_input("Enter the project ID: ")
query_string = raw_input("Enter the Bigquery SQL Query: ")
batch = raw_input("Run query as batch (y/n)?: ") in ('True',
'true',
'y',
'Y',
'yes',
'Yes')

batch = raw_input("Run query as batch (y/n)?: ") in (
'True', 'true', 'y', 'Y', 'yes', 'Yes')

num_retries = raw_input(
"Enter number of times to retry in case of 500 error: ")
"Enter number of times to retry in case of 500 error: ")
interval = raw_input(
"Enter how often to poll the query for completion (seconds): ")
"Enter how often to poll the query for completion (seconds): ")

for result in run(project_id, query_string, batch, num_retries, interval):
print(result)
Expand Down
52 changes: 29 additions & 23 deletions bigquery/samples/discovery_doc.py
Original file line number Diff line number Diff line change
@@ -1,35 +1,38 @@
# Copyright 2015, Google, Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Copyright 2015, Google, Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import os
"""
A module that takes care of caching and updating discovery docs for
google-api-python-clients (until such a feature is integrated).
"""

import json
import httplib2
import os
import time

# [START build_and_update]
import httplib2

RESOURCE_PATH='..' #look for discovery docs in the parent folder
MAX_AGE = 86400 #update discovery docs older than a day
# [START build_and_update]

# A module that takes care of caching and updating discovery docs
# for google-api-python-clients (until such a feature is integrated)
RESOURCE_PATH = '..' # look for discovery docs in the parent folder
MAX_AGE = 86400 # update discovery docs older than a day
BIGQUERY_SCOPES = ['https://www.googleapis.com/auth/bigquery']


def build_and_update(api, version):
from oauth2client.client import GoogleCredentials
from googleapiclient.discovery import build_from_document


path = os.path.join(RESOURCE_PATH, '{}.{}'.format(api, version))
try:
age = time.time() - os.path.getmtime(path)
Expand All @@ -38,11 +41,14 @@ def build_and_update(api, version):
except os.error:
_update_discovery_doc(api, version, path)

credentials = GoogleCredentials.get_application_default()
if credentials.create_scoped_required():
credentials = credentials.create_scoped(BIGQUERY_SCOPES)
with open(path, 'r') as discovery_doc:
return build_from_document(discovery_doc.read(),
http=httplib2.Http(),
credentials=GoogleCredentials
.get_application_default())
http=httplib2.Http(),
credentials=credentials)


def _update_discovery_doc(api, version, path):
from apiclient.discovery import DISCOVERY_URI
Expand All @@ -61,5 +67,5 @@ def _update_discovery_doc(api, version, path):
json.dump(discovery_json, discovery_doc)
except ValueError:
raise InvalidJsonError(
'Bad JSON: %s from %s.' % (content, requested_url))
'Bad JSON: %s from %s.' % (content, requested_url))
# [END build_and_update]
49 changes: 32 additions & 17 deletions bigquery/samples/export_data_to_cloud_storage.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,21 @@
from samples.utils import get_service, poll_job
# Copyright 2015, Google, Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import uuid

from bigquery.samples.utils import get_service
from bigquery.samples.utils import poll_job


# [START export_table]
def export_table(service, cloud_storage_path,
Expand All @@ -9,21 +24,21 @@ def export_table(service, cloud_storage_path,
# Generate a unique job_id so retries
# don't accidentally duplicate export
job_data = {
'jobReference': {
'jobReference': {
'projectId': projectId,
'jobId': str(uuid.uuid4())
},
'configuration': {
'extract': {
'sourceTable': {
'projectId': projectId,
'jobId': str(uuid.uuid4())
},
'configuration': {
'extract': {
'sourceTable': {
'projectId': projectId,
'datasetId': datasetId,
'tableId': tableId,
},
'destinationUris': [cloud_storage_path],
}
}
'datasetId': datasetId,
'tableId': tableId,
},
'destinationUris': [cloud_storage_path],
}
}
}
return service.jobs().insert(
projectId=projectId,
body=job_data).execute(num_retries=num_retries)
Expand Down Expand Up @@ -52,11 +67,11 @@ def main():
datasetId = raw_input("Enter a dataset ID: ")
tableId = raw_input("Enter a table name to copy: ")
cloud_storage_path = raw_input(
"Enter a Google Cloud Storage URI: ")
"Enter a Google Cloud Storage URI: ")
interval = raw_input(
"Enter how often to poll the job (in seconds): ")
"Enter how often to poll the job (in seconds): ")
num_retries = raw_input(
"Enter the number of retries in case of 500 error: ")
"Enter the number of retries in case of 500 error: ")

run(cloud_storage_path,
projectId, datasetId, tableId,
Expand Down
29 changes: 16 additions & 13 deletions bigquery/samples/load_data_by_post.py
Original file line number Diff line number Diff line change
@@ -1,19 +1,22 @@
# Copyright 2015, Google, Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Copyright 2015, Google, Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import json

from bigquery.samples.utils import get_service, poll_job

import httplib2
from samples.utils import get_service, poll_job

from oauth2client.client import GoogleCredentials


Expand Down Expand Up @@ -64,7 +67,7 @@ def main():
datasetId = raw_input('Enter a dataset ID: ')
tableId = raw_input('Enter a table name to load the data to: ')
schema_path = raw_input(
'Enter the path to the schema file for the table: ')
'Enter the path to the schema file for the table: ')

with open(schema_path, 'r') as schema_file:
schema = schema_file.read()
Expand Down
Loading