google.auth.default raises 404 when run from Travis with Python 3.7.

[tswast]

Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response

See: https://travis-ci.org/max-sixty/pandas-gbq/jobs/422752133 for googleapis/python-bigquery-pandas#206

I think this is related to the fact that Travis CI runs on GCE but it locks down access to APIs. Somehow google.auth.default() is not catching the 404.

/cc @max-sixty

[tseaver]

google.auth._default._get_gce_credentials and google.auth.compute_engine._metadata.ping each catch exceptions.TransportError from the request object, which is by default an instance of google.auth.transport._http_client.Request. Is pandas-gbq overriding that default?

[tswast]

No, pandas-gbq isn't overriding anything in google-auth. I'm not sure why this error seems to only show up in Python 3.7.

[tswast]

Looks like this error started popping up on other versions of Python, too. https://travis-ci.org/pydata/pandas-gbq/jobs/437226967

I think the issue is that google.auth.default() can throw a google.auth.exceptions.RefreshError on Travis and that pandas-gbq is only looking for DefaultCredentialsError. Shouldn't python-google-auth catch RefreshError and re-throw as DefaultCredentialsError in this case? (Or maybe skip checking metadata server in this case and move on to the next step?)

[theacodes]

What's the case where it throws a RefreshError?

[tswast]

Now that I look more closely at the stack trace, it's not google.auth.default() that's raising the RefreshError, it's when we call QueryJob._begin() in the google-cloud-bigquery client library.

I'm guessing that means somehow google.auth.default() is returning GCE credentials on Travis, but then it fails when we try to use / refresh those credentials. Travis does run on GCE, so I guess I'm not too surprised by this, but it does seem to be a somewhat new error. Maybe Travis changed their configuration and now calls to the metadata server aren't blocked anymore?

[Fryuni]

Where did you define the default credentials?
If there are no default credentials on the environment, the library checks gcloud application default, if it is not there it checks app engine metadata endpoint and then the GCE metadata endpoint.

There is the line export SERVICE_ACCOUNT_KEY=[secure] which I think is the content of the service account key you are trying to use... right?!
If it is, add the following before you start your code (normally I would put it at the end of before_install):

echo "$SERVICE_ACCOUNT_KEY" > /tmp/credentials.json
export GOOGLE_APPLICATION_CREDENTIALS="/tmp/credentias.json"

Obviously, you can choose any path you want to store it.

The default credentials that takes precedence over all other is the environment variable GOOGLE_APPLICATION_CREDENTIALS, and it needs to point to a service account key file. If you just put the content of the key on the variable it won't work.

[tswast]

I should clarify, this error happens on PRs from forks (not branches), meaning the GOOGLE_APPLICATION_CREDENTIALS environment variable is not available.

I've changed several things with pandas-gbq system tests on CI since filing this bug, including migrating to CircleCI. We can close this issue.