CI Rehash

[lukesneeringer]

This is a continuation of #3135. It is moved to google-cloud-python:ci (from my personal branch) because system tests do not run from personal branches.

---

This is an attempt to fix the CI situation on this repository. In particular, this will hit the following goals:

• Move to parallel builds on a per-API basis. Superceded by not running unnecessary APIs.
• Move system tests into the individual API packages.
• Eliminate Travis, move CircleCI to 2.0

In doing this, I made several changes:

• Move from tox to nox (Python-based configuration is really useful for the multi-project situation)
• Move from pylint to flake8 (pylint does not work on Python 3.6, has multiple difficult-to-silence false positives)

This is not done. In particular, several things need to be put back into place:

• Automatic docs build.
  • Note to self: Move docs into individual packages (not part of this PR).
• Automatic PyPI releasing on appropriate tags.
• Only testing APIs that have changed files.

[tseaver]

@lukesneeringer I tagged this one "do not merge" because you are still iterating rapidly.

[lukesneeringer]

@lukesneeringer I tagged this one "do not merge" because you are still iterating rapidly.

Thank you!

[lukesneeringer]

This should be almost ready. Because this commit moves literally everything, it triggers all the system tests now, so they need to pass before it is.

Once it is ready, I will rebase into a smaller number of commits so it is reviewable on a per-commit basis. Once approved, we will still squash. The rebase will be for review's sake.

[lukesneeringer]

Hey @tseaver -- Do you happen to know off hand what this failure is about? Is it that the test (or backend) is flaky, or is there something else I should be investigating?

No need to do a ton of research; this is mostly a lazy if-you-know-off-the-top-of-your-head thing.

[tseaver]

@lukesneeringer I'm not seeing the error detail show up on Circle CI today: just spinning arrows. I do have a pending Spaner flaky-unit-test fix (#3147), which I'm about to merge.

[lukesneeringer]

@tseaver This one is actually a system test; here is the failure:

REDACTED FOR EXCESSIVE LENGTH

[lukesneeringer]

I now have CircleCI's build passing...sometimes. We have some system tests that are still flaky and we need to weed out. That said, this is probably close to ready, and still an improvement over where we were.

I will rebase tomorrow so that the PR is parse-able and reviewable.

[lukesneeringer]

Also, the total serial test run-time is down to 40 minutes. (I think I could implement some CircleCI caching of the virtualenvs to cut that in half.)

[lukesneeringer]

Caught a new system test bug on merge; filed #3193

[lukesneeringer]

The previous build passed on AppVeyor; this one should pass now that #3193 is fixed. Assuming both pass without incident, will squash and merge.

Travis is failing because this build has no .travis.yml at all; after the merge, I will turn Travis completely off.

[lukesneeringer]

I finally have a green build. Squashing and merging.
I will then turn off Travis entirely.

[dhermes]

@lukesneeringer Why the merge without review?

[lukesneeringer]

It has already gone through a non-trivial amount of review, by everyone here. The only "unreviewed" work was the AppVeyor thrashing.

[csilvers]

It looks like CONTRIBUTING.rst still talks about running tox. Possibly other documentation does too. Can you fix it to show how to run tests in the new world? I can't figure out how to even install nox, much less run it.

[csilvers]

Ah, I see, this is still open in #3205