"Fix"/schroedinger tests

[whilei]

• consider what to do with schroedingers_cat.go -- move to janus? relocate somewhere else? leave as-is? moved to https://github.com/etcdevteam/go-schroedinger, and use go get to install command on CIs
• possibly rename "schroedinger" things to be something a little more "professional" ...

Schrödinger coined the term Verschränkung (entanglement) in the course of developing the thought experiment. https://en.wikipedia.org/wiki/Schr%C3%B6dinger's_cat

• include more nondeterministic tests in schroedinger-tests.txt, I know there are a few I missed

• decide for or against alternate design, ala:
  An alternative design would be to refactor to use

// +build deterministic

tags for all other *_test.go files which ARE expected to pass consistently, and then use a smaller bash script to run whole files at a time in a loop. AFAIK this tagging verbosity is a limitation of go build tags and inclusion/exclusion rules. It will also mean that without http://github.com/ETCDEVTeam/go-schroedinger a lot of tests will need to be run redundantly and will thus take up a lot more time (since schroedinger picks only tests that fail from packages and reruns those individually)or will take a lot of hardcoding probably-nondeterministic tests and still some redundancy, eg.

while [[ ! go test ./eth/downloader/... ]]; then
  # This re-runs a lot of tests that already passed and take a long time to run anyways.
  echo Trying again...
done

rel #245
importantly related: https://github.com/etcdevteam/go-schroedinger

[tzdybal]

To make testing more granular and explicit, we can also rename (prefix or suffix) all non-deterministic tests and use -run parameter with appropriate regexp. What do you think about it @whilei, @r8d8?

[tzdybal]

From go 1.10 passing tests are cached by default. So they won't be executed multiple times. The most KISS solution is to rerun all tests (cache is salvation) in a loop.

[whilei]

Thinking toward explicitness and granularity... we can build that list with a couple of seds from your (@tzdybal) previous analysis of (non)determinism rates and occurences using -count, yea?

The prefix/suffix + -run <regex> idea would allow us to include or exclude tests as desired... but still doesn't give us much granular control over re-running only individual failing tests, or? I'm imagining it would look something like:

# pseudocode
go test -tags="sputnikvm deterministic" -v ./...
allowedTrials=4
trials=0
while [[ ! go test -tags='sputnikvm !deterministic' -v ./... -run "(.*Schroedinger$)" ]]; do
    trials++
    if [[ trials > allowedTrails ]];
       break;
    fi
done

[tzdybal]

It's not ideal solution, but will save us hours ;)

[whilei]

@tzdybal -- I'm not completely sure of your mod 40a8e56 because I'm not familiar with the circle ci workflow setups -- but basically LGTM if LGTY

[whilei]

@tzdybal Tag, you're it; your merge if LTGY. LGTM

Get used to green. 💚

[tzdybal]

I have to add my glog rotation tests to the shame-list 👎

[whilei]

@tzdybal Do you want to add them before merge, or separately after?

[tzdybal]

I created an issue for this and I want to fix them.

[whilei]

Ah, gotcha -- #564

[tzdybal]

See: #580.

So let's merge this PR.