Run E2E tests during CI #363

carolynvs · 2018-07-30T17:07:36Z

Add ./scripts/test_e2e.sh which builds a go program that relies on go modules using GOPROXY.
Add make test-e2e target and document it. It's important to run make setup-dev-env once, before running the tests.
Use the same setup for travis and local development for the storage services

Closes #353

carolynvs · 2018-07-30T17:14:45Z

Oops, I think I dorked something up on this PR. Please ignore and I'll fix later.

arschles · 2018-08-01T06:14:50Z

Thanks @carolynvs ! Let me know if I can help on this in any way. These tests would be awesome to have in here 👍 🎉

Don't error out if it's been cloned already

codecov-io · 2018-08-02T03:00:48Z

Codecov Report

Merging #363 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #363   +/-   ##
=======================================
  Coverage   39.94%   39.94%           
=======================================
  Files          99       99           
  Lines        2671     2671           
=======================================
  Hits         1067     1067           
  Misses       1505     1505           
  Partials       99       99

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54106b9...54ea3ef. Read the comment docs.

carolynvs · 2018-08-02T03:09:12Z

@arschles Ok! I've gotten our travis and local dev setup more consistent and a very simple end-to-end test passing.

Someone may prefer to rewrite it in Go, vs Bash, but this at least gets us some coverage for now! 😀

carolynvs · 2018-08-02T03:12:46Z

At some point we may be able to use gimme go1.11beta2 in our travis config instead of building go from source, I didn't want to make that jump at this time though. I'm pretty sure we may need to make more changes once start building Athens using go 1.11

michalpristas · 2018-08-02T06:31:06Z

.travis.yml

-  - mongodb
-  - postgresql
-  - redis-server
+  - docker


this escalated quickly 👍

michalpristas

approved 💯 times

komuw

awesome.

komuw · 2018-08-02T07:19:15Z

scripts/test_e2e.sh

+
+# Use a version of Go that supports Go Modules
+export GO111MODULES=on
+GOMOD_CACHE=$(go env GOPATH)/src/mod


@carolynvs
In Go1.11beta the location of the module cache is $GOPATH/src/mod
However on tip the location has changed to $GOPATH/pkg/mod[1]

Maybe you could just declare;

clearGoModCache () { # The sudo is a necessary workaround until go is fixed sudo rm -fr $GOPATH/* }

ref:

https://go-review.googlesource.com/c/go/+/126755

I think this should be ignored until the change is in an official Go release as we do not work with go from tip.

Agreed, I'm explicitly using a tagged release, 1.11beta2 and would prefer to not try to follow tip. Once they tag a new release with that change, then yes we should update this to keep pace with the new location.

I definitely do not want to do a rm -fr $GOPATH/* though, and just stick to nuking the mod cache directory, as this is something that should be safe for a dev to run locally as well as run on Travis.

komuw · 2018-08-02T07:23:09Z

scripts/test_e2e.sh

+# test_e2e.sh
+# Execute end-to-end (e2e) tests to verify that everything is working right
+# from the end user perpsective
+set -xeuo pipefail


@carolynvs
I do not know what version of bash travisCI uses,
but Bash version 4.3 and older doesn't handle set -u very well in some circumstances(arrays)
But I don't think you are using arrays so, this may be fine

https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md#how-to-begin-a-bash-script

It seems to be working fine, and it's super helpful to have the variable checking. Let me know if this is a must fix, otherwise I think it would be good to stick with -u.

komuw · 2018-08-02T07:24:37Z

scripts/test_e2e.sh

+# Start the proxy in the background and wait for it to be ready
+export GO_BINARY_PATH=${GO_SOURCE}/bin/go
+cd $REPO_DIR/cmd/proxy
+pkill buffalo || true # cleanup old buffalos


cleanup old buffalos 😄

ghost · 2018-08-02T16:22:31Z

scripts/test_e2e.sh

+# Clone our test repo
+TEST_SOURCE=${TMPDIR}go-depmgmt-testrepo
+rm -fr ${TEST_SOURCE} 2> /dev/null || true
+git clone https://github.com/carolynvs/go-depmgmt-testrepo.git ${TEST_SOURCE}


We could consider adding this with #340

carolynvs · 2018-08-03T01:47:54Z

I've updated the PR with the recent changes from master.

michalpristas · 2018-08-03T06:10:15Z

@carolynvs i think this is good to go, do you plan to do any more changes?

carolynvs · 2018-08-03T14:16:33Z

This is ready to merge! I can’t, not a committee on this repo, so anyone who is feel free to

michalpristas · 2018-08-03T14:33:34Z

I'm just bringing this even with master and will merge

ghost · 2018-08-03T14:35:02Z

Thanks, was just about to as well 😁 sorry @carolynvs I got distracted at work yesterday.

carolynvs force-pushed the e2e branch from 0bbca34 to db0fa0b Compare July 30, 2018 18:47

carolynvs-msft added 2 commits August 1, 2018 21:33

Use go1.11beta2 instead of vgo

32fe18b

Make the go 1.11 source directory configurable

6ba5b94

Don't error out if it's been cloned already

carolynvs force-pushed the e2e branch 2 times, most recently from a8b634b to 058b009 Compare August 2, 2018 02:53

carolynvs-msft and others added 4 commits August 1, 2018 23:06

Add end-to-end (e2e) tests

e9a137b

Make sure the repo bin is on the path

fba78e9

Call the dev target from setup-dev-env

3db118b

Use the same dev setup on travis and locally

a1b6480

carolynvs force-pushed the e2e branch from 058b009 to a1b6480 Compare August 2, 2018 03:06

carolynvs changed the title ~~[WIP] Run E2E tests during CI~~ Run E2E tests during CI Aug 2, 2018

michalpristas reviewed Aug 2, 2018

View reviewed changes

.travis.yml

- mongodb

- postgresql

- redis-server

- docker

Copy link

Member

michalpristas Aug 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this escalated quickly 👍

carolynvs reacted with laugh emoji

michalpristas approved these changes Aug 2, 2018

View reviewed changes

michalpristas added the testing improvements or changes to our test code label Aug 2, 2018

komuw reviewed Aug 2, 2018

View reviewed changes

ghost reviewed Aug 2, 2018

View reviewed changes

ghost approved these changes Aug 2, 2018

View reviewed changes

Merge branch 'master' into e2e

8c926e8

komuw approved these changes Aug 3, 2018

View reviewed changes

Merge branch 'master' into e2e

54ea3ef

michalpristas merged commit f1efade into gomods:master Aug 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run E2E tests during CI #363

Run E2E tests during CI #363

carolynvs commented Jul 30, 2018 •

edited

Loading

carolynvs commented Jul 30, 2018

arschles commented Aug 1, 2018

codecov-io commented Aug 2, 2018 •

edited

Loading

carolynvs commented Aug 2, 2018

carolynvs commented Aug 2, 2018

michalpristas Aug 2, 2018

michalpristas left a comment

komuw left a comment

komuw Aug 2, 2018

ghost Aug 2, 2018

carolynvs Aug 3, 2018

komuw Aug 2, 2018

carolynvs Aug 3, 2018

komuw Aug 2, 2018

ghost Aug 2, 2018

carolynvs commented Aug 3, 2018

michalpristas commented Aug 3, 2018

carolynvs commented Aug 3, 2018 •

edited

Loading

michalpristas commented Aug 3, 2018

ghost commented Aug 3, 2018

Run E2E tests during CI #363

Run E2E tests during CI #363

Conversation

carolynvs commented Jul 30, 2018 • edited Loading

carolynvs commented Jul 30, 2018

arschles commented Aug 1, 2018

codecov-io commented Aug 2, 2018 • edited Loading

Codecov Report

carolynvs commented Aug 2, 2018

carolynvs commented Aug 2, 2018

Choose a reason for hiding this comment

michalpristas left a comment

Choose a reason for hiding this comment

komuw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolynvs commented Aug 3, 2018

michalpristas commented Aug 3, 2018

carolynvs commented Aug 3, 2018 • edited Loading

michalpristas commented Aug 3, 2018

ghost commented Aug 3, 2018

carolynvs commented Jul 30, 2018 •

edited

Loading

codecov-io commented Aug 2, 2018 •

edited

Loading

carolynvs commented Aug 3, 2018 •

edited

Loading