Squashed 'tools/' changes from a3b18bf..334708c

334708c Merge pull request weaveworks#75 from kinvolk/alban/external-build-1
da2505d gce.sh: template: print creation date
e676854 integration tests: fix user account
8530836 host nameing: add repo name
b556c0a gce.sh: fix deletion of gce instances
2ecd1c2 integration: fix GCE --zones/--zone parameter
51785b5 Use rm -f and set current dir using BASH_SOURCE.
f5c6d68 Merge pull request weaveworks#71 from kinvolk/schu/fix-linter-warnings
0269628 Document requirement for `lint_sh`
9a3f09e Fix linter warnings
efcf9d2 Merge pull request weaveworks#53 from weaveworks/2647-testing-mvp
d31ea57 Weave Kube playbook now works with multiple nodes.
27868dd Add GCP firewall rule for FastDP crypto.
edc8bb3 Differentiated name of dev and test playbooks, to avoid confusion.
efa3df7 Moved utility Ansible Yaml to library directory.
fcd2769 Add shorthands to run Ansible playbooks against Terraform-provisioned virtual machines.
f7946fb Add shorthands to SSH into Terraform-provisioned virtual machines.
aad5c6f Mention Terraform and Ansible in README.md.
dddabf0 Add Terraform output required for templates' creation.
dcc7d02 Add Ansible configuration playbooks for development environments.
f86481c Add Ansible configuration playbooks for Docker, K8S and Weave-Net.
efedd25 Git-ignore Ansible retry files.
765c4ca Add helper functions to setup Terraform programmatically.
801dd1d Add Terraform cloud provisioning scripts.
b8017e1 Install hclfmt on CircleCI.
4815e19 Git-ignore Terraform state files.
0aaebc7 Add script to generate cartesian product of dependencies of cross-version testing.
007d90a Add script to list OS images from GCP, AWS and DO.
ca65cc0 Add script to list relevant versions of Go, Docker and Kubernetes.
aa66f44 Scripts now source dependencies using absolute path (previously breaking make depending on current directory).
7865e86 Add -p option to parallelise lint.
36c1835 Merge pull request weaveworks#69 from weaveworks/mflag
9857568 Use mflag and mflagext package from weaveworks/common.
9799112 Quote bash variable.
10a36b3 Merge pull request weaveworks#67 from weaveworks/shfmt-ignore
a59884f Add support for .lintignore.
03cc598 Don't lint generated protobuf code.
2b55c2d Merge pull request weaveworks#66 from weaveworks/reduce-test-timeout
d4e163c Make timeout a flag
49a8609 Reduce test timeout
8fa15cb Merge pull request weaveworks#63 from weaveworks/test-defaults
b783528 Tweak test script so it can be run on a mca

git-subtree-dir: tools
git-subtree-split: 334708cadceeee395d37c8eb36a8b3191519def0

.gitignore

@@ -4,3 +4,6 @@ socks/image.tar
 runner/runner
 *.pyc
 *~
+terraform.tfstate
+terraform.tfstate.backup
+*.retry

README.md

@@ -2,12 +2,14 @@
 
 Included in this repo are tools shared by weave.git and scope.git.  They include
 
+- ```provisioning```: a set of Terraform scripts to provision virtual machines in GCP, AWS or Digital Ocean.
+- ```config_management```: a set of Ansible playbooks to configure virtual machines for development, testing, etc.
 - ```cover```: a tool which merges overlapping coverage reports generated by go
   test
 - ```files-with-type```: a tool to search directories for files of a given
   MIME type
-- ```lint```: a script to lint Go project; runs various tools like golint, go
-  vet, errcheck etc
+- ```lint```: a script to lint go, sh and hcl files; runs various tools like
+  golint, go vet, errcheck, shellcheck etc
 - ```rebuild-image```: a script to rebuild docker images when their input files
   change; useful when you using docker images to build your software, but you
   don't want to build the image every time.
@@ -24,6 +26,11 @@ Included in this repo are tools shared by weave.git and scope.git.  They include
 - ```scheduler```: an appengine application that can be used to distribute
   tests across different shards in CircleCI.
 
+## Requirements
+
+- ```lint``` requires shfmt to lint sh files; get shfmt with
+  ```go get -u gopkg.in/mvdan/sh.v1/cmd/shfmt```
+
 ## Using build-tools.git
 
 To allow you to tie your code to a specific version of build-tools.git, such

circle.yml

@@ -19,6 +19,7 @@ dependencies:
         github.com/fzipp/gocyclo \
         github.com/golang/lint/golint \
         github.com/kisielk/errcheck \
+        github.com/fatih/hclfmt \
         gopkg.in/mvdan/sh.v1/cmd/shfmt
 
 test:

config_management/README.md

@@ -0,0 +1,113 @@
+# Weaveworks configuration management
+
+## Introduction
+
+This project allows you to configure a machine with:
+
+* Docker and Weave Net: `setup_docker_weave-net.yml`
+* Docker, Kubernetes and Weave Kube (CNI plugin): `setup_docker_k8s_weave-kube.yml`
+
+You can then use these environments for development, testing and debugging.
+
+## Set up
+
+You will need [Python](https://www.python.org/downloads/) and [Ansible 2.+](http://docs.ansible.com/ansible/intro_installation.html) installed on your machine and added to your `PATH` in order to be able to configure environments automatically.
+
+* On any platform, if you have Python installed: `pip install ansible`
+* On macOS: `brew install ansible`
+* On Linux (via Aptitude): `sudo apt install ansible`
+* On Linux (via YUM): `sudo yum install ansible`
+* For other platforms or more details, see [here](http://docs.ansible.com/ansible/intro_installation.html)
+
+Frequent errors during installation are:
+
+* `fatal error: Python.h: No such file or directory`: install `python-dev`
+* `fatal error: ffi.h: No such file or directory`: install `libffi-dev`
+* `fatal error: openssl/opensslv.h: No such file or directory`: install `libssl-dev`
+
+Full steps for a blank Ubuntu/Debian Linux machine:
+
+    sudo apt-get install -qq -y python-pip python-dev libffi-dev libssl-dev
+    sudo pip install -U cffi
+    sudo pip install ansible
+
+## Tags
+
+These can be used to selectively run (`--tags "tag1,tag2"`) or skip (`--skip-tags "tag1,tag2"`) tasks.
+
+  * `output`: print potentially useful output from hosts (e.g. output of `kubectl get pods --all-namespaces`)
+
+## Usage
+
+### Local machine
+
+```
+ansible-playbook -u <username> -i "localhost", -c local setup_docker_k8s_weave-kube.yml
+```
+
+### Vagrant
+
+Provision your local VM using Vagrant:
+
+```
+cd $(mktemp -d -t XXX)
+vagrant init ubuntu/xenial64  # or, e.g. centos/7
+vagrant up
+```
+
+then set the following environment variables by extracting the output of `vagrant ssh-config`:
+
+```
+eval $(vagrant ssh-config | sed \
+-ne 's/\ *HostName /vagrant_ssh_host=/p' \
+-ne 's/\ *User /vagrant_ssh_user=/p' \
+-ne 's/\ *Port /vagrant_ssh_port=/p' \
+-ne 's/\ *IdentityFile /vagrant_ssh_id_file=/p')
+```
+
+and finally run:
+
+```
+ansible-playbook --private-key=$vagrant_ssh_id_file -u $vagrant_ssh_user \
+--ssh-extra-args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
+-i "$vagrant_ssh_host:$vagrant_ssh_port," setup_docker_k8s_weave-kube.yml
+```
+
+or, for specific versions of Kubernetes and Docker:
+
+```
+ansible-playbook --private-key=$vagrant_ssh_id_file -u $vagrant_ssh_user \
+--ssh-extra-args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
+-i "$vagrant_ssh_host:$vagrant_ssh_port," setup_docker_k8s_weave-kube.yml \
+--extra-vars "docker_version=1.12.3 kubernetes_version=1.4.4"
+```
+
+### Terraform
+
+Provision your machine using the Terraform scripts from `../provisioning`, then run:
+
+```
+ansible-playbook -u `terraform output username` -i "`terraform output public_ips`," \
+--ssh-extra-args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
+../../config_management/setup_docker_k8s_weave-kube.yml
+
+```
+
+or, for specific versions of Kubernetes and Docker:
+
+```
+ansible-playbook -u `terraform output username` -i "`terraform output public_ips`," \
+--ssh-extra-args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
+../../config_management/setup_docker_k8s_weave-kube.yml \
+--extra-vars "docker_version=1.12.3 kubernetes_version=1.4.4"
+```
+
+N.B.: `--ssh-extra-args` is used to provide:
+
+* `StrictHostKeyChecking=no`: as VMs come and go, the same IP can be used by a different machine, so checking the host's SSH key may fail. Note that this introduces a risk of a man-in-the-middle attack.
+* `UserKnownHostsFile=/dev/null`: if you previously connected a VM with the same IP but a different public key, and added it to `~/.ssh/known_hosts`, SSH may still fail to connect, hence we use `/dev/null` instead of `~/.ssh/known_hosts`.
+
+## Resources
+
+* [https://www.vagrantup.com/docs/provisioning/ansible.html](https://www.vagrantup.com/docs/provisioning/ansible.html)
+* [http://docs.ansible.com/ansible/guide_vagrant.html](http://docs.ansible.com/ansible/guide_vagrant.html)

config_management/group_vars/all

@@ -0,0 +1,10 @@
+---
+go_version: 1.7.4
+terraform_version: 0.8.5
+docker_version: 1.11.2
+kubernetes_version: 1.5.2
+kubernetes_cni_version: 0.3.0.1
+kubernetes_token: 123456.0123456789123456
+etcd_container_version: 2.2.5
+kube_discovery_container_version: 1.0
+pause_container_version: 3.0

config_management/library/setup_ansible_dependencies.yml

@@ -0,0 +1,33 @@
+---
+################################################################################
+# Install Ansible's dependencies: python and lsb_release, required respectively 
+# to run Ansible modules and gather Ansible facts.
+#
+# See also:
+# - http://docs.ansible.com/ansible/intro_installation.html#managed-node-requirements
+# - http://docs.ansible.com/ansible/setup_module.html
+################################################################################
+
+- name: check if python is installed (as required by ansible modules)
+  raw: test -e /usr/bin/python
+  register: is_python_installed
+  failed_when: is_python_installed.rc not in [0, 1]
+  changed_when: false  # never mutates state.
+
+- name: install python if missing (as required by ansible modules)
+  when: is_python_installed|failed  # skip otherwise
+  raw: (test -e /usr/bin/apt-get && apt-get update && apt-get install -y python-minimal) || (test -e /usr/bin/yum && yum update && yum install -y python)
+  changed_when: is_python_installed.rc == 1
+
+- name: check if lsb_release is installed (as required for ansible facts)
+  raw: test -e /usr/bin/lsb_release
+  register: is_lsb_release_installed
+  failed_when: is_lsb_release_installed.rc not in [0, 1]
+  changed_when: false  # never mutates state.
+
+- name: install lsb_release if missing (as required for ansible facts)
+  when: is_lsb_release_installed|failed  # skip otherwise
+  raw: (test -e /usr/bin/apt-get && apt-get install -y lsb_release) || (test -e /usr/bin/yum && yum install -y redhat-lsb-core)
+  changed_when: is_lsb_release_installed.rc == 1
+
+- setup:  # gather 'facts', i.e. compensates for 'gather_facts: false' in calling playbook.

config_management/roles/dev-tools/tasks/main.yml

@@ -0,0 +1,40 @@
+---
+# Set up Development Environment.
+
+- name: install development tools
+  package:
+    name: "{{ item }}"
+    state: present
+  with_items: 
+    # weave net dependencies
+    - make
+    - vagrant
+    # ansible dependencies
+    - python-pip
+    - python-dev
+    - libffi-dev
+    - libssl-dev
+    # terraform dependencies
+    - unzip
+    # other potentially useful tools:
+    - aufs-tools
+    - ethtool
+    - iputils-arping
+    - libpcap-dev
+    - git
+    - mercurial
+    - bc
+    - jq
+
+- name: install ansible
+  pip:
+    name: ansible
+    state: present
+
+- name: install terraform
+  unarchive:
+    src: 'https://releases.hashicorp.com/terraform/{{ terraform_version }}/terraform_{{ terraform_version }}_linux_{{ {"x86_64": "amd64", "i386": "386"}[ansible_architecture] }}.zip'
+    remote_src: yes
+    dest: /usr/bin
+    mode: 0555
+    creates: /usr/bin/terraform

config_management/roles/docker-configuration/files/docker_over_tcp.conf

@@ -0,0 +1,3 @@
+[Service]
+ExecStart=
+ExecStart=/usr/bin/docker daemon -H fd:// -H unix:///var/run/alt-docker.sock -H tcp://0.0.0.0:2375 -s overlay

config_management/roles/docker-configuration/tasks/main.yml

@@ -0,0 +1,34 @@
+---
+# Configure Docker
+# See also: https://docs.docker.com/engine/installation/linux/ubuntulinux/#install
+
+- name: ensure docker group is present (or create it)
+  group:
+    name: docker
+    state: present
+
+- name: add user to docker group (avoids sudo-ing)
+  user:
+    name: "{{ ansible_user }}"
+    group: docker
+    state: present
+
+- name: ensure docker's systemd directory exists
+  file:
+    path: /etc/systemd/system/docker.service.d
+    state: directory
+    recurse: yes
+
+- name: enable docker remote api over tcp
+  copy:
+    src: "{{ role_path }}/files/docker_over_tcp.conf"
+    dest: /etc/systemd/system/docker.service.d/docker_over_tcp.conf
+  register: docker_over_tcp
+
+- name: restart docker service
+  systemd: 
+    name: docker
+    state: restarted
+    daemon_reload: yes  # ensure docker_over_tcp.conf is picked up.
+    enabled: yes
+  when: docker_over_tcp.changed

config_management/roles/docker-from-docker-repo/tasks/debian.yml

@@ -0,0 +1,35 @@
+---
+# Debian / Ubuntu specific:
+
+- name: install dependencies for docker repository
+  package:
+    name: "{{ item }}"
+    state: present
+  with_items: 
+    - apt-transport-https
+    - ca-certificates
+
+- name: add apt key for the docker repository
+  apt_key: 
+    keyserver: hkp://ha.pool.sks-keyservers.net:80
+    id: 58118E89F3A912897C070ADBF76221572C52609D
+    state: present
+  register: apt_key_docker_repo
+
+- name: add docker's apt repository ({{ ansible_distribution | lower }}-{{ ansible_distribution_release }})
+  apt_repository:
+    repo: deb https://apt.dockerproject.org/repo {{ ansible_distribution | lower }}-{{ ansible_distribution_release }} main
+    state: present
+  register: apt_docker_repo
+
+- name: update apt's cache
+  apt: 
+    update_cache: yes
+  when: apt_key_docker_repo.changed or apt_docker_repo.changed
+
+- name: install docker-engine
+  package:
+    name: "{{ item }}"
+    state: present
+  with_items: 
+    - docker-engine={{ docker_version }}*

config_management/roles/docker-from-docker-repo/tasks/main.yml

@@ -0,0 +1,16 @@
+---
+# Set up Docker
+# See also: https://docs.docker.com/engine/installation/linux/ubuntulinux/#install
+
+- include_role:
+    name: docker-prerequisites
+
+# Distribution-specific tasks:
+- include: debian.yml
+  when: ansible_os_family == "Debian"
+
+- include: redhat.yml
+  when: ansible_os_family == "RedHat"
+
+- include_role:
+    name: docker-configuration

config_management/roles/docker-from-docker-repo/tasks/redhat.yml

@@ -0,0 +1,25 @@
+---
+# RedHat / CentOS specific:
+
+- name: add docker' yum repository (centos/{{ ansible_lsb.major_release }})
+  yum_repository:
+    name: docker
+    description: Docker YUM repo
+    file: external_repos
+    baseurl: https://yum.dockerproject.org/repo/main/centos/{{ ansible_lsb.major_release }}
+    enabled: yes
+    gpgkey: https://yum.dockerproject.org/gpg
+    gpgcheck: yes
+    state: present
+
+- name: update yum's cache
+  yum: 
+    name: "*"
+    update_cache: yes
+
+- name: install docker-engine
+  package:
+    name: "{{ item }}"
+    state: present
+  with_items: 
+    - docker-engine-{{ docker_version }}

config_management/roles/docker-from-get.docker.com/tasks/main.yml

@@ -0,0 +1,15 @@
+---
+# Set up Docker
+# See also: legacy gce.sh script
+
+- include_role:
+    name: docker-prerequisites
+
+- name: add apt key for the docker repository
+  shell: curl -sSL https://get.docker.com/gpg | sudo apt-key add -
+
+- name: install docker
+  shell: 'curl -sSL https://get.docker.com/ | sed -e s/docker-engine/docker-engine={{ docker_version }}*/ | sh'
+
+- include_role:
+    name: docker-configuration