Squashed commit of the following:

commit 2b08b6d Author: Nell Shamrell-Harrington <[email protected]> Date: Sun Sep 13 23:10:18 2020 -0700 docs: Updates reference to gitter to point to the Discord server (#3858) Signed-off-by: Nell Shamrell <[email protected]> commit 27ba8b9 Author: James Turnbull <[email protected]> Date: Sat Sep 12 17:26:13 2020 -0400 chore: Removing CI references to Nix (#3844) Signed-off-by: James Turnbull <[email protected]> commit 7c95aa9 Author: James Turnbull <[email protected]> Date: Sat Sep 12 17:25:54 2020 -0400 chore: Fix make prepare-environment (#3841) commit cc00cf0 Author: James Turnbull <[email protected]> Date: Sat Sep 12 13:38:39 2020 -0400 chore: Added workflow_dispatch for the env workflow (#3842) commit b7f656e Author: James Turnbull <[email protected]> Date: Sat Sep 12 12:14:21 2020 -0400 chore: Fix Debian post-install process to reload systemd daemons (#3832) commit c2078b7 Author: binarylogic <[email protected]> Date: Sat Sep 12 10:28:41 2020 -0400 chore: Remove check-code.sh since it is no longer used Signed-off-by: binarylogic <[email protected]> commit 7887d4e Author: James Turnbull <[email protected]> Date: Fri Sep 11 23:50:56 2020 -0400 chore: Added cross installation to fix nightly builds (#3831) commit 648e82c Author: Ana Hobden <[email protected]> Date: Fri Sep 11 17:33:29 2020 -0700 chore(platforms): Migrate x86 gnu target to cross, add new aarch64 gnu target (no sasl) (#3657) * Add initial cross adoption Signed-off-by: Ana Hobden <[email protected]> * Add archiving Signed-off-by: Ana Hobden <[email protected]> * Clean up the makefile a bit and get tarballs working Signed-off-by: Ana Hobden <[email protected]> * checkpoint Signed-off-by: Ana Hobden <[email protected]> * wip Signed-off-by: Ana Hobden <[email protected]> * Add to CI Signed-off-by: Ana Hobden <[email protected]> * Add autoinstall to CI Signed-off-by: Ana Hobden <[email protected]> * Remove cruft Signed-off-by: Ana Hobden <[email protected]> * Can't test yet Signed-off-by: Ana Hobden <[email protected]> * Autoinstall in e2e Signed-off-by: Ana Hobden <[email protected]> * Readd docker image Signed-off-by: Ana Hobden <[email protected]> * More comments Signed-off-by: Ana Hobden <[email protected]> * Set right CI task Signed-off-by: Ana Hobden <[email protected]> * Add note about TZ issue Signed-off-by: Ana Hobden <[email protected]> commit b28eb18 Author: Bruce Guenter <[email protected]> Date: Fri Sep 11 16:22:58 2020 -0600 chore(networking): Rework internal auto concurrency tests (#3802) * Move auto concurrency test case data into TOML files * Use tokio manual time tracking to speed up tests By pausing and then manually advancing the clock, this provides for both very fast running tests and nearly perfect time measurements for highly repeatable test results. * Collect all the failures for each test This has the collateral benefit of being able to drop the large `assert_within` macro that was only used here. * The tests should now work predictably on MacOS Signed-off-by: Bruce Guenter <[email protected]> commit b37dea7 Author: Ana Hobden <[email protected]> Date: Fri Sep 11 14:53:14 2020 -0700 chore: Add some additional Event impls (#3621) * Move log_event to new file Signed-off-by: Ana Hobden <[email protected]> * Move value to new file Signed-off-by: Ana Hobden <[email protected]> * Move log_schema to config Signed-off-by: Ana Hobden <[email protected]> * chore: Add some event impls Signed-off-by: Ana Hobden <[email protected]> * Do some namespace squashing Signed-off-by: Ana Hobden <[email protected]> * Fixup lints Signed-off-by: Ana Hobden <[email protected]> * Remove From<Event> for Vec<u8> Signed-off-by: Ana Hobden <[email protected]> * Fixup leveldb Signed-off-by: Ana Hobden <[email protected]> * fmt Signed-off-by: Ana Hobden <[email protected]> * Jean's feedback Signed-off-by: Ana Hobden <[email protected]> commit f03898e Author: Luke Steensen <[email protected]> Date: Fri Sep 11 15:53:51 2020 -0500 enhancement(tokenizer transform): add internal events (#3807) Signed-off-by: Luke Steensen <[email protected]> commit ff555aa Author: Luke Steensen <[email protected]> Date: Fri Sep 11 15:53:27 2020 -0500 enhancement(dedupe transform): add internal events (#3809) Closes #3393 Signed-off-by: Luke Steensen <[email protected]> Co-authored-by: Jean Mertz <[email protected]> commit 54a4d8e Author: MOZGIII <[email protected]> Date: Fri Sep 11 17:09:57 2020 +0300 chore: Generate Kubernetes distribution YAMLs from Helm Chart (#3614) * Add a script to manage the Kubernetes YAML configs Signed-off-by: MOZGIII <[email protected]> * Update Helm Chart to generate config more like YAML files Signed-off-by: MOZGIII <[email protected]> * Switch to generated YAML Signed-off-by: MOZGIII <[email protected]> * Define check-all target in multiple lines Signed-off-by: MOZGIII <[email protected]> * Add Kubernetes YAML tasks to Makefile Signed-off-by: MOZGIII <[email protected]> * Add check-kubernetes-yaml to CI Signed-off-by: MOZGIII <[email protected]> * Add kubernetes-yaml.sh to .github/CODEOWNERS Signed-off-by: MOZGIII <[email protected]> * Add /scripts/kubernetes-yaml/ to .github/CODEOWNERS Signed-off-by: MOZGIII <[email protected]> * Add a TODO for passing the app-version Signed-off-by: MOZGIII <[email protected]> commit 68a5c43 Author: Do Duy <[email protected]> Date: Fri Sep 11 10:23:03 2020 +0700 docs: Add docs for retry_backoff_secs option (#3819) Signed-off-by: Duy Do <[email protected]> commit 1a1352b Author: James Turnbull <[email protected]> Date: Thu Sep 10 17:34:16 2020 -0400 chore: Remove docker-compose from integration tests (#3622) commit 94def0f Author: Luke Steensen <[email protected]> Date: Thu Sep 10 16:17:03 2020 -0500 enhancement(reduce transform): add internal events (#3812) Closes #3401 Signed-off-by: Luke Steensen <[email protected]> commit 74b43b7 Author: Bruce Guenter <[email protected]> Date: Thu Sep 10 14:05:52 2020 -0600 fix(networking): Rework auto concurrency backpressure logic (#3804) Previously, all responses were marked as not being backpressure. This changes that logic to: 1. Treat RetryAction::Retry responses as backpressure 2. Only use the RTT values from RetryAction::Successful responses Signed-off-by: Bruce Guenter <[email protected]> commit eb09b92 Author: Ana Hobden <[email protected]> Date: Thu Sep 10 13:01:23 2020 -0700 chore: Split up event (#3619) * Move log_event to new file Signed-off-by: Ana Hobden <[email protected]> * Move value to new file Signed-off-by: Ana Hobden <[email protected]> * Move log_schema to config Signed-off-by: Ana Hobden <[email protected]> * Do some namespace squashing Signed-off-by: Ana Hobden <[email protected]> * Fixup lints Signed-off-by: Ana Hobden <[email protected]> * Fixup leveldb Signed-off-by: Ana Hobden <[email protected]> * fmt Signed-off-by: Ana Hobden <[email protected]> commit 2fd9ccf Author: Jesse Szwedko <[email protected]> Date: Thu Sep 10 15:34:52 2020 -0400 chore: Add expected review counts to CONTRIBUTING.md (#3666) * chore: Add expected review counts to CONTRIBUTING.md Signed-off-by: Jesse Szwedko <[email protected]>
vectordotdev · Sep 14, 2020 · 4467875 · 4467875
1 parent 8c34364
commit 4467875
Show file tree

Hide file tree

Showing 130 changed files with 2,446 additions and 2,022 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -38,6 +38,8 @@
 /scripts/release-helm.sh @MOZGIII
 /scripts/deploy-kubernetes-test.sh @MOZGIII @JeanMertz
 /scripts/test-e2e-kubernetes.sh @MOZGIII @JeanMertz
+/scripts/kubernetes-yaml.sh @MOZGIII
+/scripts/kubernetes-yaml/ @MOZGIII
 
 /src/metrics.rs @lukesteensen
 /src/main.rs @lukesteensen

diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
@@ -40,11 +40,13 @@ on:
 
 env:
   VERBOSE: true
+  AUTOINSTALL: true
   RUST_TEST_THREADS: 1
   TEST_LOG: vector=debug
   RUST_BACKTRACE: full
   AWS_ACCESS_KEY_ID: "dummy"
   AWS_SECRET_ACCESS_KEY: "dummy"
+  CONTAINER_TOOL: "docker"
 
 jobs:
   cancel-previous:
@@ -152,7 +154,7 @@ jobs:
           KUBERNETES_VERSION: ${{ matrix.kubernetes_version.version }}
           MINIKUBE_VERSION: ${{ matrix.minikube_version }}
           CONTAINER_RUNTIME: ${{ matrix.container_runtime }}
-      - run: USE_CONTAINER=none make slim-builds
+      - run: make slim-builds
       - run: make test-e2e-kubernetes
         env:
           USE_MINIKUBE_CACHE: "true"

diff --git a/.github/workflows/environment.yml b/.github/workflows/environment.yml
@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - master
+  workflow_dispatch:
 
 env:
   VERBOSE: true
@@ -17,5 +18,11 @@ jobs:
     steps:
       - run: docker login https://docker.pkg.github.com -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }}
       - uses: actions/checkout@v2
+      - name: free disk space
+        run: |
+          sudo swapoff -a
+          sudo rm -f /swapfile
+          sudo apt clean
+          docker rmi $(docker image ls -aq)
+          df -h
       - run: make environment-prepare
-      - run: docker push docker.pkg.github.com/timberio/vector/environment:${{ github.sha }}
diff --git a/.github/workflows/install-sh.yml b/.github/workflows/install-sh.yml
@@ -14,5 +14,4 @@ jobs:
       - env:
           AWS_ACCESS_KEY_ID: "${{ secrets.CI_AWS_ACCESS_KEY_ID }}"
           AWS_SECRET_ACCESS_KEY: "${{ secrets.CI_AWS_SECRET_ACCESS_KEY }}"
-          USE_CONTAINER: none
         run: make sync-install
diff --git a/.github/workflows/lints.yml b/.github/workflows/lints.yml
@@ -13,6 +13,7 @@ env:
   RUST_BACKTRACE: full
   AWS_ACCESS_KEY_ID: "dummy"
   AWS_SECRET_ACCESS_KEY: "dummy"
+  CONTAINER_TOOL: "docker"
 
 jobs:
   cancel-previous:
@@ -52,6 +53,7 @@ jobs:
       - run: make check-version
       - run: make check-scripts
       - run: make check-helm
+      - run: make check-kubernetes-yaml
       - uses: EmbarkStudios/cargo-deny-action@v1
         with:
           command: check advisories

diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml
@@ -6,7 +6,9 @@ on:
     - cron: "0 4 * * *"
 
 env:
+  AUTOINSTALL: true
   CHANNEL: nightly
+  VERBOSE: true
 
 jobs:
   build-x86_64-unknown-linux-musl-packages:
@@ -73,7 +75,6 @@ jobs:
       - name: "Build archive"
         env:
           TARGET: "x86_64-apple-darwin"
-          USE_CONTAINER: none
           NATIVE_BUILD: true
         run: |
           export PATH="$HOME/.cargo/bin:$PATH"
@@ -130,7 +131,6 @@ jobs:
           export RUST_LTO=""
           export TARGET="x86_64-pc-windows-msvc"
           export NATIVE_BUILD="true"
-          export USE_CONTAINER="none"
           make package-archive
       - name: "Build package"
         shell: bash
@@ -173,7 +173,6 @@ jobs:
           DOCKER_USERNAME: "${{ secrets.CI_DOCKER_USERNAME }}"
           DOCKER_PASSWORD: "${{ secrets.CI_DOCKER_PASSWORD }}"
           PLATFORM: "linux/amd64,linux/arm64"
-          USE_CONTAINER: none
         run: |
           ./scripts/upgrade-docker.sh
           export VERSION=$(make version)
@@ -232,7 +231,6 @@ jobs:
       - env:
           AWS_ACCESS_KEY_ID: "${{ secrets.CI_AWS_ACCESS_KEY_ID }}"
           AWS_SECRET_ACCESS_KEY: "${{ secrets.CI_AWS_SECRET_ACCESS_KEY }}"
-          USE_CONTAINER: none
         run: make release-s3
 
   release-helm:
@@ -249,5 +247,4 @@ jobs:
         env:
           AWS_ACCESS_KEY_ID: "${{ secrets.CI_AWS_ACCESS_KEY_ID }}"
           AWS_SECRET_ACCESS_KEY: "${{ secrets.CI_AWS_SECRET_ACCESS_KEY }}"
-          USE_CONTAINER: none
         run: make release-helm
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -6,6 +6,11 @@ on:
       - v0.*
       - v1.*
 
+
+env:
+  AUTOINSTALL: true
+  VERBOSE: true
+
 jobs:
   build-x86_64-unknown-linux-musl-packages:
     runs-on: ubuntu-latest
@@ -71,7 +76,6 @@ jobs:
       - name: "Build archive"
         env:
           TARGET: "x86_64-apple-darwin"
-          USE_CONTAINER: none
           NATIVE_BUILD: true
         run: |
           export PATH="$HOME/.cargo/bin:$PATH"
@@ -95,7 +99,6 @@ jobs:
           export RUST_LTO=""
           export TARGET="x86_64-pc-windows-msvc"
           export NATIVE_BUILD="true"
-          export USE_CONTAINER="none"
           make package-archive
       - name: "Build package"
         shell: bash
@@ -138,7 +141,6 @@ jobs:
           DOCKER_USERNAME: "${{ secrets.CI_DOCKER_USERNAME }}"
           DOCKER_PASSWORD: "${{ secrets.CI_DOCKER_PASSWORD }}"
           PLATFORM: "linux/amd64,linux/arm64,linux/arm"
-          USE_CONTAINER: none
         run: |
           ./scripts/upgrade-docker.sh
           export VERSION=$(make version)
@@ -197,7 +199,6 @@ jobs:
       - env:
           AWS_ACCESS_KEY_ID: "${{ secrets.CI_AWS_ACCESS_KEY_ID }}"
           AWS_SECRET_ACCESS_KEY: "${{ secrets.CI_AWS_SECRET_ACCESS_KEY }}"
-          USE_CONTAINER: none
         run: make release-s3
 
   release-github:
@@ -329,5 +330,4 @@ jobs:
         env:
           AWS_ACCESS_KEY_ID: "${{ secrets.CI_AWS_ACCESS_KEY_ID }}"
           AWS_SECRET_ACCESS_KEY: "${{ secrets.CI_AWS_SECRET_ACCESS_KEY }}"
-          USE_CONTAINER: none
         run: make release-helm
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -36,11 +36,13 @@ on:
 
 env:
   VERBOSE: true
+  AUTOINSTALL: true
   RUST_TEST_THREADS: 1
   TEST_LOG: vector=debug
   RUST_BACKTRACE: full
   AWS_ACCESS_KEY_ID: "dummy"
   AWS_SECRET_ACCESS_KEY: "dummy"
+  CONTAINER_TOOL: "docker"
 
 jobs:
   cancel-previous:
@@ -54,7 +56,7 @@ jobs:
           access_token: ${{ secrets.GITHUB_TOKEN }}
 
   test-linux:
-    name: Unit - Linux
+    name: Unit - x86_64-unknown-linux-gnu
     runs-on: ubuntu-20.04
     if: |
       !contains(github.event.pull_request.labels.*.name, 'ci-condition: skip')
@@ -71,6 +73,46 @@ jobs:
       - run: make slim-builds
       - run: make test
 
+  # TODO: Upgrade to test once the tz issues are solved.
+  # https://github.com/timberio/vector/issues/3771
+  build-x86_64-unknown-linux-gnu:
+    name: Unit - x86_64-unknown-linux-gnu
+    runs-on: ubuntu-20.04
+    if: |
+      !contains(github.event.pull_request.labels.*.name, 'ci-condition: skip')
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/cache@v2
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+      - run: sudo bash scripts/environment/bootstrap-ubuntu-20.04.sh
+      - run: bash scripts/environment/prepare.sh
+      - run: make slim-builds
+      - run: make build-x86_64-unknown-linux-gnu
+
+  # TODO: Upgrade to test once the tz issues are solved.
+  # https://github.com/timberio/vector/issues/3771
+  build-aarch64-unknown-linux-gnu:
+    name: Unit - aarch64-unknown-linux-gnu
+    runs-on: ubuntu-20.04
+    if: |
+      !contains(github.event.pull_request.labels.*.name, 'ci-condition: skip')
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/cache@v2
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+      - run: sudo bash scripts/environment/bootstrap-ubuntu-20.04.sh
+      - run: bash scripts/environment/prepare.sh
+      - run: make slim-builds
+      - run: make build-aarch64-unknown-linux-gnu
+
   test-mac:
     name: Unit - Mac
     # Full CI suites for this platform were only recently introduced.
@@ -103,7 +145,6 @@ jobs:
     steps:
       - uses: actions/checkout@v1
       - env:
-          USE_CONTAINER: none
           RUSTFLAGS: "-D warnings"
         run: cargo test --no-default-features --features default-msvc
 

diff --git a/.meta/sources/docker.toml.erb b/.meta/sources/docker.toml.erb
@@ -81,6 +81,14 @@ incomplete message (i.e. partial events). If set to `""`, no field will be \
 added to partial event. This allows to opt-out of partial event detection.\
 """
 
+[sources.docker.options.retry_backoff_secs]
+type = "uint"
+common = false
+default = 1
+description = """\
+The amount of time to wait before retrying after an error.\
+"""
+
 [sources.docker.fields.log.fields.container_created_at]
 type = "timestamp"
 examples = ["2019-11-01T21:15:47.443232Z"]

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -165,14 +165,24 @@ docs: fix typos
 
 #### Reviews & Approvals
 
-All pull requests must be reviewed and approved by at least one Vector team
-member. The review process is outlined in the [Review guide](REVIEWING.md).
+All pull requests should be reviewed by:
+
+- No review required for cosmetic changes like whitespace, typos, and spelling
+  by a maintainer
+- One Vector team member for minor changes or trivial changes from contributors
+- Two Vector team members for major changes
+- Three Vector team members for RFCs
+
+If there are any CODEOWNERs automatically assigned, you should also wait for
+their review.
+
+The review process is outlined in the [Review guide](REVIEWING.md).
 
 #### Merge Style
 
 All pull requests are squashed and merged. We generally discourage large pull
 requests that are over 300-500 lines of diff. If you would like to propose a
-change that is larger we suggest coming onto our gitter channel and discuss it
+change that is larger we suggest coming onto our [Discord server](https://chat.vector.dev/) and discuss it
 with one of our engineers. This way we can talk through the solution and
 discuss if a change that large is even needed! This will produce a quicker
 response to the change and likely produce code that aligns better with our
@@ -209,13 +219,13 @@ $ curl -u "$GITHUB_USERNAME:$GITHUB_TOKEN" \
 
 We're super excited to have you interested in working on Vector! Before you start you should pick how you want to develop.
 
-For small or first-time contributions, we recommend the Docker method. If you do a lot of contributing, try adopting the Nix method! It'll be way faster and feel more smooth. Prefer to do it yourself? That's fine too!
+For small or first-time contributions, we recommend the Docker method. Prefer to do it yourself? That's fine too!
 
 #### Using a Docker or Podman environment
 
 > **Targets:** You can use this method to produce AARCH64, Arm6/7, as well as x86/64 Linux builds.
 
-Since not everyone has a full working native environment, or can use Nix, we took our Nix environment and stuffed it into a Docker (or Podman) container!
+Since not everyone has a full working native environment, we took our environment and stuffed it into a Docker (or Podman) container!
 
 This is ideal for users who want it to "Just work" and just want to start contributing. It's also what we use for our CI, so you know if it breaks we can't do anything else until we fix it. 😉
 

diff --git a/Cargo.toml b/Cargo.toml
@@ -12,6 +12,12 @@ publish = false
 [profile.bench]
 debug = true
 
+[profile.release]
+opt-level = 3
+debug = false
+lto = "fat"
+codegen-units = 1
+
 [package.metadata.deb]
 maintainer-scripts = "distribution/debian/scripts/"
 conf-files = ["/etc/vector/vector.toml"]
@@ -211,6 +217,12 @@ default-cmake = ["sources", "transforms", "sinks", "vendor-all", "unix", "leveld
 # TODO: Enable SASL https://github.com/timberio/vector/pull/3081#issuecomment-659298042
 default-msvc = ["sources", "transforms", "sinks", "vendor-openssl", "vendor-libz", "leveldb-cmake", "rdkafka-cmake"]
 
+# Target specific release features.
+# The `make` tasks will select this according to the appropriate triple.
+# Use this section to turn off or on specific features for specific triples.
+target-x86_64-unknown-linux-gnu = ["sources", "transforms", "sinks", "vendor-all", "unix", "leveldb-cmake", "rdkafka-cmake"]
+target-aarch64-unknown-linux-gnu = ["sources", "transforms", "sinks", "vendor-openssl", "vendor-libz", "unix", "leveldb-cmake", "rdkafka-cmake"]
+
 # Enables features that work only on systems providing `cfg(unix)`
 unix = ["jemallocator"]
 # These are **very** useful on Cross compilations!