Add support for Junit5 tests and fix CI that has been broken (#4197)

* Add support for Junit5 tests - upgrade surefire plugin - replace <forkMode>always</forkMode> with supported way in newer maven-surefire-plugin - maven-failsafe-plugin version must match maven-surefire-plugin version - remove invalid test case. Log4j is in the classpath for other reasons checked by running "mvn -pl tests/integration-tests-utils/pom.xml test -X" also verified by debugging that it's not in the system classpath, it is loaded by the MavenClassLoader - add assertj-core for fluent assertions in tests * Upgrade arquillian-junit-standalone * Fix error in pom.xml * Fix java.lang.ClassNotFoundException: javax.ws.rs.ProcessingException * Pick docker-java-api version that is compatible with arquillian-cube-docker * Fix snakeyaml issue * Fix running of bookkeeper in Docker in tests * Improve GitHub Actions CI workflow logging * Increase client tests timeout * Add script for picking fast ubuntu mirror for docker build * upgrade maven-dependency-plugin version (used for copying .tar.gz for docker image building) * Fix building of the docker image * Increase timeouts for integration tests * Use eclipse-temurin base image * Replace "FROM ubuntu:22.10" with "FROM ubuntu:22.04" to use LTS version of ubuntu 22.10 (kinetic) is EOL since July 20 2023 * Build docker image also for backward-compatibility-tests * Install gpg that is required by the scripts * Add UBUNTU_MIRROR * Remove redundancy from integration tests * Fix missing dependency * Fix classpath issues for arquillian-cube * Upgrade actions versions * Fix invalid references in native-io/pom.xml * Fix circe-checksum native library compilation * Migrate Gradle specific settings to Maven * Another attempt to disable backward-compat tests in integration test build job * Add working solution for skipping backward compat tests in the int tests * Attempt to fix issue in locating the file * Fix tar file extraction * Improve solution * Backward compat tests build commands without int tests * Copy tune-runner-vm and clean-disk from Pulsar CI and clean disk * Revisit skipping of backward compat tests once again * Use similar MAVEN_OPTS as there is in Pulsar * Fix directory * Add way to find git root directory * Revisit one more time * Add current-version-image back * Enable tests for backwardCompatTests * Don't create /ledgers in ZK if it already exists * Add uncaughtexceptionhandler * Add exclusions for reload4j * Use maven-surefire 2.8.1 for certain tests * Add failing test back in, also fails with surefire 2.8.1 * Exclude logging implementations * Revert "Use maven-surefire 2.8.1 for certain tests" This reverts commit 94e2b8e. * Exclude log4j * Add instructions how to run on Mac Apple Silicon * Find slf4j version * Block loading some classes * Change context classloader while creating new bookkeeper * Route commons logging to slf4j * Update shrinkwrap.version * Wait for ZK client to shutdown before returning from close * Upgrade Groovy version * Remove arquillian-junit-container which causes tests to be run twice * Skip backward compat tests unless -DbackwardCompatTests is passed * Delay closing of classloaders by 5 seconds * Fix checkstyle * Improve instructions for running tests * Upgrade Testcontainers and import the bom in dependencyManagement * Fix synchronization bug pointed out by spotbugs * upgrade markdown-link-check * Ignore checking Google calendar for dead links * Ignore anchors * Add 2 more patterns to dead link checker exclusions * Collect logs * Log integration tests to console to observe progress, disable retries * Use maven version provided by GitHub Action runner * Improve instructions for docker socket proxy * Add test timeout of 5 minutes for BookieShellTestBase * Fix test to match changes made in e41d672 * Upgrade mockito to 4.11.0 * Align hamcrest with junit * Revert "Upgrade mockito to 4.11.0" This reverts commit 55b7fcd. * Make JAVA_HOME optional * Tune Java's DNS cache timeouts for docker images * Use different directory for JDK 8 * Add tooling to containers that help debugging * Upgrade JMH version and configuration * Specify maven.compiler.release to fix issues with building with Java 11 * Remove gmavenplus plugin since there's already groovy-eclipse compiler in use * Upgrade apache-rat-plugin * Fix validation step * Don't import shaded class * Move freebuilder out of distributed libraries (there was invalid config for the annotation processor) * Remove use of shaded class in test * Upgrade junit5, assertj-core & awaitility to latest stable * Fix simpletest logic - LAC is updated asynchronously. Entries might be unconfirmed when read before ledger is closed. - The last entry id passed to the read method was invalid * Fix SimpleTestCommandTest --------- Co-authored-by: Matteo Merli <[email protected]>
apache · Feb 8, 2024 · 7bd91a7 · 7bd91a7
1 parent 6fd8778
commit 7bd91a7
Show file tree

Hide file tree

Showing 70 changed files with 1,376 additions and 461 deletions.
diff --git a/.dlc.json b/.dlc.json
@@ -62,6 +62,18 @@
     },
     {
       "pattern": "^http://daxue.qq.com/content/content/id/2492"
+    },
+    {
+      "pattern": "^https://calendar.google.com/"
+    },
+    {
+      "pattern": "^#"
+    },
+    {
+      "pattern": ".*\\{\\{.*"
+    },
+    {
+      "pattern": "^//"
     }
   ],
   "timeout": "10s",

diff --git a/.github/actions/clean-disk/action.yml b/.github/actions/clean-disk/action.yml
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: clean disk
+description: makes some more space available on the disk by removing files
+inputs:
+  mode:
+    description: "Use 'full' to clean as much as possible"
+    required: false
+runs:
+  using: composite
+  steps:
+    - run: |
+        if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+          directories=(/usr/local/lib/android /opt/ghc)
+          if [[ "${{ inputs.mode }}" == "full" ]]; then
+            # remove these directories only when mode is 'full'
+            directories+=(/usr/share/dotnet /opt/hostedtoolcache/CodeQL)
+          fi
+          emptydir=/tmp/empty$$/
+          mkdir $emptydir
+          echo "::group::Available diskspace"
+          time df -BM / /mnt
+          echo "::endgroup::"
+          for directory in "${directories[@]}"; do
+            echo "::group::Removing $directory"
+            # fast way to delete a lot of files on linux
+            time sudo eatmydata rsync -a --delete $emptydir ${directory}/
+            time sudo eatmydata rm -rf ${directory}
+            time df -BM / /mnt
+            echo "::endgroup::"
+          done
+          echo "::group::Cleaning apt state"
+          time sudo bash -c "apt-get clean; apt-get autoclean; apt-get -y --purge autoremove"
+          time df -BM / /mnt
+          echo "::endgroup::"
+        fi
+        echo "::group::Available diskspace"
+        time df -BM / /mnt
+        echo "::endgroup::"
+      shell: bash
diff --git a/.github/actions/tune-runner-vm/action.yml b/.github/actions/tune-runner-vm/action.yml
@@ -24,8 +24,79 @@ runs:
   steps:
     - run: |
         if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+            echo "::group::Configure and tune OS"
             # Ensure that reverse lookups for current hostname are handled properly
             # Add the current IP address, long hostname and short hostname record to /etc/hosts file
             echo -e "$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)\t$(hostname -f) $(hostname -s)" | sudo tee -a /etc/hosts
+
+            # The default vm.swappiness setting is 60 which has a tendency to start swapping when memory
+            # consumption is high.
+            # Set vm.swappiness=1 to avoid swapping and allow high RAM usage
+            echo 1 | sudo tee /proc/sys/vm/swappiness
+            (
+              shopt -s nullglob
+              # Set swappiness to 1 for all cgroups and sub-groups
+              for swappiness_file in /sys/fs/cgroup/memory/*/memory.swappiness /sys/fs/cgroup/memory/*/*/memory.swappiness; do
+                echo 1 | sudo tee $swappiness_file > /dev/null
+              done
+            ) || true
+
+            # use "madvise" Linux Transparent HugePages (THP) setting
+            # https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html
+            # "madvise" is generally a better option than the default "always" setting
+            # Based on Azul instructions from https://docs.azul.com/prime/Enable-Huge-Pages#transparent-huge-pages-thp
+            echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
+            echo advise | sudo tee /sys/kernel/mm/transparent_hugepage/shmem_enabled
+            echo defer+madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
+            echo 1 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
+    
+            # tune filesystem mount options, https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
+            # commit=999999, effectively disables automatic syncing to disk (default is every 5 seconds)
+            # nobarrier/barrier=0, loosen data consistency on system crash (no negative impact to empheral CI nodes)
+            sudo mount -o remount,nodiscard,commit=999999,barrier=0 /
+            sudo mount -o remount,nodiscard,commit=999999,barrier=0 /mnt
+            # disable discard/trim at device level since remount with nodiscard doesn't seem to be effective
+            # https://www.spinics.net/lists/linux-ide/msg52562.html
+            for i in /sys/block/sd*/queue/discard_max_bytes; do
+              echo 0 | sudo tee $i
+            done
+            # disable any background jobs that run SSD discard/trim
+            sudo systemctl disable fstrim.timer || true
+            sudo systemctl stop fstrim.timer || true
+            sudo systemctl disable fstrim.service || true
+            sudo systemctl stop fstrim.service || true
+
+            # stop php-fpm
+            sudo systemctl stop php8.0-fpm.service || true
+            sudo systemctl stop php7.4-fpm.service || true
+            # stop mono-xsp4
+            sudo systemctl disable mono-xsp4.service || true
+            sudo systemctl stop mono-xsp4.service || true
+            sudo killall mono || true
+
+            # stop Azure Linux agent to save RAM
+            sudo systemctl stop walinuxagent.service || true
+          
+            # enable docker experimental mode which is
+            # required for using "docker build --squash" / "-Ddocker.squash=true"
+            daemon_json="$(sudo cat /etc/docker/daemon.json  | jq '.experimental = true')"
+            echo "$daemon_json" | sudo tee /etc/docker/daemon.json
+            # restart docker daemon
+            sudo systemctl restart docker
+            echo '::endgroup::'
+
+            # show memory
+            echo "::group::Available Memory"
+            free -m
+            echo '::endgroup::'
+            # show disk
+            echo "::group::Available diskspace"
+            df -BM
+            echo "::endgroup::"
+            # show cggroup
+            echo "::group::Cgroup settings for current cgroup $CURRENT_CGGROUP"
+            CURRENT_CGGROUP=$(cat /proc/self/cgroup | grep '0::' | awk -F: '{ print $3 }')
+            sudo cgget -a $CURRENT_CGGROUP || true
+            echo '::endgroup::'
         fi
-      shell: bash
+      shell: bash