Add support for Junit5 tests and fix CI that has been broken (apache#…

…4197) * Add support for Junit5 tests - upgrade surefire plugin - replace <forkMode>always</forkMode> with supported way in newer maven-surefire-plugin - maven-failsafe-plugin version must match maven-surefire-plugin version - remove invalid test case. Log4j is in the classpath for other reasons checked by running "mvn -pl tests/integration-tests-utils/pom.xml test -X" also verified by debugging that it's not in the system classpath, it is loaded by the MavenClassLoader - add assertj-core for fluent assertions in tests * Upgrade arquillian-junit-standalone * Fix error in pom.xml * Fix java.lang.ClassNotFoundException: javax.ws.rs.ProcessingException * Pick docker-java-api version that is compatible with arquillian-cube-docker * Fix snakeyaml issue * Fix running of bookkeeper in Docker in tests * Improve GitHub Actions CI workflow logging * Increase client tests timeout * Add script for picking fast ubuntu mirror for docker build * upgrade maven-dependency-plugin version (used for copying .tar.gz for docker image building) * Fix building of the docker image * Increase timeouts for integration tests * Use eclipse-temurin base image * Replace "FROM ubuntu:22.10" with "FROM ubuntu:22.04" to use LTS version of ubuntu 22.10 (kinetic) is EOL since July 20 2023 * Build docker image also for backward-compatibility-tests * Install gpg that is required by the scripts * Add UBUNTU_MIRROR * Remove redundancy from integration tests * Fix missing dependency * Fix classpath issues for arquillian-cube * Upgrade actions versions * Fix invalid references in native-io/pom.xml * Fix circe-checksum native library compilation * Migrate Gradle specific settings to Maven * Another attempt to disable backward-compat tests in integration test build job * Add working solution for skipping backward compat tests in the int tests * Attempt to fix issue in locating the file * Fix tar file extraction * Improve solution * Backward compat tests build commands without int tests * Copy tune-runner-vm and clean-disk from Pulsar CI and clean disk * Revisit skipping of backward compat tests once again * Use similar MAVEN_OPTS as there is in Pulsar * Fix directory * Add way to find git root directory * Revisit one more time * Add current-version-image back * Enable tests for backwardCompatTests * Don't create /ledgers in ZK if it already exists * Add uncaughtexceptionhandler * Add exclusions for reload4j * Use maven-surefire 2.8.1 for certain tests * Add failing test back in, also fails with surefire 2.8.1 * Exclude logging implementations * Revert "Use maven-surefire 2.8.1 for certain tests" This reverts commit 94e2b8e. * Exclude log4j * Add instructions how to run on Mac Apple Silicon * Find slf4j version * Block loading some classes * Change context classloader while creating new bookkeeper * Route commons logging to slf4j * Update shrinkwrap.version * Wait for ZK client to shutdown before returning from close * Upgrade Groovy version * Remove arquillian-junit-container which causes tests to be run twice * Skip backward compat tests unless -DbackwardCompatTests is passed * Delay closing of classloaders by 5 seconds * Fix checkstyle * Improve instructions for running tests * Upgrade Testcontainers and import the bom in dependencyManagement * Fix synchronization bug pointed out by spotbugs * upgrade markdown-link-check * Ignore checking Google calendar for dead links * Ignore anchors * Add 2 more patterns to dead link checker exclusions * Collect logs * Log integration tests to console to observe progress, disable retries * Use maven version provided by GitHub Action runner * Improve instructions for docker socket proxy * Add test timeout of 5 minutes for BookieShellTestBase * Fix test to match changes made in e41d672 * Upgrade mockito to 4.11.0 * Align hamcrest with junit * Revert "Upgrade mockito to 4.11.0" This reverts commit 55b7fcd. * Make JAVA_HOME optional * Tune Java's DNS cache timeouts for docker images * Use different directory for JDK 8 * Add tooling to containers that help debugging * Upgrade JMH version and configuration * Specify maven.compiler.release to fix issues with building with Java 11 * Remove gmavenplus plugin since there's already groovy-eclipse compiler in use * Upgrade apache-rat-plugin * Fix validation step * Don't import shaded class * Move freebuilder out of distributed libraries (there was invalid config for the annotation processor) * Remove use of shaded class in test * Upgrade junit5, assertj-core & awaitility to latest stable * Fix simpletest logic - LAC is updated asynchronously. Entries might be unconfirmed when read before ledger is closed. - The last entry id passed to the read method was invalid * Fix SimpleTestCommandTest --------- Co-authored-by: Matteo Merli <[email protected]> (cherry picked from commit 7bd91a7)
lhotari · Feb 9, 2024 · 7f1912b · 7f1912b
1 parent d2b9c7b
commit 7f1912b
Show file tree

Hide file tree

Showing 64 changed files with 1,387 additions and 457 deletions.
diff --git a/.github/actions/clean-disk/action.yml b/.github/actions/clean-disk/action.yml
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: clean disk
+description: makes some more space available on the disk by removing files
+inputs:
+  mode:
+    description: "Use 'full' to clean as much as possible"
+    required: false
+runs:
+  using: composite
+  steps:
+    - run: |
+        if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+          directories=(/usr/local/lib/android /opt/ghc)
+          if [[ "${{ inputs.mode }}" == "full" ]]; then
+            # remove these directories only when mode is 'full'
+            directories+=(/usr/share/dotnet /opt/hostedtoolcache/CodeQL)
+          fi
+          emptydir=/tmp/empty$$/
+          mkdir $emptydir
+          echo "::group::Available diskspace"
+          time df -BM / /mnt
+          echo "::endgroup::"
+          for directory in "${directories[@]}"; do
+            echo "::group::Removing $directory"
+            # fast way to delete a lot of files on linux
+            time sudo eatmydata rsync -a --delete $emptydir ${directory}/
+            time sudo eatmydata rm -rf ${directory}
+            time df -BM / /mnt
+            echo "::endgroup::"
+          done
+          echo "::group::Cleaning apt state"
+          time sudo bash -c "apt-get clean; apt-get autoclean; apt-get -y --purge autoremove"
+          time df -BM / /mnt
+          echo "::endgroup::"
+        fi
+        echo "::group::Available diskspace"
+        time df -BM / /mnt
+        echo "::endgroup::"
+      shell: bash
diff --git a/.github/actions/tune-runner-vm/action.yml b/.github/actions/tune-runner-vm/action.yml
@@ -24,8 +24,79 @@ runs:
   steps:
     - run: |
         if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+            echo "::group::Configure and tune OS"
             # Ensure that reverse lookups for current hostname are handled properly
             # Add the current IP address, long hostname and short hostname record to /etc/hosts file
             echo -e "$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)\t$(hostname -f) $(hostname -s)" | sudo tee -a /etc/hosts
+
+            # The default vm.swappiness setting is 60 which has a tendency to start swapping when memory
+            # consumption is high.
+            # Set vm.swappiness=1 to avoid swapping and allow high RAM usage
+            echo 1 | sudo tee /proc/sys/vm/swappiness
+            (
+              shopt -s nullglob
+              # Set swappiness to 1 for all cgroups and sub-groups
+              for swappiness_file in /sys/fs/cgroup/memory/*/memory.swappiness /sys/fs/cgroup/memory/*/*/memory.swappiness; do
+                echo 1 | sudo tee $swappiness_file > /dev/null
+              done
+            ) || true
+
+            # use "madvise" Linux Transparent HugePages (THP) setting
+            # https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html
+            # "madvise" is generally a better option than the default "always" setting
+            # Based on Azul instructions from https://docs.azul.com/prime/Enable-Huge-Pages#transparent-huge-pages-thp
+            echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
+            echo advise | sudo tee /sys/kernel/mm/transparent_hugepage/shmem_enabled
+            echo defer+madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
+            echo 1 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
+    
+            # tune filesystem mount options, https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
+            # commit=999999, effectively disables automatic syncing to disk (default is every 5 seconds)
+            # nobarrier/barrier=0, loosen data consistency on system crash (no negative impact to empheral CI nodes)
+            sudo mount -o remount,nodiscard,commit=999999,barrier=0 /
+            sudo mount -o remount,nodiscard,commit=999999,barrier=0 /mnt
+            # disable discard/trim at device level since remount with nodiscard doesn't seem to be effective
+            # https://www.spinics.net/lists/linux-ide/msg52562.html
+            for i in /sys/block/sd*/queue/discard_max_bytes; do
+              echo 0 | sudo tee $i
+            done
+            # disable any background jobs that run SSD discard/trim
+            sudo systemctl disable fstrim.timer || true
+            sudo systemctl stop fstrim.timer || true
+            sudo systemctl disable fstrim.service || true
+            sudo systemctl stop fstrim.service || true
+
+            # stop php-fpm
+            sudo systemctl stop php8.0-fpm.service || true
+            sudo systemctl stop php7.4-fpm.service || true
+            # stop mono-xsp4
+            sudo systemctl disable mono-xsp4.service || true
+            sudo systemctl stop mono-xsp4.service || true
+            sudo killall mono || true
+
+            # stop Azure Linux agent to save RAM
+            sudo systemctl stop walinuxagent.service || true
+          
+            # enable docker experimental mode which is
+            # required for using "docker build --squash" / "-Ddocker.squash=true"
+            daemon_json="$(sudo cat /etc/docker/daemon.json  | jq '.experimental = true')"
+            echo "$daemon_json" | sudo tee /etc/docker/daemon.json
+            # restart docker daemon
+            sudo systemctl restart docker
+            echo '::endgroup::'
+
+            # show memory
+            echo "::group::Available Memory"
+            free -m
+            echo '::endgroup::'
+            # show disk
+            echo "::group::Available diskspace"
+            df -BM
+            echo "::endgroup::"
+            # show cggroup
+            echo "::group::Cgroup settings for current cgroup $CURRENT_CGGROUP"
+            CURRENT_CGGROUP=$(cat /proc/self/cgroup | grep '0::' | awk -F: '{ print $3 }')
+            sudo cgget -a $CURRENT_CGGROUP || true
+            echo '::endgroup::'
         fi
-      shell: bash
+      shell: bash
diff --git a/.github/workflows/bot.yml b/.github/workflows/bot.yml
@@ -28,10 +28,10 @@ jobs:
 
     steps:
       - name: clone repository
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
 
       - name: bot actions
-        uses: actions/github-script@v1
+        uses: actions/github-script@v7
         env:
           PROVIDER: 'apache'
           REPOSITORY: 'bookkeeper'

diff --git a/.github/workflows/website-deploy.yaml b/.github/workflows/website-deploy.yaml
@@ -38,16 +38,16 @@ jobs:
     timeout-minutes: 180
     steps:
       - name: Checkout
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
 
       - name: Set up JDK 11
-        uses: actions/setup-java@v2
+        uses: actions/setup-java@v4
         with:
           distribution: 'temurin'
           java-version: 11
 
       - name: Setup NodeJS
-        uses: actions/setup-node@v2
+        uses: actions/setup-node@v4
         with:
           node-version: '14'
 

diff --git a/.github/workflows/website-pr-validation.yml b/.github/workflows/website-pr-validation.yml
@@ -20,37 +20,38 @@
 name: Website PR validation
 
 on:
+  workflow_dispatch:
   pull_request:
     branches:
       - master
       - branch-*
     paths:
       - 'site3/**'
       - '.github/workflows/website-pr-validation.yaml'
-    workflow_dispatch:
 
 jobs:
   website-pull-validation:
     runs-on: ubuntu-latest
     timeout-minutes: 60
     steps:
       - name: Checkout
-          uses: actions/checkout@v2
+        uses: actions/checkout@v4
 
-            - name: Set up JDK 11
-              uses: actions/setup-java@v2
-              with:
-                distribution: 'temurin'
-                java-version: 11
+      - name: Set up JDK 11
+        uses: actions/setup-java@v4
+        with:
+          distribution: 'temurin'
+          java-version: 11
 
-            - name: Setup NodeJS
-              uses: actions/setup-node@v2
-              with:
-                node-version: '14'
 
-            - name: Setup yarn
-              run: npm install -g yarn
+      - name: Setup NodeJS
+        uses: actions/setup-node@v4
+        with:
+          node-version: '16'
 
-            - name: Build website
-              run: |
-                ./site3/website/scripts/build-website.sh
+      - name: Setup yarn
+        run: npm install -g yarn
+
+      - name: Build website
+        run: |
+          ./site3/website/scripts/build-website.sh
diff --git a/bin/bookkeeper-daemon.sh b/bin/bookkeeper-daemon.sh
@@ -108,75 +108,111 @@ rotate_out_log ()
 
 mkdir -p "$BOOKIE_LOG_DIR"
 
-case $startStop in
-  (start)
-    if [ -f $pid_file ]; then
-      PREVIOUS_PID=$(cat $pid_file)
-      if kill -0 $PREVIOUS_PID > /dev/null 2>&1; then
-        echo $command running as process $PREVIOUS_PID.  Stop it first.
-        exit 1
-      fi
-    fi
-
-    rotate_out_log $out
-    echo starting $command, logging to $logfile
-    bookkeeper=$BK_HOME/bin/bookkeeper
-    nohup $bookkeeper $command "$@" > "$out" 2>&1 < /dev/null &
-    echo $! > $pid_file
-    sleep 1; head $out
-    sleep 2;
-    if ! kill -0 $! > /dev/null ; then
+start()
+{
+  if [ -f $pid_file ]; then
+    PREVIOUS_PID=$(cat $pid_file)
+    if kill -0 $PREVIOUS_PID > /dev/null 2>&1; then
+      echo $command running as process $PREVIOUS_PID.  Stop it first.
       exit 1
     fi
-    ;;
+  fi
 
-  (stop)
-    if [ -f $pid_file ]; then
-      TARGET_PID=$(cat $pid_file)
-      if kill -0 $TARGET_PID > /dev/null 2>&1; then
-        echo stopping $command
-        kill $TARGET_PID
-
-        count=0
-        location=$BOOKIE_LOG_DIR
-        while kill -0 $TARGET_PID > /dev/null 2>&1;
-        do
-          echo "Shutdown is in progress... Please wait..."
-          sleep 1
-          count=$(expr $count + 1)
-
-          if [ "$count" = "$BOOKIE_STOP_TIMEOUT" ]; then
-            break
-          fi
-         done
+  rotate_out_log $out
+  echo starting $command, logging to $logfile
+  bookkeeper=$BK_HOME/bin/bookkeeper
+  nohup $bookkeeper $command "$@" > "$out" 2>&1 < /dev/null &
+  echo $! > $pid_file
+  sleep 1; head $out
+  sleep 2;
+  if ! kill -0 $! > /dev/null ; then
+    exit 1
+  fi
+}
 
-        if [ "$count" != "$BOOKIE_STOP_TIMEOUT" ]; then
-          echo "Shutdown completed."
+stop()
+{
+  if [ -f $pid_file ]; then
+    TARGET_PID=$(cat $pid_file)
+    if kill -0 $TARGET_PID > /dev/null 2>&1; then
+      echo stopping $command
+      kill $TARGET_PID
+
+      count=0
+      location=$BOOKIE_LOG_DIR
+      while kill -0 $TARGET_PID > /dev/null 2>&1;
+      do
+        echo "Shutdown is in progress... Please wait..."
+        sleep 1
+        count=$(expr $count + 1)
+
+        if [ "$count" = "$BOOKIE_STOP_TIMEOUT" ]; then
+          break
         fi
+      done
 
-        if kill -0 $TARGET_PID > /dev/null 2>&1; then
-          fileName=$location/$command.out
-          $JAVA_HOME/bin/jstack $TARGET_PID > $fileName
-          echo Thread dumps are taken for analysis at $fileName
-          if [ "$1" == "-force" ]
-          then
-            echo forcefully stopping $command
-            kill -9 $TARGET_PID >/dev/null 2>&1
-            echo Successfully stopped the process
-          else
-            echo "WARNNING :  Bookie Server is not stopped completely."
+      if [ "$count" != "$BOOKIE_STOP_TIMEOUT" ]; then
+        echo "Shutdown completed."
+      fi
+
+      if kill -0 $TARGET_PID > /dev/null 2>&1; then
+        fileName=$location/$command.out
+        # Check for the java to use
+        if [[ -z ${JAVA_HOME} ]]; then
+          JSTACK=$(which jstack)
+          if [ $? -ne 0 ]; then
+            echo "Error: JAVA_HOME not set, and no jstack executable found in $PATH." 1>&2
             exit 1
           fi
+        else
+          JSTACK=${JAVA_HOME}/bin/jstack
+        fi
+        $JSTACK $TARGET_PID > $fileName
+        echo Thread dumps are taken for analysis at $fileName
+        if [ "$1" == "-force" ]
+        then
+          echo forcefully stopping $command
+          kill -9 $TARGET_PID >/dev/null 2>&1
+          echo Successfully stopped the process
+        else
+          echo "WARNNING :  Bookie Server is not stopped completely."
+          exit 1
         fi
-      else
-        echo no $command to stop
       fi
-      rm $pid_file
     else
       echo no $command to stop
     fi
+    rm $pid_file
+  else
+    echo no $command to stop
+  fi
+}
+case $startStop in
+  (start)
+    start "$*"
     ;;
 
+  (stop)
+    stop $1
+    ;;
+  (restart)
+    forceStopFlag=$(echo "$*"|grep "\-force")
+    if [[ "$forceStopFlag" != "" ]]
+    then
+      stop "-force"
+    else
+      stop
+    fi
+    if [ "$?" == 0 ]
+    then
+      sleep 3
+      paramaters="$*"
+      startParamaters=${paramaters//-force/}
+      start "$startParamaters"
+    else
+      echo "WARNNING :  $command failed restart, for $command is not stopped completely."
+    fi
+    ;;
   (*)
     usage
     echo $supportedargs