[Remote Vector Index Build] Initial implementation of Index Build Ser…

…vice (#2532) * fix broken build flag, move build to one directory (#2442) * move build to one directory, fix broken flag Signed-off-by: Samuel Herman <[email protected]> * fix make path Signed-off-by: Samuel Herman <[email protected]> * changelog update Signed-off-by: Samuel Herman <[email protected]> * add fix for classpath change and for cmake discovery on macos Signed-off-by: Samuel Herman <[email protected]> * fix make discovery for gradle Signed-off-by: Samuel Herman <[email protected]> * fix cmake path for macOS Signed-off-by: Samuel Herman <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]> * Update package name to fix compilation issue (#2513) * Update package name to fix compilation issue Core renamed this package in opensearch-project/OpenSearch#17272 This commit renames package accordingly Signed-off-by: Balasubramanian <[email protected]> * Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions (#2506) (#2508) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1) Co-authored-by: Peter Zhu <[email protected]> --------- Signed-off-by: Balasubramanian <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <[email protected]> * Adds debug logs for KNNQuery and KNNWeight (#2466) (#2470) * Adds debug logs for KNNQuery and KNNWeight Signed-off-by: Tejas Shah <[email protected]> * Adds check to see if log is enabled to start and stop StopWatch Signed-off-by: Tejas Shah <[email protected]> * Addressing comments on the PR Signed-off-by: Tejas Shah <[email protected]> * Adds shard and segment info in the logs Signed-off-by: Tejas Shah <[email protected]> * Removes unnecessary segment name param from exact search Signed-off-by: Tejas Shah <[email protected]> * Fixes the build Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Tejas Shah <[email protected]> (cherry picked from commit f322e27) Co-authored-by: Tejas Shah <[email protected]> * Clean JNI artifacts with ./gradlew clean (#2516) * clean JNI artifacts with ./gradlew clean Signed-off-by: Samuel Herman <[email protected]> * nest release under build directory Signed-off-by: Samuel Herman <[email protected]> * adjust all references to the old release path Signed-off-by: Samuel Herman <[email protected]> * remove irrelevant paths from gitignore, add jni/build Signed-off-by: Samuel Herman <[email protected]> * fix logging on linux Signed-off-by: Samuel Herman <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]> * Backport to main (#2520) * Remove skip building graph check for quantization use case (#2430) For quantization indices, we don't have to apply building graph check since it is already faster, this is now only applied for fp32/16 indices and where threshold is configured. Signed-off-by: Vijayan Balasubramanian <[email protected]> * Update default to 0 to always build graph as default behavior (#2452) Signed-off-by: Balasubramanian <[email protected]> * Update changelog Signed-off-by: Balasubramanian <[email protected]> --------- Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Balasubramanian <[email protected]> * [Backport main] Add release notes for 2.19.0 (#2503) * Add release notes for 2.19.0 Signed-off-by: Kunal Kotwani <[email protected]> * Fix links for release notes Co-authored-by: John Mazanec <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> --------- Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: John Mazanec <[email protected]> * Fix main knnlib dir in build script based on #2442 (#2526) Signed-off-by: Peter Zhu <[email protected]> * Minor performance improvments in KNNQueryBuilder (#2528) Signed-off-by: Tejas Shah <[email protected]> * Initial implementation of control flow Signed-off-by: Rohan Chitale <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]> Signed-off-by: Balasubramanian <[email protected]> Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Peter Zhu <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Rohan Chitale <[email protected]> Co-authored-by: sam-herman <[email protected]> Co-authored-by: Vijayan Balasubramanian <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]> Co-authored-by: John Mazanec <[email protected]>
opensearch-project · Feb 17, 2025 · 91d66eb · 91d66eb
1 parent 55652ca
commit 91d66eb
Show file tree

Hide file tree

Showing 72 changed files with 1,196 additions and 158 deletions.
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
@@ -40,6 +40,11 @@ jobs:
       matrix:
         java: [21, 23]
 
+    env:
+       CC: gcc10-gcc
+       CXX: gcc10-g++
+       FC: gcc10-gfortran
+
     name: Build and Test k-NN Plugin on Linux
     runs-on: ubuntu-latest
     needs: Get-CI-Image-Tag

diff --git a/.github/workflows/test_security.yml b/.github/workflows/test_security.yml
@@ -39,6 +39,10 @@ jobs:
     strategy:
       matrix:
         java: [21]
+    env:
+       CC: gcc10-gcc
+       CXX: gcc10-g++
+       FC: gcc10-gfortran
 
     name: Run Integration Tests on Linux
     runs-on: ubuntu-latest

diff --git a/.gitignore b/.gitignore
@@ -14,26 +14,11 @@ out/
 oss/*
 *.iml
 
-jni/CMakeCache.txt
-jni/CMakeFiles
-jni/Makefile
-jni/cmake_install.cmake
-jni/release
-jni/packages
-jni/CTestTestfile.cmake
-jni/KNNPlugin_JNI.cbp
 jni/Testing/
-jni/_deps/
 jni/bin/
-jni/lib/
-jni/jni_test*
-jni/googletest*
+jni/build/
 jni/cmake/*.cmake-e
-jni/.cmake
 jni/.idea
-jni/build.ninja
-jni/.ninja_deps
-jni/.ninja_log
 
 benchmarks/perf-tool/okpt/output
 benchmarks/perf-tool/okpt/dev

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -13,50 +13,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 * Upgrade min JDK compatibility to JDK 21 [#2422](https://github.com/opensearch-project/k-NN/pull/2422)
 ### Documentation
 ### Maintenance
+* Update package name to fix compilation issue [#2513](https://github.com/opensearch-project/k-NN/pull/2513)
 ### Refactoring
 
-## [Unreleased 2.x](https://github.com/opensearch-project/k-NN/compare/2.18...2.x)
+## [Unreleased 2.x](https://github.com/opensearch-project/k-NN/compare/2.19...2.x)
 ### Features
-- Add Support for Multi Values in innerHit for Nested k-NN Fields in Lucene and FAISS (#2283)[https://github.com/opensearch-project/k-NN/pull/2283]
-- Add binary index support for Lucene engine. (#2292)[https://github.com/opensearch-project/k-NN/pull/2292]
-- Add expand_nested_docs Parameter support to NMSLIB engine (#2331)[https://github.com/opensearch-project/k-NN/pull/2331]
-- Add a new build mode, `FAISS_OPT_LEVEL=avx512_spr`, which enables the use of advanced AVX-512 instructions introduced with Intel(R) Sapphire Rapids (#2404)[https://github.com/opensearch-project/k-NN/pull/2404]
-- Add cosine similarity support for faiss engine (#2376)[https://github.com/opensearch-project/k-NN/pull/2376]
-- Add derived source feature for vector fields (#2449)[https://github.com/opensearch-project/k-NN/pull/2449]
 ### Enhancements
-- Introduced a writing layer in native engines where relies on the writing interface to process IO. (#2241)[https://github.com/opensearch-project/k-NN/pull/2241]
-- Allow method parameter override for training based indices (#2290) https://github.com/opensearch-project/k-NN/pull/2290]
-- Optimizes lucene query execution to prevent unnecessary rewrites (#2305)[https://github.com/opensearch-project/k-NN/pull/2305]
-- Add check to directly use ANN Search when filters match all docs. (#2320)[https://github.com/opensearch-project/k-NN/pull/2320]
-- Use one formula to calculate cosine similarity (#2357)[https://github.com/opensearch-project/k-NN/pull/2357]
-- Add WithFieldName implementation to KNNQueryBuilder (#2398)[https://github.com/opensearch-project/k-NN/pull/2398]
-- Make the build work for M series MacOS without manual code changes and local JAVA_HOME config (#2397)[https://github.com/opensearch-project/k-NN/pull/2397]
-- Enabled concurrent graph creation for Lucene engine with index thread qty settings(#2480)[https://github.com/opensearch-project/k-NN/pull/2480]
-- Remove DocsWithFieldSet reference from NativeEngineFieldVectorsWriter (#2408)[https://github.com/opensearch-project/k-NN/pull/2408]
 ### Bug Fixes
-* Fixing the bug when a segment has no vector field present for disk based vector search (#2282)[https://github.com/opensearch-project/k-NN/pull/2282]
-* Fixing the bug where search fails with "fields" parameter for an index with a knn_vector field (#2314)[https://github.com/opensearch-project/k-NN/pull/2314]
-* Fix for NPE while merging segments after all the vector fields docs are deleted (#2365)[https://github.com/opensearch-project/k-NN/pull/2365]
-* Allow validation for non knn index only after 2.17.0 (#2315)[https://github.com/opensearch-project/k-NN/pull/2315]
-* Fixing the bug to prevent updating the index.knn setting after index creation(#2348)[https://github.com/opensearch-project/k-NN/pull/2348]
-* Release query vector memory after execution (#2346)[https://github.com/opensearch-project/k-NN/pull/2346]
-* Fix shard level rescoring disabled setting flag (#2352)[https://github.com/opensearch-project/k-NN/pull/2352]
-* Fix filter rewrite logic which was resulting in getting inconsistent / incorrect results for cases where filter was getting rewritten for shards (#2359)[https://github.com/opensearch-project/k-NN/pull/2359]
-* Fixing it to retrieve space_type from index setting when both method and top level don't have the value. [#2374](https://github.com/opensearch-project/k-NN/pull/2374)
-* Fixing the bug where setting rescore as false for on_disk knn_vector query is a no-op (#2399)[https://github.com/opensearch-project/k-NN/pull/2399]
-* Fixing bug where mapping accepts both dimension and model-id (#2410)[https://github.com/opensearch-project/k-NN/pull/2410]
-* Add version check for full field name validation (#2477)[https://github.com/opensearch-project/k-NN/pull/2477]
 ### Infrastructure
-* Updated C++ version in JNI from c++11 to c++17 [#2259](https://github.com/opensearch-project/k-NN/pull/2259)
-* Upgrade bytebuddy and objenesis version to match OpenSearch core and, update github ci runner for macos [#2279](https://github.com/opensearch-project/k-NN/pull/2279)
 ### Documentation
 ### Maintenance
-* Select index settings based on cluster version[2236](https://github.com/opensearch-project/k-NN/pull/2236)
-* Added periodic cache maintenance for QuantizationStateCache and NativeMemoryCache [#2308](https://github.com/opensearch-project/k-NN/pull/2308)
-* Added null checks for fieldInfo in ExactSearcher to avoid NPE while running exact search for segments with no vector field (#2278)[https://github.com/opensearch-project/k-NN/pull/2278]
-* Added Lucene BWC tests (#2313)[https://github.com/opensearch-project/k-NN/pull/2313]
-* Upgrade jsonpath from 2.8.0 to 2.9.0[2325](https://github.com/opensearch-project/k-NN/pull/2325)
-* Bump Faiss commit from 1f42e81 to 0cbc2a8 to accelerate hamming distance calculation using _mm512_popcnt_epi64  intrinsic and also add avx512-fp16 instructions to boost performance [#2381](https://github.com/opensearch-project/k-NN/pull/2381)
 * Enabled indices.breaker.total.use_real_memory setting via build.gradle for integTest Cluster to catch heap CB in local ITs and github CI actions [#2395](https://github.com/opensearch-project/k-NN/pull/2395/) 
 * Fixing Lucene912Codec Issue with BWC for Lucene 10.0.1 upgrade[#2429](https://github.com/opensearch-project/k-NN/pull/2429)
+* Enabled idempotency of local builds when using `./gradlew clean` and nest `jni/release` directory under `jni/build` for easier cleanup [#2516](https://github.com/opensearch-project/k-NN/pull/2516)
 ### Refactoring
diff --git a/build.gradle b/build.gradle
@@ -329,11 +329,52 @@ task windowsPatches(type:Exec) {
     commandLine 'cmd', '/c', "Powershell -File $rootDir\\scripts\\windowsScript.ps1"
 }
 
-task cmakeJniLib(type:Exec) {
-    workingDir 'jni'
+def findExecutable(String executableName, List<String> additionalPaths = []) {
+    // Print the task's environment before setting it
+    // Print PATH specifically
+    logger.lifecycle("\nSystem PATH:")
+    logger.lifecycle(System.getenv("PATH"))
+
+    def commonBasePaths = [
+            "/opt/homebrew/bin",
+            "/usr/local/bin",
+            "/usr/bin"
+    ]
+
+    // Start with just the executable name (will use system PATH)
+    def execPath = executableName
+
+    if (Os.isFamily(Os.FAMILY_MAC)) {
+        def searchPaths = []
+        // Add common paths
+        commonBasePaths.each { basePath ->
+            searchPaths.add("${basePath}/${executableName}")
+        }
+        // Add any additional specific paths
+        searchPaths.addAll(additionalPaths)
+
+        for (path in searchPaths) {
+            if (new File(path).exists()) {
+                logger.lifecycle("Found ${executableName} at: ${path}")
+                execPath = path
+                break
+            }
+        }
+    }
+
+    return execPath
+}
+
+tasks.register('cmakeJniLib', Exec) {
+    def cmakePath = findExecutable("cmake")
+    logger.lifecycle("Using cmake at: ${cmakePath}")
+    // Inherit the current environment
+    environment System.getenv()
+
     def args = []
-    args.add("cmake")
-    args.add(".")
+    args.add(cmakePath)
+    args.add("-S jni") // CMakelists.txt directory
+    args.add("-B jni/build") // Build directory
     args.add("-DKNN_PLUGIN_VERSION=${opensearch_version}")
     args.add("-DAVX2_ENABLED=${avx2_enabled}")
     args.add("-DAVX512_ENABLED=${avx512_enabled}")
@@ -343,7 +384,7 @@ task cmakeJniLib(type:Exec) {
     def javaHome = Jvm.current().getJavaHome()
     logger.lifecycle("Java home directory used by gradle: $javaHome")
     if (Os.isFamily(Os.FAMILY_MAC)) {
-        environment('JAVA_HOME',javaHome)
+        environment('JAVA_HOME', javaHome)
     }
     if (Os.isFamily(Os.FAMILY_WINDOWS)) {
         dependsOn windowsPatches
@@ -353,35 +394,75 @@ task cmakeJniLib(type:Exec) {
         args.add("-DLAPACK_LIBRARIES=$rootDir\\src\\main\\resources\\windowsDependencies\\libopenblas.dll")
     }
 
+    // Print the task's environment after setting it
+    logger.lifecycle("\nTask Environment PATH:")
+    logger.lifecycle(environment.get('PATH'))
+
+    // Print the command that will be executed
+    logger.lifecycle("CMake command: ${args.join(' ')}")
+    def outputStream = new ByteArrayOutputStream()
     commandLine args
+    standardOutput = outputStream
+}
+
+// Makes sure that `./gradlew clean` removes all JNI build artifacts
+tasks.clean.doFirst {
+    // Delete JNI build directory
+    delete "${projectDir}/jni/build"
+}
+
+task buildNmslib(type:Exec) {
+    dependsOn cmakeJniLib
+    def cmakePath = findExecutable("cmake")
+    logger.lifecycle("Using cmake at: ${cmakePath}")
+
+    if (cmakePath.isEmpty()) {
+        throw new GradleException("CMake not found in PATH. Please install CMake.")
+    }
+
+    commandLine cmakePath,
+            '--build', 'jni/build',
+            '--target', 'opensearchknn_nmslib',
+            '--parallel', "${nproc_count}"
 }
 
 task buildJniLib(type:Exec) {
     dependsOn cmakeJniLib
-    workingDir 'jni'
-    commandLine 'make', 'opensearchknn_nmslib', 'opensearchknn_faiss', 'opensearchknn_common', '-j', "${nproc_count}"
+    def cmakePath = findExecutable("cmake")
+    logger.lifecycle("Using cmake at: ${cmakePath}")
+
+    if (cmakePath.isEmpty()) {
+        throw new GradleException("CMake not found in PATH. Please install CMake.")
+    }
+
+    commandLine cmakePath,
+            '--build', 'jni/build',
+            '--target', 'opensearchknn_faiss', 'opensearchknn_common',
+            '--parallel', "${nproc_count}"
 }
 
 test {
+    dependsOn buildNmslib
     dependsOn buildJniLib
     systemProperty 'tests.security.manager', 'false'
-    systemProperty "java.library.path", "$rootDir/jni/release"
+    systemProperty "java.library.path", "$rootDir/jni/build/release"
     //this change enables mockito-inline that supports mocking of static classes/calls
     systemProperty "jdk.attach.allowAttachSelf", true
     if (Os.isFamily(Os.FAMILY_WINDOWS)) {
         // Add the paths of built JNI libraries and its dependent libraries to PATH variable in System variables
-        environment('PATH', System.getenv('PATH') + ";$rootDir/jni/release" + ";$rootDir/src/main/resources/windowsDependencies")
+        environment('PATH', System.getenv('PATH') + ";$rootDir/jni/build/release" + ";$rootDir/src/main/resources/windowsDependencies")
     }
 }
 
 def _numNodes = findProperty('numNodes') as Integer ?: 1
 integTest {
     if (integTestDependOnJniLib) {
+        dependsOn buildNmslib
         dependsOn buildJniLib
     }
     systemProperty 'tests.security.manager', 'false'
     systemProperty 'java.io.tmpdir', opensearch_tmp_dir.absolutePath
-    systemProperty "java.library.path", "$rootDir/jni/release"
+    systemProperty "java.library.path", "$rootDir/jni/build/release"
     // allows integration test classes to access test resource from project root path
     systemProperty('project.root', project.rootDir.absolutePath)
 
@@ -433,7 +514,7 @@ testClusters.integTest {
     plugin(project.tasks.bundlePlugin.archiveFile)
     if (Os.isFamily(Os.FAMILY_WINDOWS)) {
         // Add the paths of built JNI libraries and its dependent libraries to PATH variable in System variables
-        environment('PATH', System.getenv('PATH') + ";$rootDir/jni/release" + ";$rootDir/src/main/resources/windowsDependencies")
+        environment('PATH', System.getenv('PATH') + ";$rootDir/jni/build/release" + ";$rootDir/src/main/resources/windowsDependencies")
     }
 
     // Cluster shrink exception thrown if we try to set numberOfNodes to 1, so only apply if > 1
@@ -448,7 +529,7 @@ testClusters.integTest {
             debugPort += 1
         }
     }
-    systemProperty("java.library.path", "$rootDir/jni/release")
+    systemProperty("java.library.path", "$rootDir/jni/build/release")
     systemProperty propertyKeys.breaker.useRealMemory, getBreakerSetting()
 }
 
@@ -475,6 +556,7 @@ task integTestRemote(type: RestIntegTestTask) {
 
 run {
     useCluster project.testClusters.integTest
+    dependsOn buildNmslib
     dependsOn buildJniLib
     doFirst {
         // There seems to be an issue when running multi node run or integ tasks with unicast_hosts

diff --git a/jni/cmake/init-faiss.cmake b/jni/cmake/init-faiss.cmake
@@ -111,10 +111,10 @@ endif()
 if(NOT DEFINED AVX512_SPR_ENABLED)
     # Check if the system is Intel(R) Sapphire Rapids or a newer-generation processor
     execute_process(COMMAND bash -c "lscpu | grep -q 'GenuineIntel' && lscpu | grep -i 'avx512_fp16' | grep -i 'avx512_bf16' | grep -i 'avx512_vpopcntdq'" OUTPUT_VARIABLE SPR_FLAGS OUTPUT_STRIP_TRAILING_WHITESPACE)
-    if (AND NOT "${SPR_FLAGS}" STREQUAL "")
-	set(AVX512_SPR_ENABLED true)
+    if (NOT "${SPR_FLAGS}" STREQUAL "")
+	      set(AVX512_SPR_ENABLED true)
     else()
-	set(AVX512_SPR_ENABLED false)
+	      set(AVX512_SPR_ENABLED false)
     endif()
 endif()
 

diff --git a/jni/cmake/macros.cmake b/jni/cmake/macros.cmake
@@ -9,8 +9,8 @@ macro(opensearch_set_common_properties TARGET)
 
     if (NOT "${WIN32}" STREQUAL "")
         # Use RUNTIME_OUTPUT_DIRECTORY, to build the target library in the specified directory at runtime.
-        set_target_properties(${TARGET} PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/release)
+        set_target_properties(${TARGET} PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/release)
     else()
-        set_target_properties(${TARGET} PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/release)
+        set_target_properties(${TARGET} PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/release)
     endif()
 endmacro()
diff --git a/jni/external/faiss b/jni/external/faiss
diff --git a/jni/external/nmslib b/jni/external/nmslib
diff --git a/qa/restart-upgrade/build.gradle b/qa/restart-upgrade/build.gradle
@@ -26,7 +26,7 @@ testClusters {
         environment "LD_LIBRARY_PATH", "${buildDir}/testclusters/${baseName}-0/distro/${knn_bwc_version_no_qualifier}-ARCHIVE/plugins/opensearch-knn/knnlib;${buildDir}/testclusters/${baseName}-0/distro/${knn_bwc_version_no_qualifier}-ARCHIVE/plugins/opensearch-knn/lib"
         if (Os.isFamily(Os.FAMILY_WINDOWS)) {
             // While running on Windows OS, setting the PATH environment variable to include the paths to dlls of JNI libraries and windows dependencies
-            environment('PATH', System.getenv('PATH') + ";$rootDir/jni/release" + ";$rootDir/src/main/resources/windowsDependencies")
+            environment('PATH', System.getenv('PATH') + ";$rootDir/jni/build/release" + ";$rootDir/src/main/resources/windowsDependencies")
             systemProperty "java.library.path", "${buildDir}/testclusters/${baseName}-0/distro/${knn_bwc_version_no_qualifier}-ARCHIVE/plugins/opensearch-knn/knnlib;${buildDir}/testclusters/${baseName}-0/distro/${knn_bwc_version_no_qualifier}-ARCHIVE/plugins/opensearch-knn/lib"
         } else {
             systemProperty "java.library.path", "${buildDir}/testclusters/${baseName}-0/distro/${knn_bwc_version_no_qualifier}-ARCHIVE/plugins/opensearch-knn/knnlib:${buildDir}/testclusters/${baseName}-0/distro/${knn_bwc_version_no_qualifier}-ARCHIVE/plugins/opensearch-knn/lib"
@@ -135,12 +135,13 @@ testClusters {
 // All nodes are upgraded to latest version and run the tests
     task testRestartUpgrade(type: StandaloneRestIntegTestTask) {
         dependsOn "testAgainstOldCluster"
+        dependsOn rootProject.tasks.buildNmslib
         dependsOn rootProject.tasks.buildJniLib
         dependsOn rootProject.tasks.assemble
         useCluster testClusters."${baseName}"
         doFirst {
-            testClusters."${baseName}".environment("LD_LIBRARY_PATH", "$rootDir/jni/release")
-            testClusters."${baseName}".systemProperty("java.library.path", "$rootDir/jni/release")
+            testClusters."${baseName}".environment("LD_LIBRARY_PATH", "$rootDir/jni/build/release")
+            testClusters."${baseName}".systemProperty("java.library.path", "$rootDir/jni/build/release")
             testClusters."${baseName}".upgradeAllNodesAndPluginsToNextVersion([rootProject.tasks.bundlePlugin.archiveFile])
         }
         systemProperty 'tests.rest.bwcsuite_cluster', 'upgraded_cluster'