Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions #2506

peterzhuamazon · 2025-02-07T02:12:00Z

Description

Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions

Related Issues

Check List

~~[ ] New functionality includes testing.~~
~~[ ] New functionality has been documented.~~
~~[ ] API changes companion pull request created.~~
Commits are signed per the DCO using --signoff.
~~[ ] Public documentation issue/PR created.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peter Zhu <[email protected]>

build.gradle

Signed-off-by: Peter Zhu <[email protected]>

build.gradle

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon · 2025-02-07T03:03:26Z

Pending https://build.ci.opensearch.org/job/docker-build/6491/console

peterzhuamazon · 2025-02-07T05:35:08Z

Re-trigger as the image is online now:
https://hub.docker.com/r/opensearchstaging/ci-runner/tags?page_size=&ordering=last_updated&name=ci-runner-al2-opensearch-build-v1.8

…ith different gcc versions (#2506) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1)

…ith different gcc versions (#2506) (#2508) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1) Co-authored-by: Peter Zhu <[email protected]>

peterzhuamazon · 2025-02-10T20:40:26Z

Based on the information that nmslib will be deprecated in 3.x, we do not need to port this to main.

Thanks.

navneet1v · 2025-02-10T20:45:25Z

Based on the information that nmslib will be deprecated in 3.x, we do not need to port this to main.

Thanks.

@peterzhuamazon just to provide some clarity we are deprecating nmslib in 3.0 but the support will still be there. Hence we should backport this change to 3.x too

…ith different gcc versions (#2506) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1)

…ith different gcc versions (opensearch-project#2506) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]>

…ith different gcc versions (opensearch-project#2506) (opensearch-project#2508) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1) Co-authored-by: Peter Zhu <[email protected]>

* Update package name to fix compilation issue Core renamed this package in opensearch-project/OpenSearch#17272 This commit renames package accordingly Signed-off-by: Balasubramanian <[email protected]> * Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions (#2506) (#2508) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1) Co-authored-by: Peter Zhu <[email protected]> --------- Signed-off-by: Balasubramanian <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <[email protected]>

* Update package name to fix compilation issue Core renamed this package in opensearch-project/OpenSearch#17272 This commit renames package accordingly Signed-off-by: Balasubramanian <[email protected]> * Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions (opensearch-project#2506) (opensearch-project#2508) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1) Co-authored-by: Peter Zhu <[email protected]> --------- Signed-off-by: Balasubramanian <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <[email protected]>

…vice (#2532) * fix broken build flag, move build to one directory (#2442) * move build to one directory, fix broken flag Signed-off-by: Samuel Herman <[email protected]> * fix make path Signed-off-by: Samuel Herman <[email protected]> * changelog update Signed-off-by: Samuel Herman <[email protected]> * add fix for classpath change and for cmake discovery on macos Signed-off-by: Samuel Herman <[email protected]> * fix make discovery for gradle Signed-off-by: Samuel Herman <[email protected]> * fix cmake path for macOS Signed-off-by: Samuel Herman <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]> * Update package name to fix compilation issue (#2513) * Update package name to fix compilation issue Core renamed this package in opensearch-project/OpenSearch#17272 This commit renames package accordingly Signed-off-by: Balasubramanian <[email protected]> * Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions (#2506) (#2508) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1) Co-authored-by: Peter Zhu <[email protected]> --------- Signed-off-by: Balasubramanian <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <[email protected]> * Adds debug logs for KNNQuery and KNNWeight (#2466) (#2470) * Adds debug logs for KNNQuery and KNNWeight Signed-off-by: Tejas Shah <[email protected]> * Adds check to see if log is enabled to start and stop StopWatch Signed-off-by: Tejas Shah <[email protected]> * Addressing comments on the PR Signed-off-by: Tejas Shah <[email protected]> * Adds shard and segment info in the logs Signed-off-by: Tejas Shah <[email protected]> * Removes unnecessary segment name param from exact search Signed-off-by: Tejas Shah <[email protected]> * Fixes the build Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Tejas Shah <[email protected]> (cherry picked from commit f322e27) Co-authored-by: Tejas Shah <[email protected]> * Clean JNI artifacts with ./gradlew clean (#2516) * clean JNI artifacts with ./gradlew clean Signed-off-by: Samuel Herman <[email protected]> * nest release under build directory Signed-off-by: Samuel Herman <[email protected]> * adjust all references to the old release path Signed-off-by: Samuel Herman <[email protected]> * remove irrelevant paths from gitignore, add jni/build Signed-off-by: Samuel Herman <[email protected]> * fix logging on linux Signed-off-by: Samuel Herman <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]> * Backport to main (#2520) * Remove skip building graph check for quantization use case (#2430) For quantization indices, we don't have to apply building graph check since it is already faster, this is now only applied for fp32/16 indices and where threshold is configured. Signed-off-by: Vijayan Balasubramanian <[email protected]> * Update default to 0 to always build graph as default behavior (#2452) Signed-off-by: Balasubramanian <[email protected]> * Update changelog Signed-off-by: Balasubramanian <[email protected]> --------- Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Balasubramanian <[email protected]> * [Backport main] Add release notes for 2.19.0 (#2503) * Add release notes for 2.19.0 Signed-off-by: Kunal Kotwani <[email protected]> * Fix links for release notes Co-authored-by: John Mazanec <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> --------- Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: John Mazanec <[email protected]> * Fix main knnlib dir in build script based on #2442 (#2526) Signed-off-by: Peter Zhu <[email protected]> * Minor performance improvments in KNNQueryBuilder (#2528) Signed-off-by: Tejas Shah <[email protected]> * Initial implementation of control flow Signed-off-by: Rohan Chitale <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]> Signed-off-by: Balasubramanian <[email protected]> Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Peter Zhu <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Rohan Chitale <[email protected]> Co-authored-by: sam-herman <[email protected]> Co-authored-by: Vijayan Balasubramanian <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]> Co-authored-by: John Mazanec <[email protected]>

peterzhuamazon added 2 commits February 6, 2025 18:09

Update build gradle to separate nmslib / faiss generation

5632969

Signed-off-by: Peter Zhu <[email protected]>

Update scripts/build.sh to separate enable gcc switch

787d49f

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon added the backport 2.x label Feb 7, 2025

peterzhuamazon self-assigned this Feb 7, 2025

peterzhuamazon requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, ryanbogan, luyuncheng, shatejas and 0ctopus13prime as code owners February 7, 2025 02:12

naveentatikonda reviewed Feb 7, 2025

View reviewed changes

build.gradle Outdated Show resolved Hide resolved

Remove test comments

95d221d

Signed-off-by: Peter Zhu <[email protected]>

naveentatikonda reviewed Feb 7, 2025

View reviewed changes

build.gradle Outdated Show resolved Hide resolved

peterzhuamazon added 2 commits February 6, 2025 21:15

Remove test comments

5c60e64

Signed-off-by: Peter Zhu <[email protected]>

Remove test comments

21fd6a6

Signed-off-by: Peter Zhu <[email protected]>

navneet1v added the skip-changelog label Feb 7, 2025

naveentatikonda previously approved these changes Feb 7, 2025

View reviewed changes

Updating restart and rolling upgrade bwc test bundle.gradle

a67f9de

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon dismissed naveentatikonda’s stale review via a67f9de February 7, 2025 02:33

Enforce gcc10 for nmslib to compile and avx512_spr have no-op

ffef498

Signed-off-by: Peter Zhu <[email protected]>

naveentatikonda approved these changes Feb 7, 2025

View reviewed changes

peterzhuamazon linked an issue Feb 7, 2025 that may be closed by this pull request

UnsatisfiedLinkError: libopensearchknn_faiss_avx512_spr.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found #2484

Closed

peterzhuamazon mentioned this pull request Feb 7, 2025

UnsatisfiedLinkError: libopensearchknn_faiss_avx512_spr.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found #2484

Closed

peterzhuamazon merged commit 107c4f1 into opensearch-project:2.19 Feb 7, 2025
99 checks passed

peterzhuamazon deleted the knn-nmslib-separation-2 branch February 7, 2025 06:10

opensearch-trigger-bot bot mentioned this pull request Feb 7, 2025

[Backport 2.x] Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions #2508

Merged

VijayanB added the backport main label Feb 10, 2025

opensearch-trigger-bot bot mentioned this pull request Feb 10, 2025

[Backport main] Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions #2514

Closed

peterzhuamazon mentioned this pull request Feb 10, 2025

[Release 3.0] Planned Breaking Changes for 3.0 in Plugin #2396

Open

VijayanB mentioned this pull request Feb 10, 2025

Update package name to fix compilation issue #2513

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions #2506

Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions #2506

peterzhuamazon commented Feb 7, 2025 •

edited

Loading

peterzhuamazon commented Feb 7, 2025

peterzhuamazon commented Feb 7, 2025

peterzhuamazon commented Feb 10, 2025

navneet1v commented Feb 10, 2025

Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions #2506

Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions #2506

Conversation

peterzhuamazon commented Feb 7, 2025 • edited Loading

Description

Related Issues

Check List

peterzhuamazon commented Feb 7, 2025

peterzhuamazon commented Feb 7, 2025

peterzhuamazon commented Feb 10, 2025

navneet1v commented Feb 10, 2025

peterzhuamazon commented Feb 7, 2025 •

edited

Loading