Optionally allow text similarity reranking to fail #121784

thecoop · 2025-02-05T16:00:18Z

First part of #116796

Add a allow_reranker_failures option to reranker configuration, allowing errors to be ignored and the search to carry on. The option will be documented as part of the second stage PR

elasticsearchmachine · 2025-02-05T16:01:05Z

Hi @thecoop, I've created a changelog YAML for you.

thecoop · 2025-02-05T16:03:09Z

server/src/main/java/org/elasticsearch/action/search/RankFeaturePhase.java

@@ -181,6 +182,11 @@ private void onPhaseDone(
        RankFeaturePhaseRankCoordinatorContext rankFeaturePhaseRankCoordinatorContext,
        SearchPhaseController.ReducedQueryPhase reducedQueryPhase
    ) {
+        RankFeatureDoc[] docs = rankPhaseResults.getSuccessfulResults()


This changes the thread this happens in - not sure if this matters or not?

...rg/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankRetrieverBuilder.java

elasticsearchmachine · 2025-02-06T16:19:29Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

...test/java/org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankTests.java

elasticsearchmachine · 2025-02-07T11:04:39Z

Hi @thecoop, I've updated the changelog YAML for you.

pmpailis

Thanks @thecoop , really nice :) Left some minor comments, with the error handling/logging pending, as well as how to propagate to the user that the results are not reranked (prob something in CompoundRetrieverBuilder ? )

pmpailis · 2025-02-10T10:30:42Z

...in/java/org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankBuilder.java

@@ -51,35 +43,26 @@ public class TextSimilarityRankBuilder extends RankBuilder {
        License.OperationMode.ENTERPRISE
    );

-    static final ConstructingObjectParser<TextSimilarityRankBuilder, Void> PARSER = new ConstructingObjectParser<>(NAME, args -> {


+1 on the cleanup :)

...test/java/org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankTests.java

.../org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankMultiNodeTests.java

...pack/inference/rank/textsimilarity/TextSimilarityRankFeaturePhaseRankCoordinatorContext.java

.../main/java/org/elasticsearch/search/rank/context/RankFeaturePhaseRankCoordinatorContext.java

pmpailis · 2025-02-10T11:21:38Z

server/src/main/java/org/elasticsearch/action/search/RankFeaturePhase.java

-                    context.onPhaseFailure(NAME, "Computing updated ranks for results failed", e);
+                    if (rankFeaturePhaseRankCoordinatorContext.failuresAllowed()) {
+                        // TODO: handle the exception somewhere
+                        // don't want to log the entire stack trace, it's not helpful here


...pack/inference/rank/textsimilarity/TextSimilarityRankFeaturePhaseRankCoordinatorContext.java

...estTest/resources/rest-api-spec/test/rrf/800_rrf_with_text_similarity_reranker_retriever.yml

pmpailis · 2025-02-12T13:08:43Z

Thanks for all the work and the iterations so far @thecoop ! Will documentation and propagating the non-reranked flag to the user be part of a second PR or will you append them on this one?

thecoop · 2025-02-12T13:22:15Z

Yes, they'll be in a second PR to add the user-visible sections

pmpailis

Thanks @thecoop ! Really nice ❤️ Left a couple of minor comments but no need for another review round :)

...pack/inference/rank/textsimilarity/TextSimilarityRankFeaturePhaseRankCoordinatorContext.java

...estTest/resources/rest-api-spec/test/rrf/800_rrf_with_text_similarity_reranker_retriever.yml

Add a lenient option to text similarity reranking

ad91d47

thecoop added >enhancement :Search Relevance/Search Catch all for Search Relevance labels Feb 5, 2025

thecoop requested a review from pmpailis February 5, 2025 16:00

elasticsearchmachine added the v9.1.0 label Feb 5, 2025

Update docs/changelog/121784.yaml

b447b33

thecoop commented Feb 5, 2025

View reviewed changes

thecoop added :Search Relevance/Ranking Scoring, rescoring, rank evaluation. and removed :Search Relevance/Search Catch all for Search Relevance labels Feb 5, 2025

propagate

1ac18fe

thecoop force-pushed the lenient-rerankers branch from 20c5735 to 1ac18fe Compare February 6, 2025 09:37

thecoop commented Feb 6, 2025

View reviewed changes

...rg/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankRetrieverBuilder.java Outdated Show resolved Hide resolved

thecoop added 2 commits February 6, 2025 10:58

leniency is only part of the text similarity builders

22bd2a0

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

505ae8b

thecoop force-pushed the lenient-rerankers branch from 65aac03 to 505ae8b Compare February 6, 2025 10:59

Remove superfluous xcontent serialization

c47b8a5

thecoop marked this pull request as ready for review February 6, 2025 16:19

elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Feb 6, 2025

Add a test for lenient rerankers

bc81bd0

thecoop force-pushed the lenient-rerankers branch from f43701f to bc81bd0 Compare February 6, 2025 16:25

thecoop commented Feb 6, 2025

View reviewed changes

...test/java/org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankTests.java Outdated Show resolved Hide resolved

Rename lenient to failuresAllowed

2ef4e4e

thecoop changed the title ~~Add a lenient option to text similarity reranking~~ Optionally allow text similarity reranking to fail Feb 7, 2025

thecoop requested a review from benwtrent February 7, 2025 08:38

thecoop added 4 commits February 7, 2025 08:40

Use correct exception

9bb1d20

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

781e736

Update xcontent field name

fd396d9

Add another test

3ee20ed

elasticsearchmachine and others added 3 commits February 7, 2025 10:30

[CI] Auto commit changes from spotless

d93a8dc

Some more test tweaks

551f640

Update docs/changelog/121784.yaml

df8b1ac

pmpailis reviewed Feb 10, 2025

View reviewed changes

thecoop added 3 commits February 10, 2025 12:19

Actually check scores are passed through

108b89c

Remove clone

6cde906

Merge branch 'main' into lenient-rerankers

36a16e1

thecoop requested a review from pmpailis February 10, 2025 13:02

thecoop added 5 commits February 10, 2025 13:25

Fix test checks

bfbcca3

Set scores where there are none

f4f13c9

Stray comment

559aa32

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

cbd13b9

splotless

7b81006

thecoop force-pushed the lenient-rerankers branch from 339acc2 to 7b81006 Compare February 10, 2025 16:55

pmpailis reviewed Feb 11, 2025

View reviewed changes

...estTest/resources/rest-api-spec/test/rrf/800_rrf_with_text_similarity_reranker_retriever.yml Outdated Show resolved Hide resolved

thecoop added 2 commits February 12, 2025 09:29

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

3112cec

Change to constant_score for determinism

c3c08f6

pmpailis approved these changes Feb 13, 2025

View reviewed changes

...pack/inference/rank/textsimilarity/TextSimilarityRankFeaturePhaseRankCoordinatorContext.java Outdated Show resolved Hide resolved

...estTest/resources/rest-api-spec/test/rrf/800_rrf_with_text_similarity_reranker_retriever.yml Show resolved Hide resolved

thecoop added 4 commits February 17, 2025 10:32

Maintain existing sort order on failure

b0670a2

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

773d610

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

7fa15e6

Fix transport version

01654d5

thecoop merged commit fc2f8fc into elastic:main Feb 17, 2025
17 checks passed

thecoop deleted the lenient-rerankers branch February 17, 2025 13:32

thecoop mentioned this pull request Feb 17, 2025

Add phase failures to search responses #122788

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optionally allow text similarity reranking to fail #121784

Optionally allow text similarity reranking to fail #121784

thecoop commented Feb 5, 2025 •

edited

Loading

elasticsearchmachine commented Feb 5, 2025

thecoop Feb 5, 2025

elasticsearchmachine commented Feb 6, 2025

elasticsearchmachine commented Feb 7, 2025

pmpailis left a comment

pmpailis Feb 10, 2025

pmpailis Feb 10, 2025

pmpailis commented Feb 12, 2025

thecoop commented Feb 12, 2025

pmpailis left a comment

Optionally allow text similarity reranking to fail #121784

Optionally allow text similarity reranking to fail #121784

Conversation

thecoop commented Feb 5, 2025 • edited Loading

elasticsearchmachine commented Feb 5, 2025

thecoop Feb 5, 2025

Choose a reason for hiding this comment

elasticsearchmachine commented Feb 6, 2025

elasticsearchmachine commented Feb 7, 2025

pmpailis left a comment

Choose a reason for hiding this comment

pmpailis Feb 10, 2025

Choose a reason for hiding this comment

pmpailis Feb 10, 2025

Choose a reason for hiding this comment

pmpailis commented Feb 12, 2025

thecoop commented Feb 12, 2025

pmpailis left a comment

Choose a reason for hiding this comment

thecoop commented Feb 5, 2025 •

edited

Loading