Why is score different from value in explanation?

[krueger]

Hi,

I did an explain request and get the following response:

                "_score": 2,
                "fields": {
                    "title": [
                        "Title"
                    ],
                    "id": [
                        "vbce77e7t9pe"
                    ]
                },
                "highlight": {
                    "content.search": [
                        "... some content ..."
                    ]
                },
                "_explanation": {
                    "value": 196.23717,
                    "description": "sum of:",
                    "details": [
                        {...
                        },
                        {
                            "value": 4.737492,
                            "description": "product of:",
                            "details": [...
                            ]
                        }
                    ]
                },

As you can see the score value is 2 and the value in the explanation is 196.23717.

Why is this different?

[jpountz]

This is a bug, they should always be the same. Can you share your query?

cc @elastic/es-search-aggs

[krueger]

This is the query

{
  "from" : 0,
  "size" : 67,
  "query" : {
    "function_score" : {
      "query" : {
        "bool" : {
          "must" : [
            {
              "bool" : {
                "should" : [
                  {
                    "nested" : {
                      "query" : {
                        "bool" : {
                          "must" : [
                            {
                              "constant_score" : {
                                "filter" : {
                                  "bool" : {
                                    "must" : [
                                      {
                                        "terms" : {
                                          "elements.aclgroups" : [
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "<some id>",
                                            "DUMMY_GROUP_ID"
                                          ],
                                          "boost" : 1.0
                                        }
                                      },
                                      {
                                        "term" : {
                                          "elements.hidden" : {
                                            "value" : false,
                                            "boost" : 1.0
                                          }
                                        }
                                      },
                                      {
                                        "bool" : {
                                          "should" : [
                                            {
                                              "term" : {
                                                "elements.approval_status" : {
                                                  "value" : "APPROVED",
                                                  "boost" : 1.0
                                                }
                                              }
                                            },
                                            {
                                              "bool" : {
                                                "must_not" : [
                                                  {
                                                    "exists" : {
                                                      "field" : "elements.approval_status",
                                                      "boost" : 1.0
                                                    }
                                                  }
                                                ],
                                                "disable_coord" : false,
                                                "adjust_pure_negative" : true,
                                                "boost" : 1.0
                                              }
                                            }
                                          ],
                                          "disable_coord" : false,
                                          "adjust_pure_negative" : true,
                                          "boost" : 1.0
                                        }
                                      },
                                      {
                                        "bool" : {
                                          "must" : [
                                            {
                                              "range" : {
                                                "elements.valid_from" : {
                                                  "from" : null,
                                                  "to" : "now",
                                                  "include_lower" : true,
                                                  "include_upper" : true,
                                                  "time_zone" : "+01:00",
                                                  "boost" : 1.0
                                                }
                                              }
                                            },
                                            {
                                              "range" : {
                                                "elements.valid_to" : {
                                                  "from" : "now",
                                                  "to" : null,
                                                  "include_lower" : true,
                                                  "include_upper" : true,
                                                  "time_zone" : "+01:00",
                                                  "boost" : 1.0
                                                }
                                              }
                                            }
                                          ],
                                          "disable_coord" : false,
                                          "adjust_pure_negative" : true,
                                          "boost" : 1.0
                                        }
                                      }
                                    ],
                                    "disable_coord" : false,
                                    "adjust_pure_negative" : true,
                                    "boost" : 1.0
                                  }
                                },
                                "boost" : 1.0
                              }
                            },
                            {
                              "bool" : {
                                "must" : [
                                  {
                                    "multi_match" : {
                                      "query" : "<query>",
                                      "fields" : [
                                        "elements.content.search^1.0",
                                        "elements.tags^1.0",
                                        "elements.title.search^1.0"
                                      ],
                                      "type" : "best_fields",
                                      "operator" : "AND",
                                      "slop" : 0,
                                      "prefix_length" : 0,
                                      "max_expansions" : 50,
                                      "lenient" : false,
                                      "cutoff_frequency" : 0.0,
                                      "zero_terms_query" : "NONE",
                                      "boost" : 1.0
                                    }
                                  }
                                ],
                                "disable_coord" : false,
                                "adjust_pure_negative" : true,
                                "boost" : 1.0
                              }
                            }
                          ],
                          "disable_coord" : false,
                          "adjust_pure_negative" : true,
                          "boost" : 1.0
                        }
                      },
                      "path" : "elements",
                      "ignore_unmapped" : true,
                      "score_mode" : "none",
                      "boost" : 1.0,
                      "inner_hits" : {
                        "name" : "elements",
                        "ignore_unmapped" : true,
                        "from" : 0,
                        "size" : 3,
                        "version" : false,
                        "explain" : false,
                        "track_scores" : false,
                        "highlight" : {
                          "pre_tags" : [
                            "<span class=\"highlighted\">"
                          ],
                          "post_tags" : [
                            "</span>"
                          ],
                          "fragment_size" : 80,
                          "number_of_fragments" : 3,
                          "type" : "fvh",
                          "fields" : {
                            "elements.content" : {
                              "type" : "fvh",
                              "matched_fields" : [
                                "elements.content",
                                "elements.content.search"
                              ]
                            }
                          }
                        }
                      }
                    }
                  },
                  {
                    "bool" : {
                      "must" : [
                        {
                          "constant_score" : {
                            "filter" : {
                              "bool" : {
                                "must" : [
                                  {
                                    "bool" : {
                                      "should" : [
                                        {
                                          "term" : {
                                            "aclusers" : {
                                              "value" : "<some id>",
                                              "boost" : 1.0
                                            }
                                          }
                                        },
                                        {
                                          "bool" : {
                                            "must" : [
                                              {
                                                "terms" : {
                                                  "resource" : [
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>",
                                                    "<some resource>"
                                                  ],
                                                  "boost" : 1.0
                                                }
                                              },
                                              {
                                                "terms" : {
                                                  "aclwritegroups" : [
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "<some id>",
                                                    "DUMMY_GROUP_ID"
                                                  ],
                                                  "boost" : 1.0
                                                }
                                              }
                                            ],
                                            "disable_coord" : false,
                                            "adjust_pure_negative" : true,
                                            "boost" : 1.0
                                          }
                                        }
                                      ],
                                      "disable_coord" : false,
                                      "adjust_pure_negative" : true,
                                      "boost" : 1.0
                                    }
                                  }
                                ],
                                "disable_coord" : false,
                                "adjust_pure_negative" : true,
                                "boost" : 1.0
                              }
                            },
                            "boost" : 1.0
                          }
                        },
                        {
                          "nested" : {
                            "query" : {
                              "bool" : {
                                "must" : [
                                  {
                                    "multi_match" : {
                                      "query" : "<query>",
                                      "fields" : [
                                        "elements.content.search^1.0",
                                        "elements.tags^1.0",
                                        "elements.title.search^1.0"
                                      ],
                                      "type" : "best_fields",
                                      "operator" : "AND",
                                      "slop" : 0,
                                      "prefix_length" : 0,
                                      "max_expansions" : 50,
                                      "lenient" : false,
                                      "cutoff_frequency" : 0.0,
                                      "zero_terms_query" : "NONE",
                                      "boost" : 1.0
                                    }
                                  }
                                ],
                                "disable_coord" : false,
                                "adjust_pure_negative" : true,
                                "boost" : 1.0
                              }
                            },
                            "path" : "elements",
                            "ignore_unmapped" : true,
                            "score_mode" : "none",
                            "boost" : 1.0
                          }
                        }
                      ],
                      "disable_coord" : false,
                      "adjust_pure_negative" : true,
                      "boost" : 1.0
                    }
                  }
                ],
                "disable_coord" : false,
                "adjust_pure_negative" : true,
                "boost" : 1.0
              }
            },
            {
              "bool" : {
                "must" : [
                  {
                    "bool" : {
                      "must_not" : [
                        {
                          "terms" : {
                            "resource" : [
                              "<some resource>",
                              "<some resource>",
                              "<some resource>",
                              "<some resource>",
                              "<some resource>",
                              "<some resource>",
                              "<some resource>",
                              "<some resource>",
                              "<some resource>",
                              "<some resource>",
                              "<some resource>"
                            ],
                            "boost" : 1.0
                          }
                        }
                      ],
                      "disable_coord" : false,
                      "adjust_pure_negative" : true,
                      "boost" : 1.0
                    }
                  }
                ],
                "disable_coord" : false,
                "adjust_pure_negative" : true,
                "boost" : 1.0
              }
            }
          ],
          "disable_coord" : false,
          "adjust_pure_negative" : true,
          "boost" : 1.0
        }
      },
      "functions" : [
        {
          "filter" : {
            "match_all" : {
              "boost" : 1.0
            }
          },
          "field_value_factor" : {
            "field" : "boost_factor",
            "factor" : 1.0,
            "modifier" : "none"
          }
        }
      ],
      "score_mode" : "multiply",
      "max_boost" : 3.4028235E38,
      "boost" : 1.0
    }
  },
  "explain" : false,
  "stored_fields" : [
    "id",
    "title"
  ],
  "highlight" : {
    "pre_tags" : [
      "<span class=\"highlighted\">"
    ],
    "post_tags" : [
      "</span>"
    ],
    "type" : "fvh",
    "fields" : {
      "content.search" : {
        "fragment_size" : 80,
        "number_of_fragments" : 3,
        "type" : "fvh",
        "highlight_query" : {
          "bool" : {
            "must" : [
              {
                "multi_match" : {
                  "query" : "<query>",
                  "fields" : [
                    "content.search^1.0"
                  ],
                  "type" : "best_fields",
                  "operator" : "AND",
                  "slop" : 0,
                  "prefix_length" : 0,
                  "max_expansions" : 50,
                  "lenient" : false,
                  "cutoff_frequency" : 0.0,
                  "zero_terms_query" : "NONE",
                  "boost" : 1.0
                }
              }
            ],
            "disable_coord" : false,
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        },
        "matched_fields" : [
          "content.search"
        ]
      }
    }
  },
  "suggest" : {
    "autocomplete" : {
      "text" : "<query>",
      "term" : {
        "field" : "autocomplete",
        "suggest_mode" : "MISSING",
        "accuracy" : 0.5,
        "sort" : "SCORE",
        "string_distance" : "INTERNAL",
        "max_edits" : 2,
        "max_inspections" : 5,
        "max_term_freq" : 0.01,
        "prefix_length" : 1,
        "min_word_length" : 4,
        "min_doc_freq" : 0.0
      }
    }
  },
  "rescore" : [
    {
      "query" : {
        "rescore_query" : {
          "multi_match" : {
            "query" : "<query>",
            "fields" : [
              "elements.content.stem^1.0",
              "elements.tags^1.0",
              "elements.title.stem^1.0"
            ],
            "type" : "phrase",
            "operator" : "OR",
            "slop" : 0,
            "prefix_length" : 0,
            "max_expansions" : 50,
            "lenient" : false,
            "zero_terms_query" : "NONE",
            "boost" : 1.0
          }
        },
        "query_weight" : 1.0,
        "rescore_query_weight" : 80.0,
        "score_mode" : "total"
      }
    },
    {
      "query" : {
        "rescore_query" : {
          "match" : {
            "elements.content.stem" : {
              "query" : "<query>",
              "operator" : "AND",
              "prefix_length" : 0,
              "max_expansions" : 50,
              "fuzzy_transpositions" : true,
              "lenient" : false,
              "zero_terms_query" : "NONE",
              "boost" : 1.0
            }
          }
        },
        "query_weight" : 1.0,
        "rescore_query_weight" : 2.0,
        "score_mode" : "total"
      }
    }
  ]
}

Which value is used for sorting the search results? The value from _score or the value from _explanation? Is the result sorted correctly?

[jpountz]

Which value is used for sorting the search results? The value from _score or the value from _explanation? Is the result sorted correctly?

Results are always sorted based on _score, _explanation is just extra debugging information. I suspect results are still sorted correctly, but it is hard to say for sure without knowing what the exact bug is.

[krueger]

This is the complete explanation part:

"_explanation": {
                    "value": 196.23717,
                    "description": "sum of:",
                    "details": [
                        {
                            "value": 191.49968,
                            "description": "product of:",
                            "details": [
                                {
                                    "value": 191.49968,
                                    "description": "sum of:",
                                    "details": [
                                        {
                                            "value": 2,
                                            "description": "product of:",
                                            "details": [
                                                {
                                                    "value": 2,
                                                    "description": "sum of:",
                                                    "details": [
                                                        {
                                                            "value": 2,
                                                            "description": "function score, product of:",
                                                            "details": [
                                                                {
                                                                    "value": 2,
                                                                    "description": "sum of:",
                                                                    "details": [
                                                                        {
                                                                            "value": 1,
                                                                            "description": "sum of:",
                                                                            "details": [
                                                                                {
                                                                                    "value": 0,
                                                                                    "description": "Score based on 1 child docs in range from 2714 to 2714, best match:",
                                                                                    "details": [
                                                                                        {
                                                                                            "value": 7.6046343,
                                                                                            "description": "sum of:",
                                                                                            "details": [
                                                                                                {
                                                                                                    "value": 7.6046343,
                                                                                                    "description": "sum of:",
                                                                                                    "details": [
                                                                                                        {
                                                                                                            "value": 1,
                                                                                                            "description": "ConstantScore(+ConstantScore(elements.aclgroups:<id>
                                                                                                            elements.aclgroups:<id> elements.aclgroups:<id> elements.aclgroups:<id> elements.aclgroups:<id> elements.aclgroups:<id> elements.aclgroups:<id> elements.aclgroups:<id> elements.aclgroups:<id> elements.aclgroups:<id> elements.aclgroups:<id> elements.aclgroups:DUMMY_GROUP_ID elements.aclgroups:ubguhplma88o) +elements.hidden:F +((elements.approval_status:APPROVED (-ConstantScore(_field_names:elements.approval_status) +*:*))~1) +(+elements.valid_from:[-9223372036854775808 TO 9223372036854775807] +elements.valid_to:[1519050328256 TO 9223372036854775807])), product of:",
                                                                                                            "details": [
                                                                                                                {
                                                                                                                    "value": 1,
                                                                                                                    "description": "boost",
                                                                                                                    "details": []
                                                                                                                },
                                                                                                                {
                                                                                                                    "value": 1,
                                                                                                                    "description": "queryNorm",
                                                                                                                    "details": []
                                                                                                                }
                                                                                                            ]
                                                                                                        },
                                                                                                        {
                                                                                                            "value": 6.6046343,
                                                                                                            "description": "max of:",
                                                                                                            "details": [
                                                                                                                {
                                                                                                                    "value": 6.6046343,
                                                                                                                    "description": "weight(elements.content.search:term in 16) [PerFieldSimilarity], result of:",
                                                                                                                    "details": [
                                                                                                                        {
                                                                                                                            "value": 6.6046343,
                                                                                                                            "description": "score(doc=16,freq=1.0 = termFreq=1.0\n), product of:",
                                                                                                                            "details": [
                                                                                                                                {
                                                                                                                                    "value": 3.9169664,
                                                                                                                                    "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                                                                                                                                    "details": [
                                                                                                                                        {
                                                                                                                                            "value": 56,
                                                                                                                                            "description": "docFreq",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 2838,
                                                                                                                                            "description": "docCount",
                                                                                                                                            "details": []
                                                                                                                                        }
                                                                                                                                    ]
                                                                                                                                },
                                                                                                                                {
                                                                                                                                    "value": 1.6861606,
                                                                                                                                    "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                                                                                                                                    "details": [
                                                                                                                                        {
                                                                                                                                            "value": 1,
                                                                                                                                            "description": "termFreq=1.0",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 1.2,
                                                                                                                                            "description": "parameter k1",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 0.75,
                                                                                                                                            "description": "parameter b",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 1944.5743,
                                                                                                                                            "description": "avgFieldLength",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 10.24,
                                                                                                                                            "description": "fieldLength",
                                                                                                                                            "details": []
                                                                                                                                        }
                                                                                                                                    ]
                                                                                                                                }
                                                                                                                            ]
                                                                                                                        }
                                                                                                                    ]
                                                                                                                }
                                                                                                            ]
                                                                                                        }
                                                                                                    ]
                                                                                                },
                                                                                                {
                                                                                                    "value": 0,
                                                                                                    "description": "match on required clause, product of:",
                                                                                                    "details": [
                                                                                                        {
                                                                                                            "value": 0,
                                                                                                            "description": "# clause",
                                                                                                            "details": []
                                                                                                        },
                                                                                                        {
                                                                                                            "value": 1,
                                                                                                            "description": "_type:__elements, product of:",
                                                                                                            "details": [
                                                                                                                {
                                                                                                                    "value": 1,
                                                                                                                    "description": "boost",
                                                                                                                    "details": []
                                                                                                                },
                                                                                                                {
                                                                                                                    "value": 1,
                                                                                                                    "description": "queryNorm",
                                                                                                                    "details": []
                                                                                                                }
                                                                                                            ]
                                                                                                        }
                                                                                                    ]
                                                                                                }
                                                                                            ]
                                                                                        }
                                                                                    ]
                                                                                },
                                                                                {
                                                                                    "value": 1,
                                                                                    "description": "sum of:",
                                                                                    "details": [
                                                                                        {
                                                                                            "value": 1,
                                                                                            "description": "ConstantScore((aclusers:4374ef463c8cb7f0013c967ea7cc4701 (+resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> resource:<resource> +ConstantScore(aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:<id> aclwritegroups:DUMMY_GROUP_ID aclwritegroups:<id>)))~1), product of:",
                                                                                            "details": [
                                                                                                {
                                                                                                    "value": 1,
                                                                                                    "description": "boost",
                                                                                                    "details": []
                                                                                                },
                                                                                                {
                                                                                                    "value": 1,
                                                                                                    "description": "queryNorm",
                                                                                                    "details": []
                                                                                                }
                                                                                            ]
                                                                                        },
                                                                                        {
                                                                                            "value": 0,
                                                                                            "description": "Score based on 1 child docs in range from 2714 to 2714, best match:",
                                                                                            "details": [
                                                                                                {
                                                                                                    "value": 6.6046343,
                                                                                                    "description": "sum of:",
                                                                                                    "details": [
                                                                                                        {
                                                                                                            "value": 6.6046343,
                                                                                                            "description": "max of:",
                                                                                                            "details": [
                                                                                                                {
                                                                                                                    "value": 6.6046343,
                                                                                                                    "description": "weight(elements.content.search:term in 16) [PerFieldSimilarity], result of:",
                                                                                                                    "details": [
                                                                                                                        {
                                                                                                                            "value": 6.6046343,
                                                                                                                            "description": "score(doc=16,freq=1.0 = termFreq=1.0\n), product of:",
                                                                                                                            "details": [
                                                                                                                                {
                                                                                                                                    "value": 3.9169664,
                                                                                                                                    "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                                                                                                                                    "details": [
                                                                                                                                        {
                                                                                                                                            "value": 56,
                                                                                                                                            "description": "docFreq",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 2838,
                                                                                                                                            "description": "docCount",
                                                                                                                                            "details": []
                                                                                                                                        }
                                                                                                                                    ]
                                                                                                                                },
                                                                                                                                {
                                                                                                                                    "value": 1.6861606,
                                                                                                                                    "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                                                                                                                                    "details": [
                                                                                                                                        {
                                                                                                                                            "value": 1,
                                                                                                                                            "description": "termFreq=1.0",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 1.2,
                                                                                                                                            "description": "parameter k1",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 0.75,
                                                                                                                                            "description": "parameter b",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 1944.5743,
                                                                                                                                            "description": "avgFieldLength",
                                                                                                                                            "details": []
                                                                                                                                        },
                                                                                                                                        {
                                                                                                                                            "value": 10.24,
                                                                                                                                            "description": "fieldLength",
                                                                                                                                            "details": []
                                                                                                                                        }
                                                                                                                                    ]
                                                                                                                                }
                                                                                                                            ]
                                                                                                                        }
                                                                                                                    ]
                                                                                                                }
                                                                                                            ]
                                                                                                        },
                                                                                                        {
                                                                                                            "value": 0,
                                                                                                            "description": "match on required clause, product of:",
                                                                                                            "details": [
                                                                                                                {
                                                                                                                    "value": 0,
                                                                                                                    "description": "# clause",
                                                                                                                    "details": []
                                                                                                                },
                                                                                                                {
                                                                                                                    "value": 1,
                                                                                                                    "description": "_type:__elements, product of:",
                                                                                                                    "details": [
                                                                                                                        {
                                                                                                                            "value": 1,
                                                                                                                            "description": "boost",
                                                                                                                            "details": []
                                                                                                                        },
                                                                                                                        {
                                                                                                                            "value": 1,
                                                                                                                            "description": "queryNorm",
                                                                                                                            "details": []
                                                                                                                        }
                                                                                                                    ]
                                                                                                                }
                                                                                                            ]
                                                                                                        }
                                                                                                    ]
                                                                                                }
                                                                                            ]
                                                                                        }
                                                                                    ]
                                                                                }
                                                                            ]
                                                                        },
                                                                        {
                                                                            "value": 1,
                                                                            "description": "sum of:",
                                                                            "details": [
                                                                                {
                                                                                    "value": 1,
                                                                                    "description": "*:*, product of:",
                                                                                    "details": [
                                                                                        {
                                                                                            "value": 1,
                                                                                            "description": "boost",
                                                                                            "details": []
                                                                                        },
                                                                                        {
                                                                                            "value": 1,
                                                                                            "description": "queryNorm",
                                                                                            "details": []
                                                                                        }
                                                                                    ]
                                                                                }
                                                                            ]
                                                                        }
                                                                    ]
                                                                },
                                                                {
                                                                    "value": 1,
                                                                    "description": "min of:",
                                                                    "details": [
                                                                        {
                                                                            "value": 1,
                                                                            "description": "field value function: none(doc['boost_factor'].value * factor=1.0)",
                                                                            "details": []
                                                                        },
                                                                        {
                                                                            "value": 3.4028235e+38,
                                                                            "description": "maxBoost",
                                                                            "details": []
                                                                        }
                                                                    ]
                                                                }
                                                            ]
                                                        },
                                                        {
                                                            "value": 0,
                                                            "description": "match on required clause, product of:",
                                                            "details": [
                                                                {
                                                                    "value": 0,
                                                                    "description": "# clause",
                                                                    "details": []
                                                                },
                                                                {
                                                                    "value": 1,
                                                                    "description": "#*:* -_type:__*, product of:",
                                                                    "details": [
                                                                        {
                                                                            "value": 1,
                                                                            "description": "boost",
                                                                            "details": []
                                                                        },
                                                                        {
                                                                            "value": 1,
                                                                            "description": "queryNorm",
                                                                            "details": []
                                                                        }
                                                                    ]
                                                                }
                                                            ]
                                                        }
                                                    ]
                                                },
                                                {
                                                    "value": 1,
                                                    "description": "primaryWeight",
                                                    "details": []
                                                }
                                            ]
                                        },
                                        {
                                            "value": 189.49968,
                                            "description": "product of:",
                                            "details": [
                                                {
                                                    "value": 2.368746,
                                                    "description": "max of:",
                                                    "details": [
                                                        {
                                                            "value": 2.368746,
                                                            "description": "weight(elements.content.stem:term in 17) [PerFieldSimilarity], result of:",
                                                            "details": [
                                                                {
                                                                    "value": 2.368746,
                                                                    "description": "score(doc=17,freq=2.0 = termFreq=2.0\n), product of:",
                                                                    "details": [
                                                                        {
                                                                            "value": 3.9530065,
                                                                            "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                                                                            "details": [
                                                                                {
                                                                                    "value": 54,
                                                                                    "description": "docFreq",
                                                                                    "details": []
                                                                                },
                                                                                {
                                                                                    "value": 2838,
                                                                                    "description": "docCount",
                                                                                    "details": []
                                                                                }
                                                                            ]
                                                                        },
                                                                        {
                                                                            "value": 0.5992265,
                                                                            "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                                                                            "details": [
                                                                                {
                                                                                    "value": 2,
                                                                                    "description": "termFreq=2.0",
                                                                                    "details": []
                                                                                },
                                                                                {
                                                                                    "value": 1.2,
                                                                                    "description": "parameter k1",
                                                                                    "details": []
                                                                                },
                                                                                {
                                                                                    "value": 0.75,
                                                                                    "description": "parameter b",
                                                                                    "details": []
                                                                                },
                                                                                {
                                                                                    "value": 238.7012,
                                                                                    "description": "avgFieldLength",
                                                                                    "details": []
                                                                                },
                                                                                {
                                                                                    "value": 1337.4694,
                                                                                    "description": "fieldLength",
                                                                                    "details": []
                                                                                }
                                                                            ]
                                                                        }
                                                                    ]
                                                                }
                                                            ]
                                                        }
                                                    ]
                                                },
                                                {
                                                    "value": 80,
                                                    "description": "secondaryWeight",
                                                    "details": []
                                                }
                                            ]
                                        }
                                    ]
                                },
                                {
                                    "value": 1,
                                    "description": "primaryWeight",
                                    "details": []
                                }
                            ]
                        },
                        {
                            "value": 4.737492,
                            "description": "product of:",
                            "details": [
                                {
                                    "value": 2.368746,
                                    "description": "weight(elements.content.stem:term in 17) [PerFieldSimilarity], result of:",
                                    "details": [
                                        {
                                            "value": 2.368746,
                                            "description": "score(doc=17,freq=2.0 = termFreq=2.0\n), product of:",
                                            "details": [
                                                {
                                                    "value": 3.9530065,
                                                    "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                                                    "details": [
                                                        {
                                                            "value": 54,
                                                            "description": "docFreq",
                                                            "details": []
                                                        },
                                                        {
                                                            "value": 2838,
                                                            "description": "docCount",
                                                            "details": []
                                                        }
                                                    ]
                                                },
                                                {
                                                    "value": 0.5992265,
                                                    "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                                                    "details": [
                                                        {
                                                            "value": 2,
                                                            "description": "termFreq=2.0",
                                                            "details": []
                                                        },
                                                        {
                                                            "value": 1.2,
                                                            "description": "parameter k1",
                                                            "details": []
                                                        },
                                                        {
                                                            "value": 0.75,
                                                            "description": "parameter b",
                                                            "details": []
                                                        },
                                                        {
                                                            "value": 238.7012,
                                                            "description": "avgFieldLength",
                                                            "details": []
                                                        },
                                                        {
                                                            "value": 1337.4694,
                                                            "description": "fieldLength",
                                                            "details": []
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                },
                                {
                                    "value": 2,
                                    "description": "secondaryWeight",
                                    "details": []
                                }
                            ]
                        }
                    ]
                }

[jimczi]

I think it's just a known issue with rescorer. If the window_size (which default to 10) of the rescorer is smaller than the requested size (from+size, in the example query 0+67) then all documents that appear after the window_size will skip the rescore and they will just apply the query_weight to the original score. Though we don't check if the document was part of the window when explain is called which is why the score within the explanation is not correct. It shows the score as if it were rescored even though the document might not have been rescored.
You should have the same score if you change the window_size to match the size+from of you query (67 ;) ).
I'll leave this issue open for now because there is a bug in the rescorer explain output but the fix is not trivial and I am not sure that it is worth fixing.

[jpountz]

I think we should fix it: scores are explanations should always match.

[jimczi]

Fair enough, I restored the adoptme tag. Also note that the score will not always match the explanations since we can have more than one rescorer so only the explanation of the last rescorer should match the final score of the hit.

[mayya-sharipova]

To implement the correct explain in QueryRescorer explain function, we need to check if a document was within the window_size - N. This is difficult to check, as documents can be resorted, and the top N documents are not necessarily the one for which rescoring was applied (in case for example rescore_query_weight is negative, they may be at the bottom). We can potentially modify RescoreContext to save all docIds for which rescore was applied, and use this info during ExplainFetchSubPhase, but I am not sure if we want to go this round.

The simpler solution would be to remove window_size parameter from rescore API, and apply rescore for all docs between [from, from+size).

What do you think @jimczi @jpountz ?

[jimczi]

The simpler solution would be to remove window_size parameter from rescore API, and apply rescore for all docs between [from, from+size).

The window_size is important because it ensures that pagination queries always rescore the same hits. If we remove it the pagination would rescore more documents on each page and this would yields different results (or duplicates) on each page.

We can potentially modify RescoreContext to save all docIds for which rescore was applied, and use this info during ExplainFetchSubPhase, but I am not sure if we want to go this round.

The window_size should be small (100-1000 or something) so I think that's ok and I don't see how we could do it differently.