fix explain action on query rewrite

[fen-qin]

Description

There is an exception when call explain in "by_doc_id" mode. Response looks like this:

{
    "error": {
        "root_cause": [
            {
                "type": "query_shard_exception",
                "reason": "failed to create query: async actions are left after rewrite",
                "index": "my-nlp-index-1",
                "index_uuid": "I6_pzGG3QBWw0B6KeOJ1pA"
            }
        ],
        "type": "query_shard_exception",
        "reason": "failed to create query: async actions are left after rewrite",
        "index": "my-nlp-index-1",
        "index_uuid": "I6_pzGG3QBWw0B6KeOJ1pA",
        "caused_by": {
            "type": "illegal_state_exception",
            "reason": "async actions are left after rewrite"
        }
    },
    "status": 400
}

The error were caught from TransportExplainAction -> Rewriteable.java because there still be asyncActions left when the flag assertNoAsyncTasks returned as true.

The conflict is down to:

1. The NeuralQueryBuilder doRewrite method introduces asynchronous actions via queryRewriteContext.registerAsyncAction. So it then returns a context with potentially unresolved async tasks.

2. Rewritable.rewrite then checksassertNoAsyncTasks flag, throws the exception if there exists any async tasks left.

Fix in this PR is resolve this conflict:

• by moving the query rewrite to coordinator - TransportExplainAction.
• verify the changes from local
  • Before fix
  • 
  • After fix
  • 

Related Issues

Resolves #1126

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[bzhangam]

I think you want to remove this line.

[fen-qin]

sure, will remove this line in next revision

[bzhangam]

Why don't we simply switch from rewrite to rewriteAndFetch since now know this process can have async actions? Now basically we are doing rewriteAndFetch here and later I think we will still will do rewrite again.

[fen-qin]

here is Rewriteable javadoc - https://artifacts.elastic.co/javadoc/org/elasticsearch/elasticsearch/8.17.1/org.elasticsearch.server/org/elasticsearch/index/query/Rewriteable.html

• rewrite -
  Rewrites each element of the list until it doesn't change and returns a new list iff there is at least one element of the list that changed during it's rewrite.

• rewriteAndFetch -
  Rewrites the given rewriteable and fetches pending async tasks for each round before rewriting again.

• so we would need rewrite to be executed here

[github-actions]

❌ Gradle check result for b1f53c0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 11ebc0d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?