KNQL Does not query exact phrases containing multiple tokens/words

[aleeVS]

Plugin version:
6.2.1

Kibana version:
6.2.1

Plugins installed:
Elastic search 6.2.0

Description of the problem including expected versus actual behavior:
Unable to query body of the document for multi-phrase. If nesting is turned off, this works fine.
Example: finding "test post" from "this is a test post"

Expected behavior: find posts that contains the phrase
Actual behavior: "No results found"
The only way to fix this is to use keyword instead of text as the type, and then query body.keyword~="the post"

Steps to reproduce:

1. add posts
   curl -XPUT 'localhost:9200/vs_views/_doc/1?refresh&pretty' -H 'Content-Type: application/json' -d'
   {
   "body": "This is a text of the post",
   "post_date": "2017-01-01",
   }

curl -XPUT 'localhost:9200/vs_views/_doc/2?refresh&pretty' -H 'Content-Type: application/json' -d'
{
"body": "This is a another text of the post",
"post_date": "2016-12-01",
}
2. in kibana "Management" --> index pattern --> create index pattern; and then turn on nested fields
3. in kibana "Discover" search 'body="the post"'

Errors in browser console (if relevant):
No results found
Unfortunately I could not find any results matching your search. I tried really hard. I looked all over the place and frankly, I just couldn't find anything good. Help me, help you. Here are some ideas:

Refine your query
The search bar at the top uses Elasticsearch's support for Lucene Query String syntax. Let's say we're searching web server logs that have been parsed into a few fields.

Examples:
Find requests that contain the number 200, in any field:
200
Or we can search in a specific field. Find 200 in the status field:
status:200
Find all status codes between 400-499:
status:[400 TO 499]
Find status codes 400-499 with the extension php:
status:[400 TO 499] AND extension:PHP
Or HTML
status:[400 TO 499] AND (extension:php OR extension:html)
Provide logs and/or server output (if relevant):

If using docker, include your docker file:

[aleeVS]

Normal query using lucene query syntax without nested support

[aleeVS]

Activating nested support on an index, same query now cannot find the multi-token phrase

[ppadovani]

Thanks for the report, this looks like a limitation in the KNQL language. The equals(=) operator is a term query, the LIKE(~=) is wildcard. I did not code in a match_phrase into the language, as it wasn't a requirement at the time I built this out.

It would seem to me that the match_phrase is similar in behavior to a wildcard query without the actual wildcard operators. I.e. Re-using the concept of a LIKE makes sense in the language.

With this in mind, I will update the parser to check the value of the string passed into the LIKE expression and use wildcard if a * or ? is present, otherwise it will use match_phrase.

[aleeVS]

Thank you Pierre!
Just to be clear, it needs to be able to match MULTIPLE tokens, not just a SINGLE token with wild cards.

Also, alternatively, could the Nested Support be used without KNQL?

[ppadovani]

Yep. I've got the code working with your example.. it really was only about 10-12 lines of code to make the change.

Unfortunately, KNQL is required for indices with nested fields. The reason for this is the need to pre-parse the query and generate a native elastic search query that injects the correct nested path information into the query. In order to do this, the plugin gathers more information about the index from elastic search, and keeps track of the field information. When you build a query, the plugin's parser uses its knowledge of the index's mapping to properly generate nested queries without you having to manually inject additional information.

This breaks down under certain circumstances, as stated in the KNQL documentation. Hence the need for the EXISTS keyword to properly scope nested queries.

So, I'm sure your wondering why the lucene query language in Kibana doesn't work with nested fields. The main reason is that Kibana doesn't actually parse the query, it just blindly passes it to elastic search and has elastic search do the processing. Additionally, the lucene query language has no concept around nested fields built into it.

I've release 6.2.1-1.0.2 which should solve this issue. Please install the new version and let me know if it works for you. Then I'll back port to the other releases.

[ppadovani]

I believe this is now resolved and back ported to all releases. Please let me know if this is still an issue and I'll reopen.

[aleeVS]

Yes, double checked it works for my test case! I can get phrase matching by 'body~="the post"'. Also checked in the visualization and works with filters. Thank you!

…

On Thu, Feb 22, 2018 at 9:36 PM, Pierre Padovani ***@***.***> wrote: Closed #43 <#43>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYFG6omMPllV0MqSV174Rhp_rlWQNkbTks5tXiQ7gaJpZM4SOA6X> .

-- .................................................................................................. | Dr. Annie En-Shiun Lee (PhD) | Lead Research Scientist | | Data Science | VerticalScope Inc | (416) 341-8950 x403 | | wemine.uwaterloo.ca | linkedin.com/in/wemine |