Kibana 4.3.1/4.3.3 discovery across multiple monthly indices - sets size=0 / search_type=‘count’

[scaph01]

Environment:
ES 2.1.1
Kibana 4.3.1 and Kibana 4.3.3

I have a set of monthly going back a couple of years. When using Kibana in discovery, searching back a few months, I am seeing the histogram is showing the proper hits, but in the table below, the data for some of the indices is missing. What I am, for instance, if I am searching for data from Jan 3 2016 until March 15 2016, I am seeing from my browser an _msearch request, for each month (Jan, Feb and March), which is perfect.

However the request for Feb is always leaving with a search_style="count" and size set to zero, like this:

{"index":["events2-cpqa1-2016-02"],"search_type":"count","ignore_unavailable":true}
{"size":0,"sort":[{"EventTime":

compared to Jan and March:

{"index":["events2-cpqa1-2016-01"],"ignore_unavailable":true}
{"size":500,"sort":[{"EventTime":  ....

The search result size in total is 240 documents and I have set the discover:sampleSize to it's max of 9,999 with no change in behavior.

Additional details

• I cannot reproduce with the really simple 1 document per monthly index, I tried to add dozens of documents to see if I found a tipping point but could not reproduce
  -- my Jan index has 690K documents (my filter will match ony 120 documents from this index)
  -- Feb has 245K documents (my filter will match only 27 documents)
  -- March as 505K (my filter will match only about 100 documents)
• When sorted in descending time mode the behavior changes slightly where the indicies are queried from newest to oldest but still the last index request is requested with size=0
• If I repeat this on data in a daily index, I cannot seem to make this happen.

[spalger]

Is it possible to share your actual data? If so a snapshot created using the snapshot API would be greatly appreciated.

[spalger]

PS: if you can share privately but not publicly you can send me a link via IRC. I'm spalger on freenode

[scaph01]

Hi Spencer, I will see what I can do.

[scaph01]

ahh, after some debugging I see the problem. The cluster was es 1.7.1 until sometime in February.

The following is the response to the initial _field_stats request, notice the change in the formatting of the min_value and max_value from a date string to the epoc.

"indices": {
    "events2-cpqa1-2016-03": {
        "fields": {
            "EventTime": {
                "max_doc": 553460,
                "doc_count": 553460,
                "density": 100,
                "sum_doc_freq": 2213840,
                "sum_total_term_freq": -1,
                "min_value": "2016-03-01T00:00:07.008Z",
                "max_value": "2016-03-31T23:59:58.864Z"
            }
        }
    },
    "events2-cpqa1-2016-01": {
        "fields": {
            "EventTime": {
                "max_doc": 693504,
                "doc_count": 693504,
                "density": 100,
                "sum_doc_freq": 2774016,
                "sum_total_term_freq": -1,
                "min_value": "1451606415240",
                "max_value": "1454284799723"
            }
        }
    },
    "events2-cpqa1-2016-02": {
        "fields": {
            "EventTime": {
                "max_doc": 409333,
                "doc_count": 409333,
                "density": 100,
                "sum_doc_freq": 1637332,
                "sum_total_term_freq": -1,
                "min_value": "1454284801467",
                "max_value": "1456790397783"
            }
        }
    }

What is happening is then in the queue we are placing min and max values that are NaN

this results in the function SegmentedReq.prototype._pickSizeForIndices to return 0, which results in the search request saying "count" and size:0

I will now focus on what the difference was with the indices between es 1.7.1 and 2.1.1, but @spalger do you know off the top of your head if the date time format changed.

[spalger]

In elasticsearch 2+ the min and max values for dates now come back as milliseconds from epoch, so that we don't have to worry about formatting.

[scaph01]

Thanks Spenser, so are you saying that existing clusters that update are S.O.L. when it comes to some of the discovery capabilities? Should I look up at the cluster side to see if I can do something to get the old indices to return milliseconds from epoch?

[rashidkpc]

@scaph01 Can you check your mapping? Are you sure things are getting indexed correctly? Make sure you're not accidentally indexing as a number into some indices?

[scaph01]

@rashidkpc - thanks - I checked production Kibana is not showing any mapping conflicts, the testbed is however showing this (for a non-time related field). I am seeing the issue on both production and the testbed...but for my sanity I will find the index that has a type mismatch. While I hunt do you mind explaining the rationale around checking the mapping? I guess I would like to understand where the index time range is calculated and how the mapping could influence that.

[scaph01]

@rashidkpc - so I cleaned up the index that was causing some field mapping issues. However I will note that I am still seeing the client sending a "size:0" request.

I then started to look at the field_stats response, and then I started to see where you were going, did the mapping for out field with the event time change? looking at the recent indices for the month there is a change:

> curl -XGET http://es1-sal-cpqa1-van:9200/events2-cpqa1-2016-0*/_mapping/event/field/EventTime?pretty=1
{
  "events2-cpqa1-2016-01" : {
    "mappings" : {
      "event" : {
        "EventTime" : {
          "full_name" : "EventTime",
          "mapping" : {
            "EventTime" : {
              "type" : "date",
              "format" : "epoch_millis||dateOptionalTime"
            }
          }
        }
      }
    }
  },
  "events2-cpqa1-2016-02" : {
    "mappings" : {
      "event" : {
        "EventTime" : {
          "full_name" : "EventTime",
          "mapping" : {
            "EventTime" : {
              "type" : "date",
              "format" : "epoch_millis||dateOptionalTime"
            }
          }
        }
      }
    }
  },
  "events2-cpqa1-2016-03" : {
    "mappings" : {
      "event" : {
        "EventTime" : {
          "full_name" : "EventTime",
          "mapping" : {
            "EventTime" : {
              "type" : "date",
              "format" : "strict_date_optional_time||epoch_millis"
            }
          }
        }
      }
    }
  },
  "events2-cpqa1-2016-04" : {
    "mappings" : {
      "event" : {
        "EventTime" : {
          "full_name" : "EventTime",
          "mapping" : {
            "EventTime" : {
              "type" : "date",
              "format" : "strict_date_optional_time||epoch_millis"
            }
          }
        }
      }
    }
  }
}

We upgraded from ES 1.7.1 to 2.1.1 in Feb 2016, with the March index being the first created under es 2.1.1. The type was always known as date. the format did change though.

I checked our mapping and it was always:
"properties": { "EventTime": { "type": "date"}

now that we are in April, seems that the new indices are returning date string formatted ranges:

> curl -XGET "http://es1-sal-cpqa1-van:9200/events2-cpqa1-2016-04/_field_stats?fields=EventTime&pretty=1" 
{                                                                                                                             
  "_shards" : {                                                                                                               
    "total" : 1,                                                                                                              
    "successful" : 1,                                                                                                         
    "failed" : 0                                                                                                              
  },                                                                                                                          
  "indices" : {                                                                                                               
    "_all" : {                                                                                                                
      "fields" : {                                                                                                            
        "EventTime" : {                                                                                                       
          "max_doc" : 70938,                                                                                                  
          "doc_count" : 70938,                                                                                                
          "density" : 100,                                                                                                    
          "sum_doc_freq" : 283752,                                                                                            
          "sum_total_term_freq" : -1,                                                                                         
          "min_value" : "2016-04-01T00:00:02.027Z",                                                                           
          "max_value" : "2016-04-05T21:39:12.444Z"                                                                            
        }                                                                                                                     
      }                                                                                                                       
    }                                                                                                                         
  }                                                                                                                                                                                                                                                         
}                                                                                                                    `

I am thinking that we will have to be specific in the mapping to make sure the format is epoch millis and re-index the March and April data

[scaph01]

FYI: using the mapping:
"EventTime": { "type": "date", format: "epoch_millis" }

does seem to force the returned field_stats for the date time field to be in epoch_millis. I think I can work around this with a mapping update and re-index of old data

[scaph01]

Sorry for the flood of details, checking production though its field_stats are all in epoch_millis, so I will debug the code against prod to see what could be happening there

Here is the response:
{
"_shards": {
"total": 145,
"successful": 145,
"failed": 0
},
"indices": {
"events2-prod-2016-01": {
"fields": {
"EventTime": {
"max_doc": 928523,
"doc_count": 928523,
"density": 100,
"sum_doc_freq": 3714092,
"sum_total_term_freq": -1,
"min_value": "1451606407577",
"max_value": "1454284784617"
}
}
},
"events2-prod-2016-03": {
"fields": {
"EventTime": {
"max_doc": 1191735,
"doc_count": 1191735,
"density": 100,
"sum_doc_freq": 4766940,
"sum_total_term_freq": -1,
"min_value": "1456790403461",
"max_value": "1459468799917"
}
}
},
"events2-prod-2016-02": {
"fields": {
"EventTime": {
"max_doc": 593393,
"doc_count": 593393,
"density": 100,
"sum_doc_freq": 2373572,
"sum_total_term_freq": -1,
"min_value": "1454284807039",
"max_value": "1456790382944"
}
}
}
}

But in Kibana, the status is saying "NaN" for each of the built objects.

[scaph01]

So, looking deeper some more, what I am seeing is that even with

"events2-prod-2016-02": {
"fields": {
"EventTime": {
...
"min_value": "1454284807039",
"max_value": "1456790382944"
}

I am seeing the map function in
sortIndexStats(indices, timeFieldName, sortDirection) { ...

is assuming that when the min_value is a string that its a date string format value. I suspect, the call to moment(VALUE) should be checking if the value is numeric and casting over to an int for instance something like this "works" moment(parseInt(field.max_value)).

So I am thinking that this is a bug in Kibana 4.x in its handling of this response information about the index. If I want to work around this, I think I really want to force my indices to use the ISO 8601 format (and not millis since epoch). I would only want to take action after some feedback, because if you deem the epoch the "better" format to receive then I will not need to reindex all data (just my most recent indices)

There is a comment in the code that references:
elastic/elasticsearch#14404

which is partially to blame here as well, where the assumption was the "String" was the ISO format.

So to recap:

mapping in place
"properties": { "EventTime": { "type": "date"}

For our EventTime:
With ES 1.7 the field stats returns as millis since epoch (with quotes so its technically a string)
With ES 2.1.1 the field stats returns as an ISO 8601 string

Kibana is passing the field stats to moment() and that is returning NaN when the value is the millis since epoch. If that value was cast to a numeric type then it works.

[scaph01]

hey guys - any thoughts on my comments above, this has been lingering for a while.
Thanks

[Bargs]

@scaph01 sorry this is been sitting so long, Rashid's out on vacation and Spencer is pretty busy with other projects.

I just read through all the details and I agree with your assessment. Switching all your mappings to strict_date_optional_time||epoch_millis should resolve the issue. Your other option would be to upgrade Kibana and Elasticsearch, since the bug you mentioned (elastic/elasticsearch#14404) was fixed in Elasticsearch 2.2.

[scaph01]

Thanks for the feedback @Bargs . I will get back into this and see about moving to es 2.3 (there are some features we can make use of)

[timroes]

I'll be closing this since this seems very old and I can't see any immediate bug report or feature request in it. Please feel free to leave a comment in case this issue is still relevant.