[ML] adding support for composite aggs in anomaly detection

docs/reference/ml/anomaly-detection/ml-configuring-aggregations.asciidoc

@@ -11,51 +11,54 @@ distributes these calculations across your cluster. You can then feed this
 aggregated data into the {ml-features} instead of raw results, which
 reduces the volume of data that must be considered while detecting anomalies.
 
-TIP: If you use a terms aggregation and the cardinality of a term is high, the
-aggregation might not be effective and you might want to just use the default
-search and scroll behavior.
+TIP: If you use a terms aggregation and the cardinality of a term is high, you
+should use composite aggregations.
 
 [discrete]
 [[aggs-limits-dfeeds]]
 == Requirements and limitations
 
 There are some limitations to using aggregations in {dfeeds}. Your aggregation
-must include a `date_histogram` aggregation, which in turn must contain a `max`
-aggregation on the time field. This requirement ensures that the aggregated data
-is a time series and the timestamp of each bucket is the time of the last record
-in the bucket.
-
-IMPORTANT: The name of the aggregation and the name of the field that the agg 
-operates on need to match, otherwise the aggregation doesn't work. For example, 
-if you use a `max` aggregation on a time field called `responsetime`, the name 
+must include a `date_histogram` aggregation or a top level `composite` aggregation,
+which in turn must contain a `max` aggregation on the time field.
+This requirement ensures that the aggregated data is a time series and the timestamp
+of each bucket is the time of the last record in the bucket.
+
+For `composite` aggregation support, there must be exactly one `date_histogram` value
+source. Additionally, that value source must NOT be sorted in descending order. Additional
+`composite` aggregation value sources are allowed, such as `terms`.
+
+IMPORTANT: The name of the aggregation and the name of the field that the agg
+operates on need to match, otherwise the aggregation doesn't work. For example,
+if you use a `max` aggregation on a time field called `responsetime`, the name
 of the aggregation must be also `responsetime`.
 
-You must also consider the interval of the date histogram aggregation carefully.
-The bucket span of your {anomaly-job} must be divisible by the value of the
-`calendar_interval` or `fixed_interval` in your aggregation (with no remainder).
-If you specify a `frequency` for your {dfeed}, it must also be divisible by this
-interval. {anomaly-jobs-cap} cannot use date histograms with an interval 
-measured in months because the length of the month is not fixed. {dfeeds-cap} 
-tolerate weeks or smaller units.
+You must also consider the interval of the `date_histogram` or `composite`
+aggregation carefully. The bucket span of your {anomaly-job} must be divisible
+by the value of the `calendar_interval` or `fixed_interval` in your aggregation
+(with no remainder). If you specify a `frequency` for your {dfeed},
+it must also be divisible by this interval. {anomaly-jobs-cap} cannot use
+`date_histogram` or `composite` aggs with an interval measured in months
+because the length of the month is not fixed. {dfeeds-cap} tolerate weeks or smaller units.
 
 TIP: As a rule of thumb, if your detectors use <<ml-metric-functions,metric>> or
-<<ml-sum-functions,sum>> analytical functions, set the date histogram
+<<ml-sum-functions,sum>> analytical functions, set the `date_histogram` or `composite`
 aggregation interval to a tenth of the bucket span. This suggestion creates
 finer, more granular time buckets, which are ideal for this type of analysis. If
 your detectors use <<ml-count-functions,count>> or <<ml-rare-functions,rare>>
 functions, set the interval to the same value as the bucket span.
 
-If your <<aggs-dfeeds,{dfeed} uses aggregations with nested `terms` aggs>> and 
-model plot is not enabled for the {anomaly-job}, neither the **Single Metric 
-Viewer** nor the **Anomaly Explorer** can plot and display an anomaly 
-chart for the job. In these cases, the charts are not visible and an explanatory 
+If your <<aggs-dfeeds,{dfeed} uses aggregations with nested `terms` aggs>> and
+model plot is not enabled for the {anomaly-job}, neither the **Single Metric
+Viewer** nor the **Anomaly Explorer** can plot and display an anomaly
+chart for the job. In these cases, the charts are not visible and an explanatory
 message is shown.
 
-When the aggregation interval of the {dfeed} and the bucket span of the job 
-don't match, the values of the chart plotted in both the **Single Metric 
-Viewer** and the **Anomaly Explorer** differ from the actual values of the job. 
-To avoid this behavior, make sure that the aggregation interval in the {dfeed} 
-configuration and the bucket span in the {anomaly-job} configuration have the 
+When the aggregation interval of the {dfeed} and the bucket span of the job
+don't match, the values of the chart plotted in both the **Single Metric
+Viewer** and the **Anomaly Explorer** differ from the actual values of the job.
+To avoid this behavior, make sure that the aggregation interval in the {dfeed}
+configuration and the bucket span in the {anomaly-job} configuration have the
 same values.
 
 
@@ -86,8 +89,8 @@ PUT _ml/anomaly_detectors/farequote
 ----------------------------------
 // TEST[skip:setup:farequote_data]
 
-<1> The `airline`, `responsetime`, and `time` fields are aggregations. Only the 
-aggregated fields defined in the `analysis_config` object are analyzed by the 
+<1> The `airline`, `responsetime`, and `time` fields are aggregations. Only the
+aggregated fields defined in the `analysis_config` object are analyzed by the
 {anomaly-job}.
 
 NOTE: When the `summary_count_field_name` property is set to a non-null value,
@@ -134,23 +137,116 @@ PUT _ml/datafeeds/datafeed-farequote
 ----------------------------------
 // TEST[skip:setup:farequote_job]
 
-<1> The aggregations have names that match the fields that they operate on. The 
+<1> The aggregations have names that match the fields that they operate on. The
 `max` aggregation is named `time` and its field also needs to be `time`.
-<2> The `term` aggregation is named `airline` and its field is also named 
+<2> The `term` aggregation is named `airline` and its field is also named
 `airline`.
-<3> The `avg` aggregation is named `responsetime` and its field is also named 
+<3> The `avg` aggregation is named `responsetime` and its field is also named
 `responsetime`.
 
-Your {dfeed} can contain multiple aggregations, but only the ones with names 
+TIP: If you are utilizing a `term` aggregation to gather influencer or partition
+detector field information, consider using a `composite` aggregation. It will perform
+better than a `date_histogram` with a nested `term` aggregation and will also include
+ALL the values of the field instead of the top values per bucket.
+
+Your {dfeed} can contain multiple aggregations, but only the ones with names
 that match values in the job configuration are fed to the job.
 
+Here is a similar example utilizing a `composite` aggregation instead of a
+`date_histogram`.
+
+Assuming the same job configuration as above.
+
+[source,console]
+----------------------------------
+PUT _ml/anomaly_detectors/farequote-composite
+{
+  "analysis_config": {
+    "bucket_span": "60m",
+    "detectors": [{
+      "function": "mean",
+      "field_name": "responsetime",
+      "by_field_name": "airline"
+    }],
+    "summary_count_field_name": "doc_count"
+  },
+  "data_description": {
+    "time_field":"time"
+  }
+}
+----------------------------------
+// TEST[skip:setup:farequote_data]
+
+Here is a datafeed utilizing a `composite` aggregation to bucket
+the metrics based on time and terms.
+
+[source,console]
+----------------------------------
+{
+  "job_id": "farequote-composite",
+  "indices": [
+    "farequote"
+  ],
+  "aggregations": {
+    "buckets": {
+      "composite": {
+        "size": 1000,  <1>
+        "sources": [
+          {
+            "time_bucket": {  <2>
+              "date_histogram": {
+                "field": "time",
+                "fixed_interval": "360s",
+                "time_zone": "UTC"
+              }
+            }
+          },
+          {
+            "airline": {  <3>
+              "terms": {
+                "field": "airline"
+              }
+            }
+          }
+        ]
+      },
+      "aggregations": {
+        "time": {  <4>
+          "max": {
+            "field": "time"
+          }
+        },
+        "responsetime": { <5>
+          "avg": {
+            "field": "responsetime"
+          }
+        }
+      }
+    }
+  }
+}
+----------------------------------
+// TEST[skip:setup:farequote_job]
+
+<1> Provide the `size` to the composite agg to control how many resources
+are used when aggregated the data. A larger `size` means a faster datafeed but
+more cluster resources are utilized when searching.
+<2> The required `date_histogram` composite aggregation source. Make sure it
+is named differently than your desired time field.
+<3> Instead of utilizing a regular `term` aggregation, adding a composite
+aggregation `term` source with the name `airline` works. Note its name
+is the same as the field.
+<4> The required `max` aggregation whose name is the time field in the
+job analysis config
+<5> The `avg` aggregation is named `responsetime` and its field is also named
+`responsetime`.
 
 [discrete]
 [[aggs-dfeeds]]
 == Nested aggregations in {dfeeds}
 
-{dfeeds-cap} support complex nested aggregations. This example uses the 
-`derivative` pipeline aggregation to find the first order derivative of the 
+{dfeeds-cap} support complex nested aggregations. This example uses the
+`derivative` pipeline aggregation to find the first order derivative of the
 counter `system.network.out.bytes` for each value of the field `beat.name`.
 
 [source,js]
@@ -247,8 +343,9 @@ number of unique entries for the `error` field.
 [[aggs-define-dfeeds]]
 == Defining aggregations in {dfeeds}
 
-When you define an aggregation in a {dfeed}, it must have the following form:
+When you define an aggregation in a {dfeed}, it must have one of the following forms:
 
+When using a `date_histogram` aggregation to bucket by time
 [source,js]
 ----------------------------------
 "aggregations": {
@@ -282,26 +379,64 @@ When you define an aggregation in a {dfeed}, it must have the following form:
 ----------------------------------
 // NOTCONSOLE
 
+When using a `composite` aggregation
+
+[source,js]
+----------------------------------
+"aggregations": {
+  "composite_agg": {
+    "sources": [
+      {
+        "date_histogram_agg": {
+          "field": "time",
+          ...settings...
+        }
+      },
+      ...other valid sources...
+      ],
+      ...composite agg settings...,
+      "aggregations": {
+        "timestamp": {
+            "max": {
+              "field": "time"
+            }
+          },
+          ...other aggregations...
+          [
+            [,"aggregations" : {
+              [<sub_aggregation>]+
+            } ]
+          }]
+      }
+   }
+}
+----------------------------------
+// NOTCONSOLE
+
 The top level aggregation must be either a
 {ref}/search-aggregations-bucket.html[bucket aggregation] containing as single
-sub-aggregation that is a `date_histogram` or the top level aggregation is the
-required `date_histogram`. There must be exactly one `date_histogram` 
-aggregation. For more information, see
-{ref}/search-aggregations-bucket-datehistogram-aggregation.html[Date histogram aggregation].
+sub-aggregation that is a `date_histogram`, the top level aggregation is the
+required `date_histogram`, or the top leve aggregation is the required `composite`.
+There must be exactly one `date_histogram` or `composite` aggregation. For more information, see
+{ref}/search-aggregations-bucket-datehistogram-aggregation.html[Date histogram aggregation] and
+{ref}/search-aggregations-bucket-composite-aggregation.html[Composite aggregation].
 
 NOTE: The `time_zone` parameter in the date histogram aggregation must be set to
 `UTC`, which is the default value.
 
-Each histogram bucket has a key, which is the bucket start time. This key cannot
-be used for aggregations in {dfeeds}, however, because they need to know the
-time of the latest record within a bucket. Otherwise, when you restart a 
-{dfeed}, it continues from the start time of the histogram bucket and possibly 
-fetches the same data twice. The max aggregation for the time field is therefore
-necessary to provide the time of the latest record within a bucket.
+Each histogram or composite bucket has a key, which is the bucket start time.
+This key cannot be used for aggregations in {dfeeds}, however, because
+they need to know the time of the latest record within a bucket.
+Otherwise, when you restart a {dfeed}, it continues from the start time of the
+histogram or composite bucket and possibly fetches the same data twice.
+The max aggregation for the time field is therefore necessary to provide
+the time of the latest record within a bucket.
 
 You can optionally specify a terms aggregation, which creates buckets for
 different values of a field.
 
+TIP: Instead of nesting a `term` aggregation, try using `composite` aggs.
+
 IMPORTANT: If you use a terms aggregation, by default it returns buckets for
 the top ten terms. Thus if the cardinality of the term is greater than 10, not
 all terms are analyzed.
@@ -311,7 +446,7 @@ determine the cardinality of your data, you can run searches such as:
 
 [source,js]
 --------------------------------------------------
-GET .../_search 
+GET .../_search
 {
   "aggs": {
     "service_cardinality": {
@@ -324,10 +459,11 @@ GET .../_search
 --------------------------------------------------
 // NOTCONSOLE
 
+
 By default, {es} limits the maximum number of terms returned to 10000. For high
 cardinality fields, the query might not run. It might return errors related to
 circuit breaking exceptions that indicate that the data is too large. In such
-cases, do not use aggregations in your {dfeed}. For more information, see
+cases, use `composite` aggregations in your {dfeed}. For more information, see
 {ref}/search-aggregations-bucket-terms-aggregation.html[Terms aggregation].
 
 You can also optionally specify multiple sub-aggregations. The sub-aggregations

docs/reference/ml/ml-shared.asciidoc

@@ -1,7 +1,6 @@
 tag::aggregations[]
 If set, the {dfeed} performs aggregation searches. Support for aggregations is
-limited and should be used only with low cardinality data. For more information,
-see
+limited. For more information, see
 {ml-docs}/ml-configuring-aggregation.html[Aggregating data for faster performance].
 end::aggregations[]
 
@@ -156,7 +155,8 @@ tag::bucket-span[]
 The size of the interval that the analysis is aggregated into, typically between
 `5m` and `1h`. The default value is `5m`. If the {anomaly-job} uses a {dfeed}
 with {ml-docs}/ml-configuring-aggregation.html[aggregations], this value must be
-divisible by the interval of the date histogram aggregation. For more
+divisible by the interval of the `date_histogram` aggregation or the 
+`date_histogram` source in the `composite` aggregation. For more
 information, see {ml-docs}/ml-buckets.html[Buckets].
 end::bucket-span[]
 
@@ -824,7 +824,8 @@ for longer bucket spans, a sensible fraction of the bucket span. For example:
 `150s`. When `frequency` is shorter than the bucket span, interim results for
 the last (partial) bucket are written then eventually overwritten by the full
 bucket results. If the {dfeed} uses aggregations, this value must be divisible
-by the interval of the date histogram aggregation.
+by the interval of the `date_histogram` or the `date_histogram` source in the
+`composite` aggregation.
 end::frequency[]
 
 tag::frequent-category-count[]
@@ -1129,8 +1130,8 @@ There are three available modes:
 * `auto`: The chunk size is dynamically calculated. This is the default and
 recommended value when the {dfeed} does not use aggregations.
 * `manual`: Chunking is applied according to the specified `time_span`. Use this
-mode when the {dfeed} uses aggregations.
-* `off`: No chunking is applied.
+mode when the {dfeed} uses aggregations other than `composite`.
+* `off`: No chunking is applied. Recommended for `composite` aggregations.
 --
 end::mode[]
 

server/src/main/java/org/elasticsearch/search/aggregations/bucket/composite/DateHistogramValuesSourceBuilder.java

@@ -212,15 +212,21 @@ public DateHistogramValuesSourceBuilder fixedInterval(DateHistogramInterval inte
      *  {@code null} then it means that the interval is expressed as a fixed
      *  {@link TimeValue} and may be accessed via {@link #getIntervalAsFixed()} ()}. */
     public DateHistogramInterval getIntervalAsCalendar() {
-        return dateHistogramInterval.getAsCalendarInterval();
+        if (dateHistogramInterval.getIntervalType().equals(DateIntervalWrapper.IntervalTypeEnum.CALENDAR)) {
+            return dateHistogramInterval.getAsCalendarInterval();
+        }
+        return null;
     }
 
     /**
      * Get the interval as a {@link TimeValue}, regardless of how it was configured. Returns null if
      * the interval cannot be parsed as a fixed time.
      */
     public DateHistogramInterval getIntervalAsFixed() {
-        return dateHistogramInterval.getAsFixedInterval();
+        if (dateHistogramInterval.getIntervalType().equals(DateIntervalWrapper.IntervalTypeEnum.FIXED)) {
+            return dateHistogramInterval.getAsFixedInterval();
+        }
+        return null;
     }
 
     /**