Skip to content

Commit

Permalink
[DOCS] Moved rolled up index content (elastic#59372)
Browse files Browse the repository at this point in the history
* [DOCS] Moved rolled up index content

* Review comments
  • Loading branch information
KOTungseth committed Mar 10, 2020
1 parent d6592cc commit 85e088f
Show file tree
Hide file tree
Showing 3 changed files with 88 additions and 85 deletions.
9 changes: 6 additions & 3 deletions docs/management/index-patterns.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -52,17 +52,20 @@ If {kib} detects an index with a timestamp, you’re asked to choose a field to
filter your data by time. If you don’t specify a field, you won’t be able
to use the time filter.



[float]
[[rollup-index-pattern]]
==== Rollup index pattern

If a rollup index is detected in the cluster, clicking *Create index pattern*
includes an item for creating a rollup index pattern.
You can match an index pattern to only rolled up data, or mix both rolled
up and raw data to explore and visualize all data together.
An index pattern can match
only one rollup index.
only one rollup index. When matching multiple indices,
use a comma to separate the names, with no space after the comma.

For specific fields, the data in a rollup index includes only summarized metrics.
From the original raw data, you are unable to search any other field.

[float]
[[management-cross-cluster-search]]
Expand Down
79 changes: 38 additions & 41 deletions docs/management/rollups/create_and_manage_rollups.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
== Rollup jobs


A rollup job is a periodic task that aggregates data from indices specified
by an index pattern and rolls it into a new index. Rollup indices are a good way to
compactly store months or years of historical
A rollup job is a periodic task that aggregates data from indices specified
by an index pattern and rolls it into a new index. Rollup indices are a good way to
compactly store months or years of historical
data for use in visualizations and reports.

You’ll find *Rollup Jobs* under *Management > Elasticsearch*. With this UI,
You’ll find *Rollup Jobs* under *Management > Elasticsearch*. With this UI,
you can:

* <<create-and-manage-rollup-job, Create a rollup job>>
Expand All @@ -17,22 +17,22 @@ you can:
[role="screenshot"]
image::images/management_rollup_list.png[][List of currently active rollup jobs]

Before using this feature, you should be familiar with how rollups work.
{ref}/xpack-rollup.html[Rolling up historical data] is a good source for more detailed information.
Before using this feature, you should be familiar with how rollups work.
{ref}/xpack-rollup.html[Rolling up historical data] is a good source for more detailed information.

[float]
[[create-and-manage-rollup-job]]
=== Create a rollup job

{kib} makes it easy for you to create a rollup job by walking you through
the process. You fill in the name, data flow, and how often you want to roll
up the data. Then you define a date histogram aggregation for the rollup job
and optionally terms, histogram, and metrics aggregations.
{kib} makes it easy for you to create a rollup job by walking you through
the process. You fill in the name, data flow, and how often you want to roll
up the data. Then you define a date histogram aggregation for the rollup job
and optionally terms, histogram, and metrics aggregations.

When defining the index pattern, you must enter a name that is different than
the output rollup index. Otherwise, the job
will attempt to capture the data in the rollup index. For example, if your index pattern is `metricbeat-*`,
you can name your rollup index `rollup-metricbeat`, but not `metricbeat-rollup`.
When defining the index pattern, you must enter a name that is different than
the output rollup index. Otherwise, the job
will attempt to capture the data in the rollup index. For example, if your index pattern is `metricbeat-*`,
you can name your rollup index `rollup-metricbeat`, but not `metricbeat-rollup`.

[role="screenshot"]
image::images/management_create_rollup_job.png[][Wizard that walks you through creation of a rollup job]
Expand All @@ -41,38 +41,39 @@ image::images/management_create_rollup_job.png[][Wizard that walks you through c
[[manage-rollup-job]]
=== Start, stop, and delete rollup jobs

Once you’ve saved a rollup job, you’ll see it the *Rollup Jobs* overview page,
where you can drill down for further investigation. The *Manage* menu in
Once you’ve saved a rollup job, you’ll see it the *Rollup Jobs* overview page,
where you can drill down for further investigation. The *Manage* menu in
the lower right enables you to start, stop, and delete the rollup job.
You must first stop a rollup job before deleting it.

[role="screenshot"]
image::images/management_rollup_job_details.png[][Rollup job details]

You can’t change a rollup job after you’ve created it. To select additional fields
or redefine terms, you must delete the existing job, and then create a new one
with the updated specifications. Be sure to use a different name for the new rollup
job&mdash;reusing the same name can lead to problems with mismatched job configurations.
You can read more at {ref}/rollup-job-config.html[rollup job configuration].
You can’t change a rollup job after you’ve created it. To select additional fields
or redefine terms, you must delete the existing job, and then create a new one
with the updated specifications. Be sure to use a different name for the new rollup
job&mdash;reusing the same name can lead to problems with mismatched job configurations.
You can read more at {ref}/rollup-job-config.html[rollup job configuration].

[float]
[[rollup-data-tutorial]]
=== Try it: Create and visualize rolled up data

This example creates a rollup job to capture log data from sample web logs.
This example creates a rollup job to capture log data from sample web logs.
To follow along, add the <<add-sample-data, sample web logs data set>>.

In this example, you want data that is older than 7 days in the target index pattern `kibana_sample_data_logs`
to roll up once a day into the index `rollup_logstash`. You’ll bucket the
rolled up data on an hourly basis, using 60m for the time bucket configuration.
to roll up once a day into the index `rollup_logstash`. You’ll bucket the
rolled up data on an hourly basis, using 60m for the time bucket configuration.
This allows for more granular queries, such as 2h and 12h.

[float]
==== Create the rollup job

As you walk through the *Create rollup job* UI, enter the data shown in
the table below. The terms, histogram, and metrics fields reflect
the key information to retain in the rolled up data: where visitors are from (geo.src),
what operating system they are using (machine.os.keyword),
As you walk through the *Create rollup job* UI, enter the data shown in
the table below. The terms, histogram, and metrics fields reflect
the key information to retain in the rolled up data: where visitors are from (geo.src),
what operating system they are using (machine.os.keyword),
and how much data is being sent (bytes).

|===
Expand Down Expand Up @@ -118,31 +119,27 @@ and how much data is being sent (bytes).
|===


You can now use the rolled up data for analysis at a fraction of the storage cost
of the original index. The original data can live side by side with the new
You can now use the rolled up data for analysis at a fraction of the storage cost
of the original index. The original data can live side by side with the new
rollup index, or you can remove or archive it using <<creating-index-lifecycle-policies,Index Lifecycle Management>>.

[float]
==== Visualize the rolled up data

Your next step is to visualize your rolled up data in a vertical bar chart.
Your next step is to visualize your rolled up data in a vertical bar chart.
Most visualizations support rolled up data, with the exception of Timelion, TSVB, and Vega visualizations.

Using the information from the example rollup configuration described above,
you can use `rollup_logstash` to match the rolled up index pattern,
and `kibana_sample_data_logs` to match the index pattern for raw data.
The notation for a combination index pattern with both raw and rolled up data
Using the information from the example rollup configuration described above,
you can use `rollup_logstash` to match the rolled up index pattern,
and `kibana_sample_data_logs` to match the index pattern for raw data.
The notation for a combination index pattern with both raw and rolled up data
is `rollup_logstash,kibana_sample_data_logs`.

[role="screenshot"]
image::images/management_rollup_job_vis.png[][Visualization of rolled up data]

You can then create a dashboard that contains visualizations of the rolled up
data, raw data, or both. See <<visualize-rollup-data, Using rolled up data in a visualization>>
for more information.
You can then create a dashboard that contains visualizations of the rolled up
data, raw data, or both. For more information, refer to <<rollup-index-pattern,rolled up data>>.

[role="screenshot"]
image::images/management_rollup_job_dashboard.png[][Dashboard with rolled up data]



85 changes: 44 additions & 41 deletions docs/user/visualize.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -7,73 +7,78 @@ _Visualize_ enables you to create visualizations of the data from your {es} indi

{kib} visualizations are based on {es} queries. By using a series of {es} {ref}/search-aggregations.html[aggregations] to extract and process your data, you can create charts that show you the trends, spikes, and dips you need to know about.

To begin, open *Visualize*, then click *Create new visualization*.

[float]
[[create-a-visualization]]
== Create visualizations
[[visualization-types]]
== Types of visualizations

{kib} supports several types of visualizations.

. Open *Visualize*.
. Click *Create new visualization*.
. Choose the visualization type:
+
* Basic charts
[horizontal]
<<lens,Lens>>::
Quickly build several types of basic visualizations by simply dragging and dropping the data fields you want to display.

* *<<most-frequent,Most frequently used visualizations>>*
[horizontal]
Line&comma; area, and bar charts:: Compare different series in X/Y charts.
<<most-frequent,Most frequently used visualizations>>::

Pie chart:: Display each source contribution to a total.
* *Line, area, and bar charts* &mdash; Compares different series in X/Y charts.

Data table:: Flattens aggregations into table format.
* *Pie chart* &mdash; Displays each source contribution to a total.

Metric:: Display a single number.
* *Data table* &mdash; Flattens aggregations into table format.

Goal and gauge:: Display a number with progress indicators.
* *Metric* &mdash; Displays a single number.

Tag cloud:: Display words in a cloud, where the size of the word corresponds to its importance.
* *Goal and gauge* &mdash; Displays a number with progress indicators.

* *Time series optimized*
[horizontal]
<<TSVB,TSVB>>:: Visualize time series data using pipeline aggregations.
* *Tag cloud* &mdash; Displays words in a cloud, where the size of the word corresponds to its importance.

<<timelion,Timelion>>:: Compute and combine data from multiple time series
<<TSVB,TSVB>>:: Visualizes time series data using pipeline aggregations.

<<timelion,Timelion>>:: Computes and combine data from multiple time series
data sets.

* *Maps*
[horizontal]
<<maps,Elastic Maps>>:: The most powerful way of visualizing map data in {kib}.
<<visualize-maps, Maps>>::
* *<<maps,Elastic Maps>>* &mdash; Displays geospatial data in {kib}.

* *Coordinate map* &mdash; Displays points on a map using a geohash aggregation.

<<visualize-maps,Coordinate map>>:: Displays points on a map using a geohash aggregation.
* *Region map* &mdash; Merges any structured map data onto a shape.

<<visualize-maps,Region map>>:: Merge any structured map data onto a shape.
* *Heat map* &mdash; Displays shaded cells within a matrix.

<<visualize-maps,Heat map>>:: Display shaded cells within a matrix.
<<for-dashboard,Dashboard tools>>::

* *<<for-dashboard,Dashboard tools>>*
[horizontal]
Controls:: Adds interactive inputs to a Dashboard.
* *Markdown widget* &mdash; Displays free-form information or instructions.

Markdown widget:: Display free-form information or instructions.
* *Controls* &mdash; Adds interactive inputs to a dashboard.

* *For developers*
[horizontal]
<<vega-graph,Vega>>:: Complete control over query and display.
<<vega-graph,Vega>>:: Completes control over query and display.

. Specify a search query to retrieve the data for your visualization:
** To enter new search criteria, select the <<index-patterns, index pattern>> for the indices that
[float]
[[choose-your-data]]
== Choose your data

Specify a search query to retrieve the data for your visualization, or used rolled up data.

* To enter new search criteria, select the <<index-patterns, index pattern>> for the indices that
contain the data you want to visualize. The visualization builder opens
with a wildcard query that matches all of the documents in the selected
indices.
** To build a visualization from a saved search, click the name of the saved

* To build a visualization from a saved search, click the name of the saved
search you want to use. The visualization builder opens and loads the
selected query.
+
NOTE: When you build a visualization from a saved search, any subsequent
modifications to the saved search are automatically reflected in the
visualization. To disable automatic updates, you can disconnect a visualization
from the saved search.
modifications to the saved search are reflected in the
visualization. To disable automatic updates, delete the visualization
on the *Saved Object* page.

* To build a visualization using <<rollup-index-pattern,rolled up data>>, select
the index pattern that includes the data. Rolled up data is summarized into
time buckets that can be split into sub buckets for numeric field values or
terms. To lower granularity, use a time aggregation that uses and combines
several time buckets. For an example, refer to <<rollup-data-tutorial,Create and visualize rolled up data>>.

[float]
[[vis-inspector]]
Expand Down Expand Up @@ -124,8 +129,6 @@ to view an embedded visualization.
* Generate a PNG report.

--
include::{kib-repo-dir}/visualize/visualize_rollup_data.asciidoc[]

include::{kib-repo-dir}/visualize/aggregations.asciidoc[]

include::{kib-repo-dir}/visualize/lens.asciidoc[]
Expand Down

0 comments on commit 85e088f

Please sign in to comment.