diff --git a/libbeat/docs/outputconfig.asciidoc b/libbeat/docs/outputconfig.asciidoc index 7c0e6f453133..5bbed6647ce3 100644 --- a/libbeat/docs/outputconfig.asciidoc +++ b/libbeat/docs/outputconfig.asciidoc @@ -106,10 +106,6 @@ The number of workers per configured host publishing events to Elasticsearch. Th is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host). -===== port - -The default port of the Elasticsearch server if the port number is missing in <> URL. The default port number is 9200. - ===== username The basic authentication username for connecting to Elasticsearch. @@ -148,9 +144,51 @@ for more information about the environment variables. ===== index -The index root name to write events to. The default is the Beat name. -For example "{beatname_lc}" generates "[{beatname_lc}-]YYYY.MM.DD" indexes (for example, -"{beatname_lc}-2015.04.26"). +The index name to write events to. The default is "{beatname_lc}-%{+yyyy.MM.dd}" (for example, "{beatname_lc}-2015.04.26"). + +===== indices + +Array of index selector rules supporting conditionals, format string +based field access and name mappings. The first rule matching will be used to +set the `index` for the event to be published. If `indices` is missing or no +rule matches, the `index` field will be used. + +Rule settings: + +*`index`*: The index format string to use. If the fields used are missing, the rule fails. + +*`mapping`*: Dictionary mapping index names to new names + +*`default`*: Default string value if `mapping` does not find a match. + +*`when`*: Condition which must succeed in order to execute the current rule. + +Examples elasticsearch output with `indices`: + +["source","yaml"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + index: "logs-%{+yyyy.MM.dd}" + indices: + - index: "critical-%{+yyyy.MM.dd}" + when.contains: + message: "CRITICAL" + - index: "error-%{+yyyy.MM.dd}" + when.contains: + message: "ERR" +------------------------------------------------------------------------------ + +===== pipeline + +format string value configuring the ingest node pipeline to write events to. + +===== pipelines + +Array of pipeline selector configurations supporting conditionals, format string +based field access and name mappings. The first rule matching will be used to +set the `pipeline` for the event to be published. If `pipelines` is missing or no +rule matches, the `pipeline` field will be used. ===== template @@ -530,6 +568,54 @@ The password for connecting to Kafka. The Kafka topic used for produced events. The setting can be a format string using any event field. To set the topic from document type use `%{[type]}`. +===== topics + +Array of topic selector rules supporting conditionals, format string +based field access and name mappings. The first rule matching will be used to +set the `topic` for the event to be published. If `topics` is missing or no +rule matches, the `topic` field will be used. + +Rule settings: + +*`topic`*: The topic format string to use. If the fields used are missing, the + rule fails. + +*`mapping`*: Dictionary mapping index names to new names + +*`default`*: Default string value if `mapping` does not find a match. + +*`when`*: Condition which must succeed in order to execute the current rule. + +===== key + +Optional Kafka event key. If configured, the event key must be unique and can be extracted from the event using a format string. + +===== partition + +Kafka output broker event partitioning strategy. Must be one of `random`, +`round_robin`, or `hash`. By default the `hash` partitioner is used. + +*`random.group_events`*: Sets the number of events to be published to the same + partition, before the partitioner selects a new partition by random. The + default value is 1 meaning after each event a new parition is picked randomly. + +*`round_robin.group_events`*: Sets the number of events to be published to the + same partition, before the partitioner selects the next partition. The default + value is 1 meaning after each event the next partition will be selected. + +*`hash.hash`*: List of fields used to compute the partitioning hash value from. + If no field is configured, the events `key` value will be used. + +*`hash.random`*: Randomly distribute events if no hash or key value can be computed. + +All partitioners will try to publish events to all partitions by default. If a +partition's leader becomes unreachable for the beat, the output might block. All +partitioners support setting `reachable_only` to overwrite this +behavior. If `reachable_only` is set to `true`, events will be published to +available partitions only. + +NOTE: Publishing to a subset of available partitions potentially increases resource usage because events may become unevenly distributed. + ===== client_id The configurable ClientID used for logging, debugging, and auditing purposes. The default is "beats". @@ -662,6 +748,57 @@ The name of the Redis list or channel the events are published to. The default i The name of the Redis list or channel the events are published to. The default is "{beatname_lc}". +The redis key can be set dynamically using a format string accessing any +fields in the event to be published. + +This configuration will use the `fields.list` field to set the redis list key. If +`fields.list` is missing, `fallback` will be used. + +["source","yaml"] +------------------------------------------------------------------------------ +output.redis: + hosts: ["localhost"] + key: "%{[fields.list]:fallback}" +------------------------------------------------------------------------------ + +===== keys + +Array of key selector configurations supporting conditionals, format string +based field access and name mappings. The first rule matching will be used to +set the `key` for the event to be published. If `keys` is missing or no +rule matches, the `key` field will be used. + +Rule settings: + +*`key`*: The key format string. If the fields used in the format string are missing, the rule fails. + +*`mapping`*: Dictionary mapping key values to new names + +*`default`*: Default string value if `mapping` does not find a match. + +*`when`*: Condition which must succeed in order to execute the current rule. + +Example `keys` settings: + +["source","yaml"] +------------------------------------------------------------------------------ +output.redis: + hosts: ["localhost"] + key: "default_list" + keys: + - key: "info_list" # send to info_list if `message` field contains INFO + when.contains: + message: "INFO" + - key: "debug_list" # send to debug_list if `message` field contains DEBUG + when.contains: + message: "DEBUG" + - key: "%{[type]}" + mapping: + "http": "frontend_list" + "nginx": "frontend_list" + "mysql": "backend_list" +------------------------------------------------------------------------------ + ===== password The password to authenticate with. The default is no authentication.