Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it clear in the docs that publish_async is still experimental #3096

Merged
merged 1 commit into from
Dec 1, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 4 additions & 27 deletions filebeat/docs/load-balancing.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
== Load Balancing

Filebeat provides configuration options that you can use to fine
tune load balancing when sending events to multiple hosts.
tune load balancing when sending events to multiple hosts.

To enable load balancing, you specify `loadbalance: true` when you configure
the output. For example:
Expand Down Expand Up @@ -32,7 +32,7 @@ The load balancer also supports multiple workers per host. The default is
connections will be used. The total number of workers participating
in load balancing is `number of hosts * workers`.
+
Example:
Example:
+
[source,yaml]
-------------------------------------------------------------------------------
Expand All @@ -46,15 +46,15 @@ output.logstash:
worker: 2
-------------------------------------------------------------------------------
+
In this example, there are 4 workers participating in load balancing.
In this example, there are 4 workers participating in load balancing.

* **Send events to `N` hosts in lock-step:**
+
You can configure Filebeat to send events to `N` hosts in lock-step by setting
`spool_size = N * bulk_max_size`. In lock-step mode, the batch collected by the
spooler is split up into smaller batches of size `bulk_max_size`. These smaller
batches are load balanced between available connections. Filebeat waits for all
sub-batches to be published before it retrieves another batch from the spooler.
sub-batches to be published before it retrieves another batch from the spooler.
+
This mode requires more memory and CPU usage than the previous mode.
+
Expand All @@ -72,26 +72,3 @@ output.logstash:
loadbalance: true
bulk_max_size: 2048
-------------------------------------------------------------------------------

* **Send events in parallel and asynchronously:**
+
You can configure Filebeat to send events in parallel and asynchronously by
setting `publish_async: true`. With this setting, Filebeat pushes a batch of
lines and then prepares a new batch of lines while waiting for the output to
ACK. This mode can improve load-balancing throughput, but requires the most
memory and CPU usage.
+
Example:
+
[source,yaml]
-------------------------------------------------------------------------------
filebeat.prospectors:
- input_type: log
paths:
- /var/log/*.log
filebeat.publish_async: true
output.logstash:
hosts: ["localhost:5044", "localhost:5045"]
loadbalance: true
-------------------------------------------------------------------------------

Original file line number Diff line number Diff line change
Expand Up @@ -483,6 +483,8 @@ See <<load-balancing>> for more information about how this setting affects load

===== publish_async

experimental[]

If enabled, the publisher pipeline in Filebeat operates in async mode preparing
a new batch of lines while waiting for ACK. This option can improve load-balancing
throughput at the cost of increased memory usage. The default value is false.
Expand Down
2 changes: 2 additions & 0 deletions filebeat/publisher/publisher.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (

"github.com/elastic/beats/filebeat/input"
"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/libbeat/logp"
"github.com/elastic/beats/libbeat/publisher"
)

Expand Down Expand Up @@ -33,6 +34,7 @@ func New(
pub publisher.Publisher,
) LogPublisher {
if async {
logp.Warn("Using publish_async is experimental!")
return newAsyncLogPublisher(in, out, pub)
}
return newSyncLogPublisher(in, out, pub)
Expand Down