Skip to content

Commit

Permalink
datasource kafka: Add in_kafka plugin description
Browse files Browse the repository at this point in the history
  • Loading branch information
cosmo0920 committed May 11, 2016
1 parent 37fbf72 commit e256e2d
Showing 1 changed file with 31 additions and 1 deletion.
32 changes: 31 additions & 1 deletion content/datasources/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,32 @@ Fluentd can setup to collect messages from Kafka. Applications include:
1. Sending Kafka messages into HDFS for analysis
2. Sending Kafka messages into Elasticsearch for analysis

## Setup
You can two choices for this purpose whether using `in_kafka` or using `kafka-fluentd-consumer`.

## Setup: fluent-plugin-kafka

1. Install the [Kafka input plugin](https://github.com/htgc/fluent-plugin-kafka) by running the following command:

```
$ fluent-gem install fluent-plugin-kafka
```
2. Open your Fluentd configuration file and add the following lines:
```
<source>
@type kafka
host <broker host>
port <broker port: default=9092>
topics <listening topics(separate with comma',')>
format <input text type (text|json|ltsv|msgpack)>
message_key <key (Optional, for text format only, default is message)>
</source>
```
With the above setup, Fluentd consumes Kafka messages via `in_kafka` plugin.
## Setup: kafka-fluentd-consumer
1. Download the latest [kafka-fluentd-consumer jar](https://github.com/treasure-data/kafka-fluentd-consumer/releases).
Expand All @@ -29,3 +54,8 @@ Fluentd can setup to collect messages from Kafka. Applications include:
```
With the above setup, Fluentd consumes Kafka messages which are specified topics in `fluentd-consumer.properties` via `in_exec` plugin.
### Note
For simplification, you can use `in_kafka` plugin to retrive kafka messages.
If you assume highly kafka traffic in production, we recommend to use `kafka-fluentd-consumer` instead of `in_kafka`. Because `in_kafka` has been reported high CPU usage when 1000req/sec environment. In more detail, please refer to [the issue](https://github.com/htgc/fluent-plugin-kafka/issues/16).

0 comments on commit e256e2d

Please sign in to comment.