Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

blog: Add Fluentd v1.17.0 release announcement #312

Merged
merged 8 commits into from
Jun 12, 2024
Merged
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
211 changes: 211 additions & 0 deletions content/blog/20240430_fluentd-v1.17.0-has-been-released.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
# Fluentd v1.17.0 has been released

Hi users!

We have released v1.17.0 on 2024-04-30. ChangeLog is [here](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430).

This release is a new release of v1.17 series.
In this release, we added some new features for some plugins and fixed bugs of Parser.

## Enhancement

### `in_tail`: Add `glob_policy` option for expanding glob capability of `path` and `exclude_path`

In this release, we added a new option [glob_policy](https://docs.fluentd.org/input/tail#glob_policy) for [in_tail](https://docs.fluentd.org/input/tail) plugin.

In previous versions, we can use only `*` in glob patterns for [path](https://docs.fluentd.org/input/tail#path) and [exclude_path](https://docs.fluentd.org/input/tail#exclude_path) option.

Example:

```
path /path/to/*
exclude_path ["/path/to/*.gz", "/path/to/*.zip"]
```

From this version, we can also use `[]`, `?`, and `{}` in glob patterns depending on the `glob_policy` option.

Example:

```
path "[0-1][2-3].log"
glob_policy extended
```

Please see [the document](https://docs.fluentd.org/input/tail#glob_policy) and [#4401](https://github.com/fluent/fluentd/pull/4401) for more information.

### `out_http`: Support AWS Signature Version 4 authentication

In this release, we added a new option `aws_sigv4` for the [method](https://docs.fluentd.org/output/http#method) setting of [out_http](https://docs.fluentd.org/output/http) plugin.

By using this option, `out_http` can use [AWS Signature Version 4](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html).

For example, this allows `out_http` to write to [Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html).

Please see [the document](https://docs.fluentd.org/output/http#method) and [#4459](https://github.com/fluent/fluentd/pull/4459) for more information.

### `out_http`: Add option to reuse connections

In this release, we add a new option [reuse_connections](https://docs.fluentd.org/output/http#reuse_connections) for [out_http](https://docs.fluentd.org/output/http) plugin.

This option will improve throughput of `out_http`.

Please see [the document](https://docs.fluentd.org/output/http#reuse_connections) and [#4330](https://github.com/fluent/fluentd/pull/4330) for more information.

### `in_http`: Recognize CSP reports as JSON data

In this release, we make the data type of the request where the `Content-Type` is `application/csp-report` be considered JSON by default.

Now, `in_http` can receive [Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)'s report by default.

Please see [#4282](https://github.com/fluent/fluentd/pull/4282) for more information.

## Bug Fixes

### Make sure parser returns hash

The record data in a Fluentd event must be a hash object.

* [Life of a Fluentd event](https://docs.fluentd.org/quickstart/life-of-a-fluentd-event)

However, in the previous versions, some parser plugins could return a non-hash object, such as an array.
It could cause errors in subsequent processing.

In this release, the following parser plugins have been fixed.

* [parser_json](https://docs.fluentd.org/parser/json)
* [parser_msgpack](https://docs.fluentd.org/parser/msgpack)

The changes are as follows.

* Make sure to return a hash record.
* Make sure to accept only a hash or an array of hash.

Here are the details for each case.

#### Example of changed behavior

Config:

```
<source>
@type tcp
tag test.tcp
<parse>
@type json
</parse>
</source>

<match test.**>
@type stdout
</match>
```

Send an array data:

```
$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]
```

The result before this version:

```
{datetime} test.tcp: [{"k":"v"},{"k2":"v2"}]
```

The result after this version:

```
{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}
```

#### Example of resolved error

Config:

```
<source>
@type tcp
tag test.tcp
<parse>
@type json
null_empty_string
</parse>
</source>

<match test.**>
@type stdout
</match>
```

Send an array data:

```
$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]
```

The result before this version:

```
{datetime} [error]: #0 unexpected error on reading data host="xxx" port=xxx error_class=NoMethodError error="undefined method `each_key' for [{\"k\"=>\"v\"}, {\"k2\"=>\"v2\"}]:Array"
```

The result after this version:

```
{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}
```

#### Remaining problem: filter_parser

In the previous versions, `filter_parser` could return an array record based on this wrong behavior.
From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case.
This should be improved in the future.

Here is an example.

```
<source>
@type sample
tag test.array
sample {"message": "[{\"k\":\"v\"}, {\"k2\":\"v2\"}]"}
</source>

<filter test.**>
@type parser
key_name message
<parse>
@type json
</parse>
</filter>

<match test.**>
@type stdout
</match>
```

The result before this version:

```
{datetime} test.array: [{"k":"v"},{"k2":"v2"}]
```

The result after this version:

```
{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: '[{\"k\":\"v\"}, {\"k2\":\"v2\"}]'" location=nil tag="test.array" time=xxx record={"k2"=>"v2"}
{datetime} test.array: {"k":"v"}
```

These are the major changes for this release.

In addition, some performance improvements have been included!
Please see [ChangeLog](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430) for details!

Enjoy logging!

TAG: Fluentd Announcement
AUTHOR: clearcode