Document mapping type name can't start with '_', found: [_doc]

[martinssipenko]

After upgrading to v2.8.0 ES started to return 400s with following error:

{
  "type": "invalid_type_name_exception",
  "reason": "Document mapping type name can't start with '_', found: [_doc]"
}

I am using AWS ElasticSearch v6.0.1.

Reverting back to v2.7.0 fixed the issue.

[ymatsiuk]

the same issue here:

2018-03-06 12: 01: 42 +0000 [error]: #0 Could not push log to Elasticsearch: {
    "took"=>2,
    "errors"=>true,
    "items"=>[
        {
            "index"=>{
                "_index"=>"index-2018.03.06",
                "_type"=>"_doc",
                "_id"=>"AWH7MEyla3epn6q_f6J2",
                "status"=>400,
                "error"=>{
                    "type"=>"invalid_type_name_exception",
                    "reason"=>"Document mapping type name can't start with '_', found: [_doc]"
                }
            }
        },
        {
            "index"=>{
                "_index"=>"index-2018.03.06",
                "_type"=>"_doc",
                "_id"=>"AWH7MEyla3epn6q_f6J7",
                "status"=>400,
                "error"=>{
                    "type"=>"invalid_type_name_exception",
                    "reason"=>"Document mapping type name can't start with '_', found: [_doc]"
                }
            }
        }
    ]
}

Version:

2018-03-06 12:01:13 +0000 [info]: starting fluentd-1.1.0 pid=7 ruby="2.3.3"
2018-03-06 12:01:13 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby2.3", "-Eascii-8bit:ascii-8bit", "/usr/local/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2018-03-06 12:01:13 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.8.0'
2018-03-06 12:01:13 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2018-03-06 12:01:13 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1'
2018-03-06 12:01:13 +0000 [info]: gem 'fluent-plugin-record-modifier' version '1.0.2'
2018-03-06 12:01:13 +0000 [info]: gem 'fluent-plugin-systemd' version '0.3.1'
2018-03-06 12:01:13 +0000 [info]: gem 'fluentd' version '1.1.0'

[cosmo0920]

Please fill issue template. Please don’t ignore it.

Thanks.

[martinssipenko]

Sorry about ignoring it, but basically the issue is that plugin sends _doc where it sent fluentd before and ES for some reason does not like it. Note here that I'm using ES provider by AWS, which might be acting slightly different than self-hosted ES.

[ymatsiuk]

This is Dockerfile you can use to re-produce this behaviour:

FROM fluent/fluentd:v1.1.0-debian-onbuild

RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev dh-autoreconf libltdl-dev" \
 && apt-get -qq update \
 && apt-get install -qqy --no-install-recommends $buildDeps \
 && sudo gem install fluent-plugin-elasticsearch \
 && sudo gem install fluent-plugin-record-modifier \
 && sudo gem install fluent-plugin-multi-format-parser \
 && sudo gem install fluent-plugin-systemd \
 && sudo gem install fluent-plugin-prometheus \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -qqy --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
           /home/fluent/.gem/ruby/2.3.0/cache/*.gem

COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
CMD exec fluentd -c /fluentd/etc/${FLUENTD_CONF} -p /fluentd/plugins $FLUENTD_OPT

[cosmo0920]

Specifying type_name fluentd can solve issue?
If so, I'll revert default value of type_name parameter and adding warning log to notify to change default behavior in the future.

[cosmo0920]

@ymatsiuk I cannot reproduce your environment.
Because your Dockerfile wants to refer local fluentd.conf, and official fluentd Dockerfile does not refer fluent-plugin-elasticsearch: https://github.com/fluent/fluentd-docker-image/blob/master/v1.0/debian-onbuild/Dockerfile

[cosmo0920]

@martinssipenko Don't you specify type_name parameter explicitly in your Fluentd configuration?
If you don't specify it, v2.8.0 affects in your environment.
And I'm planning to use _doc type_name by default in ES plugin v3 which is working on Fluentd v2.

[ymatsiuk]

@cosmo0920 adding type_name fluentd fixes the issue. Thanks.

[martinssipenko]

@cosmo0920 no I do not specify type_name fluentd, you can use _doc in v3, but if you do so in v2 it's a BC breaking change.

[cosmo0920]

I've released v2.8.1 to fix this issue.

[sandstrom]

You probably know, but in Elasticsearch 6.2 the bug causing this trouble is fixed and using _doc works.

Also, my apologies, I should have done more research before suggesting this change. Although ES docs stated that this should work from 6.x, it didn't fully work until 6.2 due to the aforementioned bug.

[cosmo0920]

Thanks for the information.
But some users seems to use old ES 6.x and not to be able to replace easily.