Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow specify dynamic templates in bulk request #69948

Merged
merged 49 commits into from
Apr 8, 2021
Merged
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
40e6a57
Add dynamic mapping type hint to bulk request
dnhatn Mar 3, 2021
eed16ef
Merge branch 'master' into dynamic-mapping-hint
dnhatn Mar 8, 2021
27a538b
Remove TODO
dnhatn Mar 8, 2021
6ee5210
Merge branch 'master' into dynamic-mapping-hint
dnhatn Mar 9, 2021
69a3849
match_mapping_hint
dnhatn Mar 9, 2021
02b2e89
fix doc
dnhatn Mar 9, 2021
96795af
fix test
dnhatn Mar 9, 2021
8f2c6c4
Merge branch 'master' into dynamic-mapping-hint
dnhatn Mar 14, 2021
3c43e98
dynamicMappingHints -> dynamicMatchMappingHints
dnhatn Mar 14, 2021
de97281
javadocs
dnhatn Mar 14, 2021
635c395
stronger yaml test
dnhatn Mar 14, 2021
12a87ed
match mapping hint -> mapping hint
dnhatn Mar 14, 2021
f99abcd
add index request test
dnhatn Mar 14, 2021
d126606
revert tests
dnhatn Mar 14, 2021
f2e6c0f
Remove left over
dnhatn Mar 15, 2021
00c6454
use existing methods for test
dnhatn Mar 15, 2021
5679e5e
add test for bulk request
dnhatn Mar 15, 2021
7922692
more rename
dnhatn Mar 15, 2021
ea394f1
stylecheck
dnhatn Mar 15, 2021
d77290c
Merge branch 'master' into dynamic-mapping-hint
dnhatn Mar 18, 2021
290ed5c
feedback
dnhatn Mar 19, 2021
c6468fd
revert
dnhatn Mar 19, 2021
58af624
Use template name
dnhatn Mar 19, 2021
6cea974
Merge branch 'master' into dynamic-mapping-hint
dnhatn Mar 22, 2021
286ce59
fix docs
dnhatn Mar 22, 2021
8c9e402
fix message
dnhatn Mar 22, 2021
d2dfda1
Remove "hint" from the parameter
dnhatn Mar 22, 2021
d5a9231
wording docs
dnhatn Mar 22, 2021
a6f9971
Add a test for dynamic template with a wrong type
dnhatn Mar 22, 2021
9c3ecba
simplify match_mapping_type
dnhatn Mar 22, 2021
00b92c1
Simplify the validation
dnhatn Mar 22, 2021
a179238
minimize changes in serialization
dnhatn Mar 22, 2021
dd84c8a
Merge branch 'master' into dynamic-mapping-hint
dnhatn Mar 22, 2021
315c8da
more yaml tests
dnhatn Mar 23, 2021
0126070
remove hint
dnhatn Mar 23, 2021
b341fcc
Reject dynamic_templates on old nodes
dnhatn Mar 23, 2021
fd87800
Merge branch 'master' into dynamic-mapping-hint
dnhatn Mar 23, 2021
aef927e
Add min node version
dnhatn Mar 23, 2021
8a80841
stylecheck
dnhatn Mar 23, 2021
18d7b39
Revert "stylecheck"
dnhatn Mar 24, 2021
dccfb8e
Revert "Add min node version"
dnhatn Mar 24, 2021
5a22dff
better error message
dnhatn Mar 24, 2021
a69d244
Merge branch 'master' into dynamic-mapping-hint
dnhatn Mar 24, 2021
5900952
Merge 'master' into dynamic-mapping-hint
dnhatn Mar 30, 2021
07029f0
Merge branch 'master' into dynamic-mapping-hint
dnhatn Apr 7, 2021
18bcb64
move yaml test
dnhatn Apr 7, 2021
bf94a83
Update docs/reference/docs/bulk.asciidoc
dnhatn Apr 7, 2021
60d11a4
Allow new dynamic template format with old indices
dnhatn Apr 7, 2021
d0010bd
fix mapping syntax
dnhatn Apr 8, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions docs/reference/docs/bulk.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,8 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-id]

include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]

include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-dynamic-templates]
--

`delete`::
Expand Down Expand Up @@ -311,6 +313,8 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-id]

include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]

include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-dynamic-templates]
--

`update`::
Expand Down Expand Up @@ -738,3 +742,38 @@ The API returns the following result.
}
----
// TESTRESPONSE[s/"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA"/"index_uuid": $body.$_path/]


[discrete]
[[bulk-dynamic-templates]]
===== Example with dynamic templates parameter

The below example creates a dynamic template, then performs a bulk request
consisting of index/create requests with the `dynamic_templates` parameter.

[source,console]
----
PUT my-index/
{
"mappings": {
"dynamic_templates": [
{
"geo_point": {
"mapping": {
"mapping" : "geo_point"
}
}
}
]
}
}

POST /_bulk
{ "index" : { "_index" : "my_index", "_id" : "1", "dynamic_templates": {"work_location": "geo_point"}} }
{ "field" : "value1", "work_location": "41.12,-71.34", "raw_location": "41.12,-71.34"}
{ "create" : { "_index" : "my_index", "_id" : "2", "dynamic_templates": {"home_location": "geo_point"}} }
{ "field" : "value2", "home_location": "41.12,-71.34"}
----

The bulk request creates two new fields `work_location` and `home_location` with type `geo_point` according
to the `dynamic_templates` parameter; however, the `raw_location` field is created with the type `text`.
4 changes: 4 additions & 0 deletions docs/reference/mapping/dynamic/templates.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ name
* <<path-match-unmatch,`path_match` and `path_unmatch`>> operate on the full
dotted path to the field

* if a dynamic template does not define `match_mapping_type` nor `match` nor
`path_match`, then it won't match any field, but it can be referred to by
name in `dynamic_templates` as part of a <bulk, bulk request>.

Use the `{name}` and `{dynamic_type}` <<template-variables,template variables>>
in the mapping specification as placeholders.

Expand Down
9 changes: 9 additions & 0 deletions docs/reference/rest-api/common-parms.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -631,6 +631,15 @@ If `true`, the destination must be an <<indices-aliases,index alias>>. Defaults
`false`.
end::require-alias[]

tag::bulk-dynamic-templates[]
`dynamic_templates`::
(Optional, map)
A map from the full name of fields to the name of <<dynamic-templates, dynamic templates>.
Defaults to an empty map. If a name matches a dynamic template, then that template will be
applied regardless of other match predicates defined in the template. And if a field is
already defined in the mapping, then this parameter won't be used.
end::bulk-dynamic-templates[]

tag::node-filter[]
`<node_filter>`::
(Optional, string)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
---
"Dynamic templates":
- skip:
version: " - 7.99.99"
reason: "Dynamic templates parameter is added to bulk requests in 8.0"

- do:
indices.create:
index: test_index
body:
mappings:
dynamic_templates:
- location:
mapping:
type: geo_point
- my_location:
match: my*
mapping:
type: geo_point
- string:
mapping:
type: keyword
- do:
bulk:
refresh: true
body:
- index:
_index: test_index
_id: id_1
dynamic_templates:
location: location
- { "location": [ -71.34, 41.12 ]}
- index:
_index: test_index
_id: id_2
dynamic_templates:
location: location
- { "location": "41.12,-71.34"}
- match: { errors: false }
- match: { items.0.index.result: created }
- match: { items.1.index.result: created }

- do:
search:
index: test_index
body:
query:
geo_bounding_box:
location:
top_left:
lat: 42
lon: -72
bottom_right:
lat: 40
lon: -74
- match: { hits.total.value: 2 }
- match: { hits.hits.0._id: id_1 }
- match: { hits.hits.1._id: id_2 }

- do:
bulk:
refresh: true
body:
- index:
_index: test_index
_id: id_3
- { "my_location": "41.12,-71.34" } # matches the field name defined in the `my_location` template
- index:
_index: test_index
_id: id_4
dynamic_templates:
my_location: my_location
- { "my_location": "41.12,-71.34" } # use dynamic_templates parameter
- do:
search:
index: test_index
body:
query:
geo_bounding_box:
my_location:
top_left:
lat: 42
lon: -72
bottom_right:
lat: 40
lon: -74
- match: { hits.total.value: 2 }
- match: { hits.hits.0._id: id_3 }
- match: { hits.hits.1._id: id_4 }

- do:
bulk:
refresh: true
body:
- index:
_index: test_index
_id: id_5
dynamic_templates:
location: foo_bar # ok as fields are defined
- { "location": [ -71.34, 41.12 ]}
- index:
_index: test_index
_id: id_6
dynamic_templates:
my_location: foo_bar # ok as fields are defined
- { "my_location": "41.12,-71.34" }
- index:
_index: test_index
_id: id_7
dynamic_templates:
location: bar_foo # ok as fields are defined
- { "location": "41.12,-71.34" }
- match: { errors: false }
- match: { items.0.index.result: created }
- match: { items.1.index.result: created }
- match: { items.2.index.result: created }

- do:
bulk:
refresh: true
body:
- index:
_index: test_index
_id: id_8
dynamic_templates:
foo_location: bar_foo
- { "foo_location": [ -71.34, 41.12 ] } # failed because dynamic template is not found
- index:
_index: test_index
_id: id_9
dynamic_templates:
foo_location: foo_bar
- { "foo_location": "41.12,-71.34" } # failed because dynamic template is not found
- index:
_index: test_index
_id: id_10
dynamic_templates:
new_location: foo
- { "location": "41.12,-71.34"} # ok as fields are defined
- match: { errors: true }
- match: { items.0.index.status: 400 }
- match: { items.0.index.error.type: mapper_parsing_exception }
- match: { items.0.index.error.reason: "Can't find dynamic template for dynamic template name [bar_foo] of field [foo_location]"}
- match: { items.1.index.status: 400 }
- match: { items.1.index.error.type: mapper_parsing_exception }
- match: { items.1.index.error.reason: "Can't find dynamic template for dynamic template name [foo_bar] of field [foo_location]"}
- match: { items.2.index.status: 201 }
- match: { items.2.index.result: created }

# Dynamic template has a wrong type
- do:
bulk:
body:
- index:
_index: test_index
_id: id_11
dynamic_templates:
foo: string
- { "foo.bar": "hello world" } # failed because dynamic template has a wrong type
- index:
_index: test_index
_id: id_12
dynamic_templates:
foo.bar: string
- { "foo.bar": "hello world" } # ok
- match: { errors: true }
- match: { items.0.index.status: 400 }
- match: { items.0.index.error.type: mapper_parsing_exception }
- match: { items.0.index.error.reason: "Field [foo] must be an object; but it's configured as [keyword] in dynamic template [string]"}
- match: { items.1.index.status: 201 }
- match: { items.1.index.result: created }
Loading