Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev guide] Add docs on optional schema arguments #12371

Merged
Merged
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 45 additions & 6 deletions docs/devguide/create-metricset.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ common.MapStr{
}
----

You can use the following code to make the transformations:
You can use the schema package to transform the data, and optionally mark some fields in a schema as required or not. For example:

[source,go]
----
Expand All @@ -229,6 +229,36 @@ import (
c "github.com/elastic/beats/libbeat/common/schema/mapstrstr"
)

var (
schema = s.Schema{
"test_string": c.Str("testString", s.Required), <1>
"test_int": c.Int("testInt"), <2>
"test_bool": c.Bool("testBool", s.Optional), <3>
"test_float": c.Float("testFloat"),
"test_obj": s.Object{
"test_obj_string": c.Str("testObjString", s.IgnoreAllErrors), <4>
},
}
)

func eventMapping(input map[string]interface{}) common.MapStr {
return schema.Apply(input) <5>
}
----
<1> Marks a field as required.
<2> If a field has no schema option set, it is equivalent to `Required`.
<3> Marks the field as optional.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@odacremolbap added a new option to ignore all errors of an specific field (#12089), it may worth to mention it here too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IgnoreAllErrors option, it is aded by #12089

<4> Ignore any value conversion error
<5> By default, `Apply` will fail and return an error if any required field is missing. Using the optional second argument, you can specify how `Apply` handles different fields of the schema. The possible values are:
- `AllRequired` is the default behavior. Returns an error if any required field is missing, including fields that are required because no schema option is set.
- `FailOnRequired` will fail if a field explicitly marked as `required` is missing.
- `NotFoundKeys(cb func([]string))` takes a callback function that will be called with a list of missing keys, allowing for finer-grained error handling.

In the above example, note that it is possible to create the schema object once
and apply it to all events. You can also use `ApplyTo` to add additional data to an existing `MapStr` object:
[source,go]
----

var (
schema = s.Schema{
"test_string": c.Str("testString"),
Expand All @@ -239,15 +269,24 @@ var (
"test_obj_string": c.Str("testObjString"),
},
}

additionalSchema = s.Schema{
"second_string": c.Str("secondString"),
"second_int": ": c.Int("secondInt"),
}
)

func eventMapping(input map[string]interface{}) common.MapStr {
return schema.Apply(input)
}
data, err := schema.apply(input)
fearful-symmetry marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return err
}

if m.parseMoreData{
additionalSchema.ApplyTo(data, input)
fearful-symmetry marked this conversation as resolved.
Show resolved Hide resolved
}

----

In the above example, note that it is possible to create the schema object once
and apply it to all events.

[float]
==== Configuration File
Expand Down