Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Saved Objects API for exporting and importing dashboards #27220

Merged
merged 27 commits into from
Aug 16, 2021
Merged

Use Saved Objects API for exporting and importing dashboards #27220

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
d584667
Initialize exporting dashboards
kvch Aug 2, 2021
b5d8612
tmp
kvch Aug 3, 2021
e9c0418
tmp
kvch Aug 3, 2021
172ca9d
cleanup
kvch Aug 4, 2021
ef2cc98
follow-up tests
kvch Aug 4, 2021
459e3de
rm old dashboards
kvch Aug 4, 2021
23c97d7
more fixes
kvch Aug 4, 2021
fc404c0
more follow-up
kvch Aug 4, 2021
2d44a6c
tmp
kvch Aug 4, 2021
3f9a156
fix go vet
kvch Aug 5, 2021
e93c8d3
follow up in test
kvch Aug 5, 2021
773ba59
add docs
kvch Aug 5, 2021
08b925c
update reference in module files
kvch Aug 5, 2021
d1c4067
more more
kvch Aug 5, 2021
88c52b3
fix e2e tests
kvch Aug 5, 2021
8ec3b6e
follo up more
kvch Aug 5, 2021
def1720
one more
kvch Aug 5, 2021
23f47fe
go vet
kvch Aug 5, 2021
60642e8
address review notes
kvch Aug 9, 2021
ffcc7a3
follow up tests
kvch Aug 9, 2021
53072cc
follow up tests
kvch Aug 9, 2021
978336e
Skip test if Kibana is not required version
kvch Aug 10, 2021
ff1ca85
create separate value for minimum required version
kvch Aug 10, 2021
eec349e
add missing winlogbeat dashboards
kvch Aug 11, 2021
85cd464
add changelog entry
kvch Aug 11, 2021
5541d2d
parse last line
kvch Aug 13, 2021
513eedf
add check for empty lines
kvch Aug 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 19 additions & 13 deletions libbeat/dashboards/decode.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,25 +47,31 @@ func DecodeExported(exported []byte) []byte {
line, err := r.ReadBytes('\n')
if err != nil {
if err == io.EOF {
return result
return append(result, decodeLine(line)...)
}
return exported
}
o := common.MapStr{}
err = json.Unmarshal(line, &o)
result = append(result, decodeLine(line)...)
}
}

func decodeLine(line []byte) []byte {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think json.Unmarshal will return an error if you try to decode an empty slice.

You might want to do a if len(bytes.TrimSpace(line)) == 0 { return }. This handles both empty slices and blank lines (which are allowable per the NDJSON spec you linked to.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I have added this check to more functions that needed it.

o := common.MapStr{}
err := json.Unmarshal(line, &o)
if err != nil {
return line
}
var result []byte
for _, key := range responseToDecode {
// All fields are optional, so errors are not caught
err := decodeValue(o, key)
if err != nil {
continue
}
for _, key := range responseToDecode {
// All fields are optional, so errors are not caught
err := decodeValue(o, key)
if err != nil {
logger := logp.NewLogger("dashboards")
logger.Debugf("Error while decoding dashboard objects: %+v", err)
}
result = append(result, []byte(o.String())...)
logger := logp.NewLogger("dashboards")
logger.Debugf("Error while decoding dashboard objects: %+v", err)
}
result = append(result, []byte(o.String())...)
}
return result
}

func decodeValue(data common.MapStr, key string) error {
Expand Down
57 changes: 33 additions & 24 deletions libbeat/dashboards/modify_json.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,40 +181,49 @@ func ReplaceIndexInDashboardObject(index string, content []byte) []byte {
line, err := r.ReadBytes('\n')
if err != nil {
if err == io.EOF {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a subtle usage issue here in that ReadBytes is documented to return both data and io.EOF in some cases. So it should handle that data that is returned in this case. I recommend using bufio.Scanner here if you can (seems easier to use).

If ReadBytes encounters an error before finding a delimiter, it returns the data read before the error and the error itself (often io.EOF).

Copy link
Contributor Author

@kvch kvch Aug 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially started with bufio.Scanner, but the tokens were too big so I switched to bufio.Reader as the documentation recommended:

Programs that need more control over error handling or large tokens, or must run sequential scans on a reader, should use bufio.Reader instead.

I think there's a subtle usage issue here in that ReadBytes is documented to return both data and io.EOF in some cases. So it should handle that data that is returned in this case. I recommend using bufio.Scanner here if you can (seems easier to use).

If ReadBytes encounters an error before finding a delimiter, it returns the data read before the error and the error itself (often io.EOF).

I might misunderstood the documentation, but I believe what I had written is correct. The data I am processing here is NDJSON. Thus, there must be a new line at the end of the JSON because it is the delimiter. If the delimiter is missing the JSON is invalid, and I return the data collected up until the error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you think it there should be some kind of error returned or error handling for malformed data that does not end in a newline? This implementation silently discards the data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There an error is logged a few lines below when there is an error. Also, the original data remain untouched, so the user can rerun the command after inspecting/fixing the input data.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case where the file ends with JSON not terminated by \n the method will return without logging any error since the method returns data and io.EOF. Here's a quick example to demonstrate it https://play.golang.org/p/OV_LTATSs98.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. I have tested with incomplete lines in a more complex scenario where it was correct. I have looked at the data exported from Kibana also, the spec from http://ndjson.org/ and it seems that EOF is a valid separator as well. So the errors we have to handle is anything besides EOF. So I still believe this is correct. :D

return result
return append(result, replaceInNDJSON(logger, index, line)...)
}
logger.Error("Error reading bytes from raw dashboard object: %+v", err)
return content
}
objectMap := make(map[string]interface{}, 0)
err = json.Unmarshal(line, &objectMap)
if err != nil {
result = append(result, append(line, newline...)...)
continue
}
result = append(result, replaceInNDJSON(logger, index, line)...)
}
}

attributes, ok := objectMap["attributes"].(map[string]interface{})
if !ok {
result = append(result, append(line, newline...)...)
continue
}
func replaceInNDJSON(logger *logp.Logger, index string, line []byte) []byte {
if len(line) == 0 {
return line
}

if kibanaSavedObject, ok := attributes["kibanaSavedObjectMeta"].(map[string]interface{}); ok {
attributes["kibanaSavedObjectMeta"] = ReplaceIndexInSavedObject(logger, index, kibanaSavedObject)
}
objectMap := make(map[string]interface{}, 0)
err := json.Unmarshal(line, &objectMap)
if err != nil {
logger.Errorf("Failed to convert bytes to map[string]interface: %+v", err)
return line
}

if visState, ok := attributes["visState"].(string); ok {
attributes["visState"] = ReplaceIndexInVisState(logger, index, visState)
}
attributes, ok := objectMap["attributes"].(map[string]interface{})
if !ok {
logger.Errorf("Object does not have attributes key")
return line
}

b, err := json.Marshal(objectMap)
if err != nil {
logger.Error("Error marshaling modified dashboard: %+v", err)
result = append(result, append(line, newline...)...)
}
if kibanaSavedObject, ok := attributes["kibanaSavedObjectMeta"].(map[string]interface{}); ok {
attributes["kibanaSavedObjectMeta"] = ReplaceIndexInSavedObject(logger, index, kibanaSavedObject)
}

result = append(result, append(b, newline...)...)
if visState, ok := attributes["visState"].(string); ok {
attributes["visState"] = ReplaceIndexInVisState(logger, index, visState)
}

b, err := json.Marshal(objectMap)
if err != nil {
logger.Error("Error marshaling modified dashboard: %+v", err)
return line
}

return append(b, newline...)

}

// ReplaceStringInDashboard replaces a string field in a dashboard
Expand Down
38 changes: 26 additions & 12 deletions libbeat/kibana/dashboard.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,23 +35,37 @@ func RemoveIndexPattern(data []byte) ([]byte, error) {
line, err := r.ReadBytes('\n')
if err != nil {
if err == io.EOF {
return result, nil
res, removeErr := removeLineIfIndexPattern(line)
if removeErr != nil {
return data, removeErr
}
return append(result, res...), nil
}
return data, err
}

var r common.MapStr
// Full struct need to not loose any data
err = json.Unmarshal(line, &r)
res, err := removeLineIfIndexPattern(line)
if err != nil {
return nil, err
}
v, err := r.GetValue("type")
if err != nil {
return nil, fmt.Errorf("type key not found or not string")
}
if v != "index-pattern" {
result = append(result, line...)
return data, err
}
result = append(result, res...)
}
}

func removeLineIfIndexPattern(line []byte) ([]byte, error) {
var r common.MapStr
// Full struct need to not loose any data
err := json.Unmarshal(line, &r)
if err != nil {
return nil, err
}
v, err := r.GetValue("type")
if err != nil {
return nil, fmt.Errorf("type key not found or not string")
}
if v != "index-pattern" {
return line, nil
}

return nil, nil
}