Investigate elastic/opensearch forks

[Jeffail]

The dust is starting to settle now since the opensearch fork so it's probably worth spending some time understanding what's needed in order to continue to support both offerings. We're currently using https://github.com/olivere/elastic which has been fantastic (more info about future direction here: olivere/elastic#1533) but we could also consider moving to the official client libraries as we only use a small fraction of ES endpoints.

Official libraries:

• https://github.com/elastic/go-elasticsearch
• https://github.com/opensearch-project/opensearch-go

Questions:

• Do we need a forked opensearch component?

[gedw99]

Hey @Jeffail

Some people ( me too ) want to run their own Elastic equivalent.

There is a golang equivalent that matches the Elastic API closely https://github.com/prabhatsharma/zinc
For collection they recommend https://github.com/fluent/fluent-logger-golang for logging into it

Its getting alot of attention. I guess tons of people want an efficient alternative.

It works well. It's based on bluge: https://github.com/blugelabs/bluge ( thankyou @mschoch ).

[ryn9]

@Jeffail I would be interested in seeing a dedicated output for OpenSearch built against the aforementioned opensearch-go. We could ask the coordinators of https://opensearch.org/events/ to allow you to present, showing off Benthos and its OpenSearch integration.

[arunx2]

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

[mihaitodor]

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

Go ahead if you have cycles to spare. All new contributors are welcome! Happy to help you get it in shape for merging

[arunx2]

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

Go ahead if you have cycles to spare. All new contributors are welcome! Happy to help you get it in shape for merging

I'm on it. Thanks!

[ryn9]

The official Go Client Library for OpenSearch - https://github.com/opensearch-project/opensearch-go

[arunx2]

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

Go ahead if you have cycles to spare. All new contributors are welcome! Happy to help you get it in shape for merging

I'm on it. Thanks!

@Jeffail here is the PR. #1388

[mihaitodor]

The opensearch output was released in v4.25.0.

We still need to migrate from olivere/elastic to https://github.com/elastic/go-elasticsearch.

[fearfate]

is there any process for opensearch input componements?