Exclude system indexes from Elasticsearch connector during ingestion

[dN0Mad]

Is your feature request related to a problem? Please describe.
When ingesting data with the Elasticsearch connector, all indexes are fetched from Elasticsearch prior to filtering - including the system indexes (e.g. .transform-internal*, .kibana*. This causes problems w.r.t. cluster and index permissions required for the service user.

Describe the solution you'd like
Prevent the Elasticsearch connector from fetching the system indices before filtering.
Perhaps by updating the "get_search_index_list" method, to include the "expand_wildcards" ('all', 'closed', 'hidden', 'none', 'open') variable in "index_list = self.client.indices.get_alias() or {}".

Describe alternatives you've considered
NA

Additional context
Se documentation: https://elasticsearch-py.readthedocs.io/en/v8.10.1/api.html#elasticsearch.client.IndicesClient.get_alias

[dN0Mad]

Hi @ulixius9, do you know which iteration this feature may be implemented?

[ulixius9]

Hey @dN0Mad, we are planning fix it with 1.3 release