Replace more places accessing Elasticsearch client directly with proxied calls #2016
Conversation
connectors/cli/connector.py
Outdated
| @@ -30,7 +30,8 @@ async def list_connectors(self): | |||
| # TODO move this on top | |||
| try: | |||
| await self.es_management_client.ensure_exists( | |||
| indices=[CONCRETE_CONNECTORS_INDEX, CONCRETE_JOBS_INDEX] | |||
| indices=[CONCRETE_CONNECTORS_INDEX, CONCRETE_JOBS_INDEX], | |||
| expand_wildcards="all", | |||
There was a problem hiding this comment.
I think don't need expand_wildcards anymore because connectors work only with one index.
I'm getting rid of the wildcards in this PR
There was a problem hiding this comment.
But this one is creating jobs and connectors indices that are hidden, I believe?
There was a problem hiding this comment.
I think expand_wildcards is needed only when you want to update multiple indices that match the pattern.
Here, we operate only with concrete indices.
connectors/cli/connector.py
Outdated
|
|
||
| async def __create_connector( | ||
| self, index_name, service_type, configuration, is_native, language, from_index | ||
| ): | ||
| try: | ||
| await self.es_management_client.ensure_exists( | ||
| indices=[CONCRETE_CONNECTORS_INDEX, CONCRETE_JOBS_INDEX] | ||
| indices=[CONCRETE_CONNECTORS_INDEX, CONCRETE_JOBS_INDEX], | ||
| expand_wildcards="all", |
There was a problem hiding this comment.
The same. I don't think we need this.
|
Great work! I left a couple of comments. Also, I don't know if it's a Pythonic way, but does it make sense to move ESManagementClient to a new file? |
artem-shelkovnikov
deleted the
artem/refactor-more-places-using-client-directly
branch
💔 Failed to create backport PR(s)The backport operation could not be completed due to the following error: The backport PRs will be merged automatically after passing CI. To backport manually run: |
Part of https://github.com/elastic/enterprise-search-team/issues/6412
This PR changes more places to use rich Elasticsearch clients instead of direct calls to python Elasticsearch client.
Example methods that were introduced or changed with this PR:
ESManagementClient.ensure_existsmethod got "expand_wildcards" argument that's default toopen- some places use "all", but most of the places use "open". "open" is also more conservative, thus the choiceESManagementClient.create_content_index- new method to create content index. Normally kibana does it, but in some cases (mostly CI or CLI) we wanna create the index ourselves. This method is just extracted from the original place that was calling non-rich Elasticsearch client with no changesESManagementClient.ensure_content_index_mappings- new method to ensure that content index has our search mappings. This method is just extracted from the original place that was calling non-rich Elasticsearch client with no changesESManagementClient.ensure_ingest_pipeline_exists- new method to ensure that ingest pipeline with provided id exists, if not it's created with provided arguments. Used bykibana.py. This method is just extracted from the original place that was calling non-rich Elasticsearch client with no changesESManagementClient.delete_indices- new method to delete indices. Used bykibana.py. This method is just extracted from the original place that was calling non-rich Elasticsearch client with no changesESManagementClient.upsert- new method to upsert a record into target index. Used bykibana.py. This method is just extracted from the original place that was calling non-rich Elasticsearch client with no changesESManagementClient.yield_existing_documents_metadata- formelyExtractor._get_existing_idsThere is still one change I haven't moved - it's call to
bulk, I will address it in later PR(s).All other changes faciliate the changes outlined above.
Checklists
Pre-Review Checklist
v7.13.2,v7.14.0,v8.0.0)