Backport: Add typless client side GetIndexRequest calls and response class #38420
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
) Today we keep the mapping on the follower in sync with the leader's using the mapping version from changes requests. There are two rare cases where the mapping on the follower is not synced properly: 1. The returned mapping version (from ClusterService) is outdated than the actual mapping. This happens because we expose the latest cluster state in ClusterService after applying it to IndexService. 2. It's possible for the FollowTask to receive an outdated mapping than the min_required_mapping. In that case, it should fetch the mapping again; otherwise, the follower won't have the right mapping. Relates to elastic#31140
…lastic#37792) The example plugins are currently built within the build-tools integ tests as a means to ensure the gradle plugin works for external plugin builds. These tests generate a dummy build.gradle, and a dummy local maven repository to find the local builds dependencies in. Currently that build-tools dependency uses "+" as the version. However, this allows gradle to find the "latest" version, and unfortunately gradle has its own plugin repository which is apparently connected to jcenter. This recently triggered a flood of CI failures when jcenter suddenly pulled alpha2, and all builds started trying to use that instead of the locally built build-tools. This commit uses the explicit version of build-tools that was build locally, which will cause resolution to stop when the local repo is first checked.
The EmptyResponse is essentially the same as returning a boolean, which is done in other places. This commit deprecates all the existing EmptyResponse methods and creates new boolean methods that have method params reordered so they can exist with the deprecated methods. A followup PR in master will remove the existing deprecated methods, fix the parameter ordering and deprecate the incorrectly ordered parameter methods. Relates elastic#36938
AuthenticateResponse did not allow unknown fields. This commit fixes the test and ConstructingObjectParser such that it does now allow unknown fields. Relates elastic#36938
The filtering by follower index was completely broken. Also the wrong persistent tasks were selected, causing the wrong status to be reported. Closes elastic#37738
Migrate ml job and datafeed config of open jobs and update the parameters of the persistent tasks as they become unallocated during a rolling upgrade. Block allocation of ml persistent tasks until the configs are migrated.
This PR deprecates the index audit output. In general, the problem with the index audit output is that event indexing can be slower than the rate with which audit events are generated, especially during the daily rollovers or the rolling cluster upgrades. In this situation audit events will be lost which is a terrible failure situation for an audit system. Besides of the settings under the `xpack.security.audit.index` namespace, the `xpack.security.audit.outputs` setting has also been deprecated and will be removed in 7. Although explicitly configuring the logfile output does not touch any deprecation bits, this setting is made redundant in 7 so this PR deprecates it as well. Relates elastic#29881
As acking can fail for any reason (unrelated node being too slow, node disconnecting), it should not be required for acking to succeed in order for index requests with dynamic mapping updates to successfully complete. Relates to elastic#30672 and Closes elastic#30844
…7710) The existing JavaDateMathParserTests were using joda formatters instead of java ones and thus the tests were not running the expected code. This fixes the above and some follow up failures due to that. First some round up formatter issues are fixed, especially when using epoch dates, the parsing could be wrong and result in exceptions. Second, some dates cannot be handled the same in java8 like in joda time due to more strict parsing, when used in combination with epoch dates.
Follow-up to elastic#31140
Support ANSI SQL's DATE type by introducing a runtime-only ES SQL date type. Closes: elastic#37340
…telyMultiType Due to elastic#37817
…lastic#37803) Improve the Exception and the error message returned when 2nd argument of PERCENTILE and PERCENTILE_RANK is not a constant.
…#37639) Users may require the sequence number and primary terms to perform optimistic concurrency control operations. Currently, you can get the sequence number via the `docvalues_fields` API but the primary term is not accessible because it is maintained by the `SeqNoFieldMapper` and the infrastructure can't find it. This commit adds a dedicated sub fetch phase to return both numbers that is connected to a new `seq_no_primary_term` parameter.
This class is pretty complex and only used in a test where we can simply fail the test with an assertion error.
(cherry picked from commit 163a27b)
Now that warning headers no longer contain a timestamp of when the warning was generated, we no longer need to extract the warning value from the warning to determine whether or not the warning value is duplicated. Instead, we can compare strings directly. Further, when de-duplicating warning headers, are constantly rebuilding sets. Instead of doing that, we can carry about the set with us and rebuild it if we find a new warning value. This commit applies both of these optimizations.
When we can't map the principal attribute from the configured SAML attribute in the realm settings, we can't complete the authentication. We return an error to the user indicating this and we present them with a list of attributes we did get from the SAML response to point out that the expected one was not part of that list. This list will never contain the NameIDs though as they are not part of the SAMLAttribute list. So we might have a NameID but just with a different format.
Similar fix as elastic#37370.
This change adds a docker compose configuration that's used with the `elasticsearch.test.fixtures` plugin to start up the image and check that the TCP ports are up. We can build on this to add other checks for culster health, run REST tests, etc. We can add multiple containers and configurations to the compose file (e.x. test different env vars) and form clusters.
* Testing conventions now checks for tests in main This is the last outstanding feature of the old NamingConventionsTask, so time to remove it. * PR review
Reverts elastic#36259 in part to make randomized test fail if no tests are ran. This is useful when filtering tests as it's easy to make a typo and think the test ran trough successfully.
…astic#38336) The explanation so far can be invaluable for troubleshooting as incorrect decisions made early on in the structure analysis can result in seemingly crazy decisions or timeouts later on. Relates elastic/kibana#29821
…) (elastic#38229) There was a bug where creating a new policy would start the ILM service, even if it was stopped. This change ensures that there is no change to the existing operation mode Backport of elastic#38134.
This commit introduces a background sync for retention leases. The idea here is that we do a heavyweight sync when adding a new retention lease, and then periodically we want to background sync any retention lease renewals to the replicas. As long as the background sync interval is significantly lower than the extended lifetime of a retention lease, it is okay if from time to time a replica misses a sync (it will still have an older version of the lease that is retaining more data as we assume that renewals do not decrease the retaining sequence number). There are two follow-ups that will come after this commit. The first is to address the fact that we have not adapted the should periodically flush logic to possibly flush the retention leases. We want to do something like flush if we have not flushed in the last five minutes and there are renewed retention leases since the last time that we flushed. An additional follow-up will remove the syncing of retention leases when a retention lease expires. Today this sync could be invoked in the background by a merge operation. Rather, we will move the syncing of retention lease expiration to be done under the background sync. The background sync will use the heavyweight sync (write action) if a lease has expired, and will use the lightweight background sync (replication action) otherwise.
The ILM status parser did not allow for unknown fields. This commit fixes that and adds an xContentTester to the response test. Relates elastic#36938 Backport of elastic#38043
…c#38364) Coalesces two calls into one in a scroll example so all callouts are at the end of the line. This is the only sort of callouts that are supported by asciidoctor and we'd like to start building our docs with asciidoctor. At present we don't have any mechanism to stop folks adding more inline callouts but we ought to be able to have one in a few weeks. For now, though, removing these inline callouts is a step in the right direction. Relates to elastic#38335
Fixes a race during the rolling upgrade with the index audit output enabled. The race is that after the upgraded node is restarted, it installs the audit template and updates the mapping of the "current" (from his perspective) audit index. But the template might be installed after a new daily rolled-over index has been created by the other old nodes, using the old templates. However, the new node, even if it installs the template after the rollover edge, can accumulate audit events before the edge, and will correctly try to update the mapping of the audit index before the edge. But this way, the mapping of the index after the edge remains un-updated, because only the master node does the mapping updates. The fix keeps the design of only allowing the master to update the mapping, but the master will try, on a best effort policy, to also possibly update the mapping of the next rollover audit index.
This test should not pass until CCR finishes integrating shard history retention leases. It currently sometimes passes (which is a bug in the test), but cannot pass reliably until the linked issue is resolved.
The User Agent ingest processor is changing to align with ECS. Users need to be made aware that any pipelines using this processor will behave differently in 7.0.
…stic#38377) This is related to elastic#35975. Currently when an index falls behind a leader it encounters a fatal exception. This commit adds a test for that scenario. Additionally, it tests that the user can stop following, close the follower index, and put follow again. After the indexing is re-bootstrapped, it will recover the documents it lost in normal following operations.
Added deprecation warnings for use of include_type_name in put/get index templates. HLRC changes: * GetIndexTemplateRequest has a new client-side class which is a copy of server's GetIndexTemplateResponse but modified to be typeless. * PutIndexTemplateRequest has a new client-side counterpart which doesn't use types in the mappings Relates to elastic#35190
In 6.x security is implicitly enabled on a trial license if transport SSL is enabled, or the trial is from pre-6.3. This is no longer true on 7.0, so this behaviour is now deprecated. Relates: elastic#38009, elastic#38075
Skip the languages in tests for which SimpleKdcServer does not handle generalized time correctly. Closes#38320
With this commit we add a monotonically strict timer to ensure time is advancing even if the timer is called in a tight loop in tests. We also relax a condition in a similar test so it only checks that time is not moving backwards. Closes elastic#33747 Relates elastic#38301 Relates elastic#38338
This adds a warning to the deprecations API for any SSL contexts that rely on the default `supported_protocols` list. This list will change in ES 7.0 and will no longer include TLS 1.0 by default.
We mention in our documentation for the token expiration configuration maximum value is 1 hour but do not enforce it. This commit adds max limit to the TOKEN_EXPIRATION setting. Note: Since this is a backport and the min max time value support was not there in 6.x, I have selectively picked the change from Setting. The changes were done for zen2.
Backport for elastic#38039 Relates to elastic#35190
…lastic#38397) Authn is enabled only if `license_type` is non `basic`, but `basic` is what the `LicenseService` generates implicitly. This commit explicitly sets license type to `trial`, which allows for authn, in the `SecuritySettingsSource` which is the settings configuration parameter for `InternalTestCluster`s. The real problem, that had created tests failures like elastic#31028 and elastic#32685, is that the check `licenseState.isAuthAllowed()` can change sporadically. If it were to return `true` or `false` during the whole test there would be no problem. The problem manifests when it turns from `true` to `false` right before `Realms.asList()`. There are other license checks before this one (request filter, token service, etc) that would not cause a problem if they would suddenly see the check as `false`. But switching to `false` before `Realms.asList()` makes it appear that no installed realms could have handled the authn token which is an authentication error, as can be seen in the failing tests. Closes elastic#31028 elastic#32685
…stic#37778) The HLRC client currently uses `org.elasticsearch.action.admin.indices.get.GetIndexRequest` and `org.elasticsearch.action.admin.indices.get.GetIndexResponse` in its get index calls. Both request and response are designed for the typed APIs, including some return types e.g. for `getMappings()` which in the maps it returns still use a level including the type name. In order to change this without breaking existing users of the HLRC API, this PR introduces two new request and response objects in the `org.elasticsearch.client.indices` client package. These are used by the IndicesClient#get and IndicesClient#exists calls now by default and support the type-less API. The old request and response objects are still kept for use in similarly named, but deprecated methods. The newly introduced client side classes are simplified versions of the server side request/response classes since they don't need to support wire serialization, and only the response needs fromXContent parsing (but no xContent-serialization, since this is the responsibility of the server-side class). Also changing the return type of `GetIndexResponse#getMapping` to `Map<String, MappingMetaData> getMappings()`, while it previously was returning another map keyed by the type-name. Similar getters return simple Maps instead of the ImmutableOpenMaps that the server side response objects return.
cbuescher
added
:Search Foundations/Mapping
|
Pinging @elastic/es-search |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport for #37778
Relates to #35190