diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc new file mode 100644 index 0000000000000..40e8303c3ba1d --- /dev/null +++ b/docs/reference/frozen-indices.asciidoc @@ -0,0 +1,82 @@ +[role="xpack"] +[testenv="basic"] +[[frozen-indices]] += Frozen Indices + +Elasticsearch indices can require a significant amount of memory available in order to be open and searchable. Yet, not all indices need +to be writable at the same time and have different access patters over time. For example indices in the time-series or logging use-case +are unlikely to be queried once they age out but still need to be kept around for retention policy purposes. + +In order to keep indices around for a longer time period but at the same time reducing their hardware requirements they can be transitioned +into a frozen state. Once an index is frozen, all it's transient shard memory aside of mappings and and necessary in-memory structures for +analysis is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is +frozen their are read-only and require their data-structures required for query execution to be loaded on demand. A search request that hits +one or more shards from frozen indices will execute it's search through a throttling component that ensures that we never search more than +`N` (`1` by default) searches concurrently. This protects nodes from exceeding the available memory due to incoming search requests. + +In contrast to ordinary open indices, frozen indices are expected to execute slow and are not designed for high query load. Parallelism is +gained only on a per-node level and loading data-structures on demand is expected to be one or more orders of a magnitude slower than query +execution on a per shard level. Depending on the index and it's data executing a search on a frozen index is expected to be in the seconds +or even minutes range compared to milliseconds on a non-frozen index. + +== Best Practices + +Since frozen indices provide a much higher disk to heap ratio for the expense of latency it is recommended to allocate frozen indices on +dedicated nodes to prevent searches on frozen indices influencing traffic on low latency nodes. There is a significant overhead in loading +data-structures on demand which can cause page-faults and garbage collections further slowing down query execution. + +Since indices that are eligible for freezing are likely to not change in the future disk space can be optimized. Many strategies are +outlined in <>. + +== Freezing and unfreezing an index + +The freeze and unfreeze index APIs allow to freeze an index, and later on +unfreeze it. A frozen index has almost no overhead on the cluster (except +for maintaining its metadata in memory), and is blocked for write operations. +A frozen index can be unfrozen which will then go through the normal recovery process. + +The REST endpoint is `/{index}/_freeze` and `/{index}/_unfreeze`. For +example: + +[source,js] +-------------------------------------------------- +POST /my_index/_freeze + +POST /my_index/_unfreeze +-------------------------------------------------- +// CONSOLE +// TEST[s/^/PUT my_index\n/] + + +[IMPORTANT] +================================ + Freezing an index will close the index and reopen it within the same API call. This causes primaries to not be allocated for a short + amount of time and causes the cluster to go red until the primaries are allocated again. This limitation might be removed in the future +================================ + +== Searching a frozen index + +Frozen indices are throttled in order limit memory consumptions per node. The number of concurrently loaded frozen indices per node is +limited by the number of threads in the `search_throttled` <> which is `1` by default. At the same time, +search requests hitting may indices on a cluster due to concrete lists, expanded wildcards or a date pattern exclude frozen indices by +default to prevent accidental slowdowns when a frozen index is hit. To include frozen indices a search request must be executed with +`ignore_throttled=false`. + +[source,js] +-------------------------------------------------- +GET /twitter/_search?q=user:kimchy&ignore_throttled=false +-------------------------------------------------- +// CONSOLE +// TEST[setup:twitter] + +[IMPORTANT] +================================ +While frozen indices are slow to search, they offer an efficient pre-filter phase. The request parameter `pre_filter_shard_size` specifies +a threshold that enforces a round-trip to filter search shards based on query rewriting if the number of shards the search +request expands to exceeds the threshold. This filter phase can limit the number of shards significantly if for instance a shard can not +match any documents based on it's rewrite method ie. if date filters are mandatory to match but the shard bounds and the query are disjoint. +The default value for `pre_filter_shard_size` is `128` while for searching frozen indices it's recommended to set it to `1`. There is no +significant overhead associated with this pre-filter phase. +================================ + + diff --git a/docs/reference/modules/threadpool.asciidoc b/docs/reference/modules/threadpool.asciidoc index 515959e4ea580..d0492e66aa29d 100644 --- a/docs/reference/modules/threadpool.asciidoc +++ b/docs/reference/modules/threadpool.asciidoc @@ -19,6 +19,10 @@ There are several thread pools, but the important ones include: `int((# of available_processors * 3) / 2) + 1`, and initial queue_size of `1000`. +`search_throttled`:: + For count/search/suggest/get operations on `search_throttled indices`. Thread pool type is + `fixed_auto_queue_size` with a size of `1`, and initial queue_size of `100`. + `get`:: For get operations. Thread pool type is `fixed` with a size of `# of available processors`, diff --git a/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexClusterStateUpdateRequest.java b/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexClusterStateUpdateRequest.java index ea3abe5e21a54..4062393167b79 100644 --- a/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexClusterStateUpdateRequest.java +++ b/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexClusterStateUpdateRequest.java @@ -28,7 +28,7 @@ public class OpenIndexClusterStateUpdateRequest extends IndicesClusterStateUpdat private ActiveShardCount waitForActiveShards = ActiveShardCount.DEFAULT; - OpenIndexClusterStateUpdateRequest() { + public OpenIndexClusterStateUpdateRequest() { } diff --git a/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexResponse.java b/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexResponse.java index 97db91f8973f4..6ff03cb5291ea 100644 --- a/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexResponse.java +++ b/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexResponse.java @@ -40,10 +40,10 @@ public class OpenIndexResponse extends ShardsAcknowledgedResponse { declareAcknowledgedAndShardsAcknowledgedFields(PARSER); } - OpenIndexResponse() { + public OpenIndexResponse() { } - OpenIndexResponse(boolean acknowledged, boolean shardsAcknowledged) { + public OpenIndexResponse(boolean acknowledged, boolean shardsAcknowledged) { super(acknowledged, shardsAcknowledged); } diff --git a/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java b/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java index 38d83b398856e..b6bb1fd058064 100644 --- a/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java +++ b/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java @@ -100,46 +100,50 @@ protected ClusterStateUpdateResponse newResponse(boolean acknowledged) { @Override public ClusterState execute(ClusterState currentState) { - Set indicesToClose = new HashSet<>(); - for (Index index : request.indices()) { - final IndexMetaData indexMetaData = currentState.metaData().getIndexSafe(index); - if (indexMetaData.getState() != IndexMetaData.State.CLOSE) { - indicesToClose.add(indexMetaData); - } - } + return closeIndex(currentState, request.indices(), indicesAsString); + } + }); + } - if (indicesToClose.isEmpty()) { - return currentState; - } + public ClusterState closeIndex(ClusterState currentState, final Index[] indices, String indicesAsString) { + Set indicesToClose = new HashSet<>(); + for (Index index : indices) { + final IndexMetaData indexMetaData = currentState.metaData().getIndexSafe(index); + if (indexMetaData.getState() != IndexMetaData.State.CLOSE) { + indicesToClose.add(indexMetaData); + } + } - // Check if index closing conflicts with any running restores - RestoreService.checkIndexClosing(currentState, indicesToClose); - // Check if index closing conflicts with any running snapshots - SnapshotsService.checkIndexClosing(currentState, indicesToClose); - logger.info("closing indices [{}]", indicesAsString); + if (indicesToClose.isEmpty()) { + return currentState; + } - MetaData.Builder mdBuilder = MetaData.builder(currentState.metaData()); - ClusterBlocks.Builder blocksBuilder = ClusterBlocks.builder() - .blocks(currentState.blocks()); - for (IndexMetaData openIndexMetadata : indicesToClose) { - final String indexName = openIndexMetadata.getIndex().getName(); - mdBuilder.put(IndexMetaData.builder(openIndexMetadata).state(IndexMetaData.State.CLOSE)); - blocksBuilder.addIndexBlock(indexName, INDEX_CLOSED_BLOCK); - } + // Check if index closing conflicts with any running restores + RestoreService.checkIndexClosing(currentState, indicesToClose); + // Check if index closing conflicts with any running snapshots + SnapshotsService.checkIndexClosing(currentState, indicesToClose); + logger.info("closing indices [{}]", indicesAsString); + + MetaData.Builder mdBuilder = MetaData.builder(currentState.metaData()); + ClusterBlocks.Builder blocksBuilder = ClusterBlocks.builder() + .blocks(currentState.blocks()); + for (IndexMetaData openIndexMetadata : indicesToClose) { + final String indexName = openIndexMetadata.getIndex().getName(); + mdBuilder.put(IndexMetaData.builder(openIndexMetadata).state(IndexMetaData.State.CLOSE)); + blocksBuilder.addIndexBlock(indexName, INDEX_CLOSED_BLOCK); + } - ClusterState updatedState = ClusterState.builder(currentState).metaData(mdBuilder).blocks(blocksBuilder).build(); + ClusterState updatedState = ClusterState.builder(currentState).metaData(mdBuilder).blocks(blocksBuilder).build(); - RoutingTable.Builder rtBuilder = RoutingTable.builder(currentState.routingTable()); - for (IndexMetaData index : indicesToClose) { - rtBuilder.remove(index.getIndex().getName()); - } + RoutingTable.Builder rtBuilder = RoutingTable.builder(currentState.routingTable()); + for (IndexMetaData index : indicesToClose) { + rtBuilder.remove(index.getIndex().getName()); + } - //no explicit wait for other nodes needed as we use AckedClusterStateUpdateTask - return allocationService.reroute( - ClusterState.builder(updatedState).routingTable(rtBuilder.build()).build(), - "indices closed [" + indicesAsString + "]"); - } - }); + //no explicit wait for other nodes needed as we use AckedClusterStateUpdateTask + return allocationService.reroute( + ClusterState.builder(updatedState).routingTable(rtBuilder.build()).build(), + "indices closed [" + indicesAsString + "]"); } public void openIndex(final OpenIndexClusterStateUpdateRequest request, diff --git a/server/src/main/java/org/elasticsearch/index/engine/ReadOnlyEngine.java b/server/src/main/java/org/elasticsearch/index/engine/ReadOnlyEngine.java index fc4b0632c8076..0ebd848859cf3 100644 --- a/server/src/main/java/org/elasticsearch/index/engine/ReadOnlyEngine.java +++ b/server/src/main/java/org/elasticsearch/index/engine/ReadOnlyEngine.java @@ -20,6 +20,7 @@ import org.apache.lucene.index.DirectoryReader; import org.apache.lucene.index.IndexCommit; +import org.apache.lucene.index.IndexReader; import org.apache.lucene.index.IndexWriter; import org.apache.lucene.index.SegmentCommitInfo; import org.apache.lucene.index.SegmentInfos; @@ -66,7 +67,7 @@ public class ReadOnlyEngine extends Engine { private final IndexCommit indexCommit; private final Lock indexWriterLock; private final DocsStats docsStats; - protected final RamAccountingSearcherFactory searcherFactory; + private final RamAccountingSearcherFactory searcherFactory; /** * Creates a new ReadOnlyEngine. This ctor can also be used to open a read-only engine on top of an already opened @@ -414,4 +415,8 @@ public void updateMaxUnsafeAutoIdTimestamp(long newTimestamp) { public void initializeMaxSeqNoOfUpdatesOrDeletes() { advanceMaxSeqNoOfUpdatesOrDeletes(seqNoStats.getMaxSeqNo()); } + + protected void processReaders(IndexReader reader, IndexReader previousReader) { + searcherFactory.processReaders(reader, previousReader); + } } diff --git a/test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTestCase.java b/test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTestCase.java index c9ef79720a29e..4199d414bf345 100644 --- a/test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTestCase.java +++ b/test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTestCase.java @@ -389,6 +389,18 @@ protected IndexShard reinitShard(IndexShard current, IndexingOperationListener.. * @param listeners new listerns to use for the newly created shard */ protected IndexShard reinitShard(IndexShard current, ShardRouting routing, IndexingOperationListener... listeners) throws IOException { + return reinitShard(current, routing, current.engineFactory, listeners); + } + + /** + * Takes an existing shard, closes it and starts a new initialing shard at the same location + * + * @param routing the shard routing to use for the newly created shard. + * @param listeners new listerns to use for the newly created shard + * @param engineFactory the engine factory for the new shard + */ + protected IndexShard reinitShard(IndexShard current, ShardRouting routing, EngineFactory engineFactory, + IndexingOperationListener... listeners) throws IOException { closeShards(current); return newShard( routing, @@ -396,7 +408,7 @@ protected IndexShard reinitShard(IndexShard current, ShardRouting routing, Index current.indexSettings().getIndexMetaData(), null, null, - current.engineFactory, + engineFactory, current.getGlobalCheckpointSyncer(), EMPTY_EVENT_LISTENER, listeners); } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java index ee1135ab53fe4..5a496a2979a5b 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java @@ -158,7 +158,7 @@ private synchronized DirectoryReader getOrOpenReader() throws IOException { listeners.beforeRefresh(); } reader = DirectoryReader.open(engineConfig.getStore().directory()); - searcherFactory.processReaders(reader, null); + processReaders(reader, null); reader = lastOpenedReader = wrapReader(reader, Function.identity()); reader.getReaderCacheHelper().addClosedListener(this::onReaderClosed); for (ReferenceManager.RefreshListener listeners : config ().getInternalRefreshListener()) { diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClient.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClient.java index 75b5fa05edbc7..e6cd2ed176c9c 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClient.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClient.java @@ -6,13 +6,15 @@ package org.elasticsearch.xpack.core; import org.elasticsearch.action.ActionListener; -import org.elasticsearch.action.support.master.AcknowledgedResponse; +import org.elasticsearch.action.support.PlainActionFuture; import org.elasticsearch.client.Client; import org.elasticsearch.common.settings.SecureString; import org.elasticsearch.license.LicensingClient; import org.elasticsearch.protocol.xpack.XPackInfoRequest; import org.elasticsearch.protocol.xpack.XPackInfoResponse; -import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction.FreezeIndexAction; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction.FreezeRequest; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction.FreezeResponse; import org.elasticsearch.xpack.core.action.XPackInfoAction; import org.elasticsearch.xpack.core.action.XPackInfoRequestBuilder; import org.elasticsearch.xpack.core.ccr.client.CcrClient; @@ -25,6 +27,7 @@ import java.util.Collections; import java.util.Map; import java.util.Objects; +import java.util.concurrent.ExecutionException; import static org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken.BASIC_AUTH_HEADER; import static org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken.basicAuthHeaderValue; @@ -106,7 +109,20 @@ public void info(XPackInfoRequest request, ActionListener lis client.execute(XPackInfoAction.INSTANCE, request, listener); } - public void freeze(TransportFreezeIndexAction.FreezeRequest request, ActionListener listener) { - client.execute(TransportFreezeIndexAction.FreezeIndexAction.INSTANCE, request, listener); + /** + * Freezes or unfreeze one or more indices + */ + public void freeze(FreezeRequest request, ActionListener listener) { + client.execute(FreezeIndexAction.INSTANCE, request, listener); + } + + /** + * Freeze or unfreeze one or more indices + */ + public FreezeResponse freeze(FreezeRequest request) + throws ExecutionException, InterruptedException { + PlainActionFuture future = new PlainActionFuture<>(); + freeze(request, future); + return future.get(); } } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java index b774b2990e096..283b059e030f9 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java @@ -37,6 +37,7 @@ import org.elasticsearch.tasks.Task; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.transport.Transport; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction; import org.elasticsearch.xpack.core.action.XPackInfoAction; import org.elasticsearch.xpack.core.action.XPackUsageAction; import org.elasticsearch.xpack.core.beats.BeatsFeatureSetUsage; @@ -344,7 +345,9 @@ public List> getClientActions() { ExplainLifecycleAction.INSTANCE, RemoveIndexLifecyclePolicyAction.INSTANCE, MoveToStepAction.INSTANCE, - RetryAction.INSTANCE + RetryAction.INSTANCE, + // Frozen indices + TransportFreezeIndexAction.FreezeIndexAction.INSTANCE ); } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackPlugin.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackPlugin.java index ae68252a941a0..bc861b3904f96 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackPlugin.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackPlugin.java @@ -63,6 +63,7 @@ import org.elasticsearch.xpack.core.action.XPackInfoAction; import org.elasticsearch.xpack.core.action.XPackUsageAction; import org.elasticsearch.xpack.core.ml.MlMetadata; +import org.elasticsearch.xpack.core.rest.action.RestFreezeIndexAction; import org.elasticsearch.xpack.core.rest.action.RestXPackInfoAction; import org.elasticsearch.xpack.core.rest.action.RestXPackUsageAction; import org.elasticsearch.xpack.core.security.authc.TokenMetaData; @@ -297,6 +298,7 @@ public List getRestHandlers(Settings settings, RestController restC List handlers = new ArrayList<>(); handlers.add(new RestXPackInfoAction(settings, restController)); handlers.add(new RestXPackUsageAction(settings, restController)); + handlers.add(new RestFreezeIndexAction(settings, restController)); handlers.addAll(licensing.getRestHandlers(settings, restController, clusterSettings, indexScopedSettings, settingsFilter, indexNameExpressionResolver, nodesInCluster)); return handlers; diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java index 693c2b4395f02..05ab20800a9e1 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java @@ -10,7 +10,10 @@ import org.elasticsearch.action.ActionListener; import org.elasticsearch.action.ActionRequestValidationException; import org.elasticsearch.action.IndicesRequest; +import org.elasticsearch.action.admin.indices.open.OpenIndexClusterStateUpdateRequest; +import org.elasticsearch.action.admin.indices.open.OpenIndexResponse; import org.elasticsearch.action.support.ActionFilters; +import org.elasticsearch.action.support.ActiveShardCount; import org.elasticsearch.action.support.DestructiveOperations; import org.elasticsearch.action.support.IndicesOptions; import org.elasticsearch.action.support.master.AcknowledgedRequest; @@ -18,12 +21,14 @@ import org.elasticsearch.action.support.master.TransportMasterNodeAction; import org.elasticsearch.cluster.AckedClusterStateUpdateTask; import org.elasticsearch.cluster.ClusterState; +import org.elasticsearch.cluster.ack.OpenIndexClusterStateUpdateResponse; import org.elasticsearch.cluster.block.ClusterBlockException; import org.elasticsearch.cluster.block.ClusterBlockLevel; import org.elasticsearch.cluster.block.ClusterBlocks; import org.elasticsearch.cluster.metadata.IndexMetaData; import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver; import org.elasticsearch.cluster.metadata.MetaData; +import org.elasticsearch.cluster.metadata.MetaDataIndexStateService; import org.elasticsearch.cluster.service.ClusterService; import org.elasticsearch.common.Priority; import org.elasticsearch.common.inject.Inject; @@ -39,22 +44,27 @@ import org.elasticsearch.transport.TransportService; import java.io.IOException; +import java.util.ArrayList; +import java.util.List; import static org.elasticsearch.action.ValidateActions.addValidationError; public final class TransportFreezeIndexAction extends - TransportMasterNodeAction { + TransportMasterNodeAction { private final DestructiveOperations destructiveOperations; + private final MetaDataIndexStateService indexStateService; @Inject - public TransportFreezeIndexAction(TransportService transportService, ClusterService clusterService, + public TransportFreezeIndexAction(MetaDataIndexStateService indexStateService, TransportService transportService, + ClusterService clusterService, ThreadPool threadPool, ActionFilters actionFilters, IndexNameExpressionResolver indexNameExpressionResolver, DestructiveOperations destructiveOperations) { super(FreezeIndexAction.NAME, transportService, clusterService, threadPool, actionFilters, indexNameExpressionResolver, FreezeRequest::new); this.destructiveOperations = destructiveOperations; + this.indexStateService = indexStateService; } @Override protected String executor() { @@ -62,27 +72,74 @@ protected String executor() { } @Override - protected void doExecute(Task task, FreezeRequest request, ActionListener listener) { + protected void doExecute(Task task, FreezeRequest request, ActionListener listener) { destructiveOperations.failDestructive(request.indices()); super.doExecute(task, request, listener); } @Override - protected AcknowledgedResponse newResponse() { - return new AcknowledgedResponse(); + protected FreezeResponse newResponse() { + return new FreezeResponse(); + } + + private Index[] resolveIndices(FreezeRequest request, ClusterState state) { + List indices = new ArrayList<>(); + for (Index index : indexNameExpressionResolver.concreteIndices(state, request)) { + IndexMetaData metaData = state.metaData().index(index); + Settings settings = metaData.getSettings(); + // only unfreeze if we are frozen and only freeze if we are not frozen already. + // this prevents all indices that are already frozen that match a pattern to + // go through the cycles again. + if ((request.freeze() && FrozenEngine.INDEX_FROZEN.get(settings) == false) || + (request.freeze() == false && FrozenEngine.INDEX_FROZEN.get(settings))) { + indices.add(index); + } + } + return indices.toArray(new Index[0]); } @Override - protected void masterOperation(FreezeRequest request, ClusterState state, ActionListener listener) { - final Index[] concreteIndices = indexNameExpressionResolver.concreteIndices(state, request); + protected void masterOperation(FreezeRequest request, ClusterState state, ActionListener listener) { + final Index[] concreteIndices = resolveIndices(request, state); if (concreteIndices == null || concreteIndices.length == 0) { - throw new ResourceNotFoundException("index not found"); + throw new ResourceNotFoundException("no index found to " + (request.freeze() ? "freeze" : "unfreeze")); } - clusterService.submitStateUpdateTask("toggle-frozen-settings", - new AckedClusterStateUpdateTask(Priority.URGENT, request, listener) { + new AckedClusterStateUpdateTask(Priority.URGENT, request, new ActionListener() { + @Override + public void onResponse(AcknowledgedResponse acknowledgedResponse) { + OpenIndexClusterStateUpdateRequest updateRequest = new OpenIndexClusterStateUpdateRequest() + .ackTimeout(request.timeout()).masterNodeTimeout(request.masterNodeTimeout()) + .indices(concreteIndices).waitForActiveShards(request.waitForActiveShards()); + indexStateService.openIndex(updateRequest, new ActionListener() { + @Override + public void onResponse(OpenIndexClusterStateUpdateResponse openIndexClusterStateUpdateResponse) { + listener.onResponse(new FreezeResponse(openIndexClusterStateUpdateResponse.isAcknowledged(), + openIndexClusterStateUpdateResponse.isShardsAcknowledged())); + } + + @Override + public void onFailure(Exception e) { + listener.onFailure(e); + } + }); + } + + @Override + public void onFailure(Exception e) { + listener.onFailure(e); + } + }) { @Override - public ClusterState execute(final ClusterState currentState) { + public ClusterState execute(ClusterState currentState) { + List toClose = new ArrayList<>(); + for (Index index : concreteIndices) { + IndexMetaData metaData = currentState.metaData().index(index); + if (metaData.getState() != IndexMetaData.State.CLOSE) { + toClose.add(index); + } + } + currentState = indexStateService.closeIndex(currentState, toClose.toArray(new Index[0]), toClose.toString()); final MetaData.Builder builder = MetaData.builder(currentState.metaData()); ClusterBlocks.Builder blocks = ClusterBlocks.builder().blocks(currentState.blocks()); for (Index index : concreteIndices) { @@ -94,7 +151,6 @@ public ClusterState execute(final ClusterState currentState) { final Settings.Builder settingsBuilder = Settings.builder() .put(currentState.metaData().index(index).getSettings()) - .put("index.blocks.write", request.freeze()) .put(FrozenEngine.INDEX_FROZEN.getKey(), request.freeze()) .put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), request.freeze()); if (request.freeze()) { @@ -110,6 +166,7 @@ public ClusterState execute(final ClusterState currentState) { @Override protected AcknowledgedResponse newResponse(boolean acknowledged) { + return new AcknowledgedResponse(acknowledged); } }); @@ -121,7 +178,17 @@ protected ClusterBlockException checkBlock(FreezeRequest request, ClusterState s indexNameExpressionResolver.concreteIndexNames(state, request)); } - public static class FreezeIndexAction extends Action { + public static class FreezeResponse extends OpenIndexResponse { + public FreezeResponse() { + super(); + } + + public FreezeResponse(boolean acknowledged, boolean shardsAcknowledged) { + super(acknowledged, shardsAcknowledged); + } + } + + public static class FreezeIndexAction extends Action { public static final FreezeIndexAction INSTANCE = new FreezeIndexAction(); public static final String NAME = "indices:admin/freeze"; @@ -131,8 +198,8 @@ private FreezeIndexAction() { } @Override - public AcknowledgedResponse newResponse() { - return new AcknowledgedResponse(); + public FreezeResponse newResponse() { + return new FreezeResponse(); } } @@ -140,7 +207,8 @@ public static class FreezeRequest extends AcknowledgedRequest implements IndicesRequest.Replaceable { private String[] indices; private boolean freeze = true; - private IndicesOptions indicesOptions = IndicesOptions.fromOptions(false, false, false, true); + private IndicesOptions indicesOptions = IndicesOptions.strictExpandOpen(); + private ActiveShardCount waitForActiveShards = ActiveShardCount.DEFAULT; public FreezeRequest(String... indices) { this.indices = indices; @@ -155,8 +223,9 @@ public ActionRequestValidationException validate() { return validationException; } - public void setFreeze(boolean freeze) { + public FreezeRequest setFreeze(boolean freeze) { this.freeze = freeze; + return this; } public boolean freeze() { @@ -169,6 +238,7 @@ public void readFrom(StreamInput in) throws IOException { indicesOptions = IndicesOptions.readIndicesOptions(in); indices = in.readStringArray(); freeze = in.readBoolean(); + waitForActiveShards = ActiveShardCount.readFrom(in); } @Override @@ -177,6 +247,7 @@ public void writeTo(StreamOutput out) throws IOException { indicesOptions.writeIndicesOptions(out); out.writeStringArray(indices); out.writeBoolean(freeze); + waitForActiveShards.writeTo(out); } /** @@ -215,5 +286,28 @@ public IndicesRequest indices(String... indices) { this.indices = indices; return this; } + + public ActiveShardCount waitForActiveShards() { + return waitForActiveShards; + } + + /** + * Sets the number of shard copies that should be active for indices opening to return. + * Defaults to {@link ActiveShardCount#DEFAULT}, which will wait for one shard copy + * (the primary) to become active. Set this value to {@link ActiveShardCount#ALL} to + * wait for all shards (primary and all replicas) to be active before returning. + * Otherwise, use {@link ActiveShardCount#from(int)} to set this value to any + * non-negative integer, up to the number of copies per shard (number of replicas + 1), + * to wait for the desired amount of shard copies to become active before returning. + * Indices opening will only wait up until the timeout value for the number of shard copies + * to be active before returning. Check {@link OpenIndexResponse#isShardsAcknowledged()} to + * determine if the requisite shard copies were all started before returning or timing out. + * + * @param waitForActiveShards number of active shard copies to wait on + */ + public FreezeRequest waitForActiveShards(ActiveShardCount waitForActiveShards) { + this.waitForActiveShards = waitForActiveShards; + return this; + } } } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rest/action/RestFreezeIndexAction.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rest/action/RestFreezeIndexAction.java new file mode 100644 index 0000000000000..9604cdd8b3183 --- /dev/null +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rest/action/RestFreezeIndexAction.java @@ -0,0 +1,46 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.xpack.core.rest.action; + +import org.elasticsearch.action.support.ActiveShardCount; +import org.elasticsearch.action.support.IndicesOptions; +import org.elasticsearch.common.Strings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.rest.RestController; +import org.elasticsearch.rest.RestRequest; +import org.elasticsearch.rest.action.RestToXContentListener; +import org.elasticsearch.xpack.core.XPackClient; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction; +import org.elasticsearch.xpack.core.rest.XPackRestHandler; + +public final class RestFreezeIndexAction extends XPackRestHandler { + public RestFreezeIndexAction(Settings settings, RestController controller) { + super(settings); + controller.registerHandler(RestRequest.Method.POST, "/{index}/_freeze", this); + controller.registerHandler(RestRequest.Method.POST, "/{index}/_unfreeze", this); + } + + @Override + protected RestChannelConsumer doPrepareRequest(RestRequest request, XPackClient client) { + boolean freeze = request.path().endsWith("/_freeze"); + TransportFreezeIndexAction.FreezeRequest freezeRequest = + new TransportFreezeIndexAction.FreezeRequest(Strings.splitStringByCommaToArray(request.param("index"))); + freezeRequest.timeout(request.paramAsTime("timeout", freezeRequest.timeout())); + freezeRequest.masterNodeTimeout(request.paramAsTime("master_timeout", freezeRequest.masterNodeTimeout())); + freezeRequest.indicesOptions(IndicesOptions.fromRequest(request, freezeRequest.indicesOptions())); + String waitForActiveShards = request.param("wait_for_active_shards"); + if (waitForActiveShards != null) { + freezeRequest.waitForActiveShards(ActiveShardCount.parseString(waitForActiveShards)); + } + freezeRequest.setFreeze(freeze); + return channel -> client.freeze(freezeRequest, new RestToXContentListener<>(channel)); + } + + @Override + public String getName() { + return "freeze_index"; + } +} diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java new file mode 100644 index 0000000000000..22c646383776c --- /dev/null +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java @@ -0,0 +1,43 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.index.engine; + +import org.elasticsearch.cluster.routing.RecoverySource; +import org.elasticsearch.cluster.routing.ShardRouting; +import org.elasticsearch.cluster.routing.ShardRoutingHelper; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase; +import org.elasticsearch.index.shard.IndexShard; + +import java.io.IOException; + +import static org.hamcrest.Matchers.equalTo; + +public class FrozenIndexRecoveryTests extends ESIndexLevelReplicationTestCase { + + /** + * Make sure we can recover from a frozen engine + */ + public void testRecoverFromFrozenPrimary() throws IOException { + IndexShard indexShard = newStartedShard(true); + indexDoc(indexShard, "_doc", "1"); + indexDoc(indexShard, "_doc", "2"); + indexDoc(indexShard, "_doc", "3"); + indexShard.close("test", true); + final ShardRouting shardRouting = indexShard.routingEntry(); + IndexShard frozenShard = reinitShard(indexShard, ShardRoutingHelper.initWithSameId(shardRouting, + shardRouting.primary() ? RecoverySource.ExistingStoreRecoverySource.INSTANCE : RecoverySource.PeerRecoverySource.INSTANCE + ), FrozenEngine::new); + recoverShardFromStore(frozenShard); + assertThat(frozenShard.getMaxSeqNoOfUpdatesOrDeletes(), equalTo(frozenShard.seqNoStats().getMaxSeqNo())); + assertDocCount(frozenShard, 3); + + IndexShard replica = newShard(false, Settings.EMPTY, FrozenEngine::new); + recoverReplica(replica, frozenShard, true); + assertDocCount(replica, 3); + closeShards(frozenShard, replica); + } +} diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java index 5ef1bbd4fa553..4e98f7f3afe24 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java @@ -6,14 +6,13 @@ package org.elasticsearch.index.engine; import org.elasticsearch.action.ActionListener; +import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse; import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse; import org.elasticsearch.action.search.SearchResponse; import org.elasticsearch.action.search.SearchType; -import org.elasticsearch.action.support.ActiveShardCount; import org.elasticsearch.action.support.IndicesOptions; -import org.elasticsearch.action.support.PlainActionFuture; -import org.elasticsearch.action.support.master.AcknowledgedResponse; import org.elasticsearch.cluster.block.ClusterBlockException; +import org.elasticsearch.cluster.metadata.IndexMetaData; import org.elasticsearch.common.Strings; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.unit.TimeValue; @@ -58,14 +57,8 @@ public void testCloseFreezeAndOpen() throws ExecutionException, InterruptedExcep client().prepareIndex("index", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); client().prepareIndex("index", "_doc", "2").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); client().prepareIndex("index", "_doc", "3").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); - - client().admin().indices().prepareFlush("index").get(); - client().admin().indices().prepareClose("index").get(); XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"), future); - assertAcked(future.get()); - assertAcked(client().admin().indices().prepareOpen("index")); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"))); expectThrows(ClusterBlockException.class, () -> client().prepareIndex("index", "_doc", "4").setSource("field", "value") .setRefreshPolicy(IMMEDIATE).get()); IndicesService indexServices = getInstanceFromNode(IndicesService.class); @@ -101,7 +94,7 @@ public void testCloseFreezeAndOpen() throws ExecutionException, InterruptedExcep } while (searchResponse.getHits().getHits().length > 0); } - public void testSearchAndGetAPIsAreThrottled() throws ExecutionException, InterruptedException, IOException { + public void testSearchAndGetAPIsAreThrottled() throws InterruptedException, IOException, ExecutionException { XContentBuilder mapping = XContentFactory.jsonBuilder().startObject().startObject("_doc") .startObject("properties").startObject("field").field("type", "text").field("term_vector", "with_positions_offsets_payloads") .endObject().endObject() @@ -110,15 +103,8 @@ public void testSearchAndGetAPIsAreThrottled() throws ExecutionException, Interr for (int i = 0; i < 10; i++) { client().prepareIndex("index", "_doc", "" + i).setSource("field", "foo bar baz").get(); } - client().admin().indices().prepareFlush("index").get(); - client().admin().indices().prepareClose("index").get(); XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - TransportFreezeIndexAction.FreezeRequest request = - new TransportFreezeIndexAction.FreezeRequest("index"); - xPackClient.freeze(request, future); - assertAcked(future.get()); - assertAcked(client().admin().indices().prepareOpen("index")); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"))); int numRequests = randomIntBetween(20, 50); CountDownLatch latch = new CountDownLatch(numRequests); ActionListener listener = ActionListener.wrap(latch::countDown); @@ -152,21 +138,17 @@ public void testSearchAndGetAPIsAreThrottled() throws ExecutionException, Interr assertEquals(numRefreshes, index.getTotal().refresh.getTotal()); } - public void testFreezeAndUnfreeze() throws ExecutionException, InterruptedException { + public void testFreezeAndUnfreeze() throws InterruptedException, ExecutionException { createIndex("index", Settings.builder().put("index.number_of_shards", 2).build()); client().prepareIndex("index", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); client().prepareIndex("index", "_doc", "2").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); client().prepareIndex("index", "_doc", "3").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); - - client().admin().indices().prepareFlush("index").get(); - client().admin().indices().prepareClose("index").get(); + if (randomBoolean()) { + // sometimes close it + assertAcked(client().admin().indices().prepareClose("index").get()); + } XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - TransportFreezeIndexAction.FreezeRequest request = - new TransportFreezeIndexAction.FreezeRequest("index"); - xPackClient.freeze(request, future); - assertAcked(future.get()); - assertAcked(client().admin().indices().prepareOpen("index")); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"))); { IndicesService indexServices = getInstanceFromNode(IndicesService.class); Index index = resolveIndex("index"); @@ -175,12 +157,7 @@ public void testFreezeAndUnfreeze() throws ExecutionException, InterruptedExcept IndexShard shard = indexService.getShard(0); assertEquals(0, shard.refreshStats().getTotal()); } - client().admin().indices().prepareClose("index").get(); - request.setFreeze(false); - PlainActionFuture future1= new PlainActionFuture<>(); - xPackClient.freeze(request, future1); - assertAcked(future1.get()); - assertAcked(client().admin().indices().prepareOpen("index")); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index").setFreeze(false))); { IndicesService indexServices = getInstanceFromNode(IndicesService.class); Index index = resolveIndex("index"); @@ -193,16 +170,61 @@ public void testFreezeAndUnfreeze() throws ExecutionException, InterruptedExcept client().prepareIndex("index", "_doc", "4").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); } - public void testIndexMustBeClosed() { + private void assertIndexFrozen(String idx) { + IndicesService indexServices = getInstanceFromNode(IndicesService.class); + Index index = resolveIndex(idx); + IndexService indexService = indexServices.indexServiceSafe(index); + assertTrue(indexService.getIndexSettings().isSearchThrottled()); + assertTrue(FrozenEngine.INDEX_FROZEN.get(indexService.getIndexSettings().getSettings())); + } + + public void testDoubleFreeze() throws ExecutionException, InterruptedException { createIndex("test-idx", Settings.builder().put("index.number_of_shards", 2).build()); XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - TransportFreezeIndexAction.FreezeRequest request = - new TransportFreezeIndexAction.FreezeRequest("test-idx"); - xPackClient.freeze(request, future); - ExecutionException executionException = expectThrows(ExecutionException.class, () -> future.get()); - assertThat(executionException.getCause(), Matchers.instanceOf(IllegalStateException.class)); - assertEquals("index [test-idx] is not closed", executionException.getCause().getMessage()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test-idx"))); + ExecutionException executionException = expectThrows(ExecutionException.class, + () -> xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test-idx"))); + assertEquals("no index found to freeze", executionException.getCause().getMessage()); + } + + public void testUnfreezeClosedIndices() throws ExecutionException, InterruptedException { + createIndex("idx", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("idx", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + createIndex("idx-closed", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("idx-closed", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx"))); + assertAcked(client().admin().indices().prepareClose("idx-closed").get()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx*").setFreeze(false) + .indicesOptions(IndicesOptions.strictExpand()))); + ClusterStateResponse stateResponse = client().admin().cluster().prepareState().get(); + assertEquals(IndexMetaData.State.CLOSE, stateResponse.getState().getMetaData().index("idx-closed").getState()); + assertEquals(IndexMetaData.State.OPEN, stateResponse.getState().getMetaData().index("idx").getState()); + assertHitCount(client().prepareSearch().get(), 1L); + } + + public void testFreezePattern() throws ExecutionException, InterruptedException { + createIndex("test-idx", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("test-idx", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + createIndex("test-idx-1", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("test-idx-1", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test-idx"))); + assertIndexFrozen("test-idx"); + + IndicesStatsResponse index = client().admin().indices().prepareStats("test-idx").clear().setRefresh(true).get(); + assertEquals(0, index.getTotal().refresh.getTotal()); + assertHitCount(client().prepareSearch("test-idx").setIndicesOptions(IndicesOptions.STRICT_EXPAND_OPEN_FORBID_CLOSED).get(), 1); + index = client().admin().indices().prepareStats("test-idx").clear().setRefresh(true).get(); + assertEquals(1, index.getTotal().refresh.getTotal()); + + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test*"))); + assertIndexFrozen("test-idx"); + assertIndexFrozen("test-idx-1"); + index = client().admin().indices().prepareStats("test-idx").clear().setRefresh(true).get(); + assertEquals(1, index.getTotal().refresh.getTotal()); + index = client().admin().indices().prepareStats("test-idx-1").clear().setRefresh(true).get(); + assertEquals(0, index.getTotal().refresh.getTotal()); } public void testCanMatch() throws ExecutionException, InterruptedException, IOException { @@ -232,15 +254,9 @@ public void testCanMatch() throws ExecutionException, InterruptedException, IOEx Strings.EMPTY_ARRAY, false, new AliasFilter(null, Strings.EMPTY_ARRAY), 1f, true, null, null))); } - client().admin().indices().prepareFlush("index").get(); - client().admin().indices().prepareClose("index").get(); + XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - TransportFreezeIndexAction.FreezeRequest request = - new TransportFreezeIndexAction.FreezeRequest("index"); - xPackClient.freeze(request, future); - assertAcked(future.get()); - assertAcked(client().admin().indices().prepareOpen("index").setWaitForActiveShards(ActiveShardCount.DEFAULT)); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"))); { IndicesService indexServices = getInstanceFromNode(IndicesService.class); @@ -266,4 +282,41 @@ public void testCanMatch() throws ExecutionException, InterruptedException, IOEx assertEquals(0, response.getTotal().refresh.getTotal()); // never opened a reader } } + + public void testWriteToFrozenIndex() throws ExecutionException, InterruptedException { + createIndex("idx", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("idx", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx"))); + assertIndexFrozen("idx"); + expectThrows(ClusterBlockException.class, () -> + client().prepareIndex("idx", "_doc", "2").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get()); + } + + public void testIgnoreUnavailable() throws ExecutionException, InterruptedException { + createIndex("idx", Settings.builder().put("index.number_of_shards", 1).build()); + createIndex("idx-close", Settings.builder().put("index.number_of_shards", 1).build()); + assertAcked(client().admin().indices().prepareClose("idx-close")); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx*", "not_available") + .indicesOptions(IndicesOptions.fromParameters(null, "true", null, null, IndicesOptions.strictExpandOpen())))); + assertIndexFrozen("idx"); + assertEquals(IndexMetaData.State.CLOSE, + client().admin().cluster().prepareState().get().getState().metaData().index("idx-close").getState()); + } + + public void testUnfreezeClosedIndex() throws ExecutionException, InterruptedException { + createIndex("idx", Settings.builder().put("index.number_of_shards", 1).build()); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx"))); + assertAcked(client().admin().indices().prepareClose("idx")); + assertEquals(IndexMetaData.State.CLOSE, + client().admin().cluster().prepareState().get().getState().metaData().index("idx").getState()); + expectThrows(ExecutionException.class, + () -> xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("id*").setFreeze(false))); + // we don't resolve to closed indices + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx").setFreeze(false))); + assertEquals(IndexMetaData.State.OPEN, + client().admin().cluster().prepareState().get().getState().metaData().index("idx").getState()); + } } diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.freeze.json b/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.freeze.json new file mode 100644 index 0000000000000..a602f341e1524 --- /dev/null +++ b/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.freeze.json @@ -0,0 +1,48 @@ +{ + "indices.freeze": { + "documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/current/frozen.html", + "methods": [ "POST" ], + "url": { + "path": "/{index}/_freeze", + "paths": [ + "/{index}/_freeze" + ], + "parts": { + "index": { + "type": "string", + "required": true, + "description": "The name of the index to freeze" + } + }, + "params": { + "timeout": { + "type" : "time", + "description" : "Explicit operation timeout" + }, + "master_timeout": { + "type" : "time", + "description" : "Specify timeout for connection to master" + }, + "ignore_unavailable": { + "type" : "boolean", + "description" : "Whether specified concrete indices should be ignored when unavailable (missing or closed)" + }, + "allow_no_indices": { + "type" : "boolean", + "description" : "Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes `_all` string or when no indices have been specified)" + }, + "expand_wildcards": { + "type" : "enum", + "options" : ["open","closed","none","all"], + "default" : "closed", + "description" : "Whether to expand wildcard expression to concrete indices that are open, closed or both." + }, + "wait_for_active_shards": { + "type" : "string", + "description" : "Sets the number of active shards to wait for before the operation returns." + } + } + }, + "body": null + } +} diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.unfreeze.json b/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.unfreeze.json new file mode 100644 index 0000000000000..b10e869a95758 --- /dev/null +++ b/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.unfreeze.json @@ -0,0 +1,46 @@ +{ + "indices.unfreeze": { + "documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/current/frozen.html", + "methods": [ "POST" ], + "url": { + "path": "/{index}/_unfreeze", + "paths": [ "/{index}/_unfreeze" ], + "parts": { + "index": { + "type": "string", + "required": true, + "description": "The name of the index to unfreeze" + } + }, + "params": { + "timeout": { + "type" : "time", + "description" : "Explicit operation timeout" + }, + "master_timeout": { + "type" : "time", + "description" : "Specify timeout for connection to master" + }, + "ignore_unavailable": { + "type" : "boolean", + "description" : "Whether specified concrete indices should be ignored when unavailable (missing or closed)" + }, + "allow_no_indices": { + "type" : "boolean", + "description" : "Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes `_all` string or when no indices have been specified)" + }, + "expand_wildcards": { + "type" : "enum", + "options" : ["open","closed","none","all"], + "default" : "closed", + "description" : "Whether to expand wildcard expression to concrete indices that are open, closed or both." + }, + "wait_for_active_shards": { + "type" : "string", + "description" : "Sets the number of active shards to wait for before the operation returns." + } + } + }, + "body": null + } +} diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/test/indices.freeze/10_basic.yml b/x-pack/plugin/src/test/resources/rest-api-spec/test/indices.freeze/10_basic.yml new file mode 100644 index 0000000000000..8f79225ddc37d --- /dev/null +++ b/x-pack/plugin/src/test/resources/rest-api-spec/test/indices.freeze/10_basic.yml @@ -0,0 +1,128 @@ +--- +"Basic": + +- skip: + version: " - 6.99.99" + reason: types are required in requests before 7.0.0 + +- do: + index: + index: test + id: "1" + body: { "foo": "Hello: 1" } +- do: + index: + index: test + id: "2" + body: { "foo": "Hello: 2" } + +- do: + indices.freeze: + index: test + +- do: + search: + index: test + ignore_throttled: false + body: + query: + match: + foo: hello + +- match: {hits.total: 2} + +# unfreeze +- do: + indices.unfreeze: + index: test + +- do: + search: + index: _all + body: + query: + match: + foo: hello + +- match: {hits.total: 2} + +- do: + index: + index: test-01 + id: "1" + body: { "foo": "Hello: 01" } + + +- do: + indices.freeze: + index: test* + +- do: + search: + index: _all + ignore_throttled: false + body: + query: + match: + foo: hello + +- match: {hits.total: 3} + +- do: + search: + index: _all + body: + query: + match: + foo: hello + +- match: {hits.total: 0} + +--- +"Test index options": + +- skip: + version: " - 6.99.99" + reason: types are required in requests before 7.0.0 + +- do: + index: + index: test + id: "1" + body: { "foo": "Hello: 1" } + +- do: + index: + index: test-close + id: "1" + body: { "foo": "Hello: 1" } + +- do: + indices.close: + index: test-close + +- do: + indices.freeze: + index: test*,not_available + ignore_unavailable: true + +- do: + search: + index: _all + body: + query: + match: + foo: hello + +- match: {hits.total: 0} + +- do: + search: + index: _all + ignore_throttled: false + body: + query: + match: + foo: hello + +- match: {hits.total: 1}