Add setting to ignore dynamic fields when field limit is reached (#96235)

Adds a new `index.mapping.total_fields.ignore_dynamic_beyond_limit`
index setting.

When set to `true`, new fields are added to the mapping as long as the
field limit (`index.mapping.total_fields.limit`) is not exceeded. Fields
that would exceed the limit are not added to the mapping, similar to
`dynamic: false`.  Ignored fields are added to the `_ignored` metadata
field.

Relates to #89911

To make this easier to review, this is split into the following PRs: -
[x] #102915 - [x]
#102936 - [x]
#104769

Related but not a prerequisite: - [ ]
#102885

Author: felixbarny

docs/changelog/96235.yaml

@@ -0,0 +1,5 @@
+pr: 96235
+summary: Add `index.mapping.total_fields.ignore_dynamic_beyond_limit` setting to ignore dynamic fields when field limit is reached
+area: Mapping
+type: enhancement
+issues: []

docs/reference/mapping/fields/ignored-field.asciidoc

@@ -4,8 +4,11 @@
 The `_ignored` field indexes and stores the names of every field in a document
 that has been ignored when the document was indexed. This can, for example,
 be the case when the field was malformed and <<ignore-malformed,`ignore_malformed`>>
-was turned on, or when a `keyword` fields value exceeds its optional
-<<ignore-above,`ignore_above`>> setting.
+was turned on, when a `keyword` field's value exceeds its optional
+<<ignore-above,`ignore_above`>> setting, or when
+<<mapping-settings-limit,`index.mapping.total_fields.limit`>> has been reached and
+<<mapping-settings-limit,`index.mapping.total_fields.ignore_dynamic_beyond_limit`>>
+is set to `true`.
 
 This field is searchable with <<query-dsl-term-query,`term`>>,
 <<query-dsl-terms-query,`terms`>> and <<query-dsl-exists-query,`exists`>>

docs/reference/mapping/mapping-settings-limit.asciidoc

@@ -20,9 +20,18 @@ limits the maximum number of clauses in a query.
 +
 [TIP]
 ====
-If your field mappings contain a large, arbitrary set of keys, consider using the <<flattened,flattened>> data type.
+If your field mappings contain a large, arbitrary set of keys, consider using the <<flattened,flattened>> data type,
+or setting the index setting `index.mapping.total_fields.ignore_dynamic_beyond_limit` to `true`.
 ====
 
+`index.mapping.total_fields.ignore_dynamic_beyond_limit`::
+    This setting determines what happens when a dynamically mapped field would exceed the total fields limit.
+    When set to `false` (the default), the index request of the document that tries to add a dynamic field to the mapping will fail with the message `Limit of total fields [X] has been exceeded`.
+    When set to `true`, the index request will not fail.
+    Instead, fields that would exceed the limit are not added to the mapping, similar to <<dynamic, `dynamic: false`>>.
+    The fields that were not added to the mapping will be added to the <<mapping-ignored-field, `_ignored` field>>.
+    The default value is `false`.
+
 `index.mapping.depth.limit`::
     The maximum depth for a field, which is measured as the number of inner
     objects. For instance, if all fields are defined at the root object level,

docs/reference/mapping/params/dynamic.asciidoc

@@ -90,3 +90,10 @@ accepts the following parameters:
             to the mapping, and new fields must be added explicitly.
 `strict`::  If new fields are detected, an exception is thrown and the document
             is rejected. New fields must be explicitly added to the mapping.
+
+[[dynamic-field-limit]]
+==== Behavior when reaching the field limit
+Setting `dynamic` to either `true` or `runtime` will only add dynamic fields until <<mapping-settings-limit,`index.mapping.total_fields.limit`>> is reached.
+By default, index requests for documents that would exceed the field limit will fail,
+unless <<mapping-settings-limit,`index.mapping.total_fields.ignore_dynamic_beyond_limit`>> is set to `true`.
+In that case, ignored fields are added to the <<mapping-ignored-field, `_ignored` metadata field>>.

docs/reference/troubleshooting/common-issues/mapping-explosion.asciidoc

@@ -41,13 +41,16 @@ timing out in the browser's Developer Tools Network tab.
 doesn't normally cause problems unless it's combined with overriding
 <<mapping-settings-limit,`index.mapping.total_fields.limit`>>. The
 default `1000` limit is considered generous, though overriding to `10000`
-doesn't cause noticable impact depending on use case. However, to give
+doesn't cause noticeable impact depending on use case. However, to give
 a bad example, overriding to `100000` and this limit being hit
 by mapping totals would usually have strong performance implications.
 
 If your index mapped fields expect to contain a large, arbitrary set of
 keys, you may instead consider:
 
+* Setting <<mapping-settings-limit,`index.mapping.total_fields.ignore_dynamic_beyond_limit`>> to `true`.
+Instead of rejecting documents that exceed the field limit, this will ignore dynamic fields once the limit is reached.
+
 * Using the <<flattened,flattened>> data type. Please note,
 however, that flattened objects is link:https://github.com/elastic/kibana/issues/25820[not fully supported in {kib}] yet. For example, this could apply to sub-mappings like { `host.name` ,
 `host.os`, `host.version` }. Desired fields are still accessed by

server/src/internalClusterTest/java/org/elasticsearch/index/mapper/DynamicMappingIT.java

@@ -151,7 +151,8 @@ private static ClusterState applyRequest(
                 MapperService.mergeMappings(
                     mapperService.documentMapper(),
                     mapping,
-                    request.autoUpdate() ? MergeReason.MAPPING_AUTO_UPDATE : MergeReason.MAPPING_UPDATE
+                    request.autoUpdate() ? MergeReason.MAPPING_AUTO_UPDATE : MergeReason.MAPPING_UPDATE,
+                    mapperService.getIndexSettings()
                 );
             }
             Metadata.Builder builder = Metadata.builder(metadata);

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataMappingService.java

@@ -151,6 +151,7 @@ public final class IndexScopedSettings extends AbstractScopedSettings {
         Store.INDEX_STORE_STATS_REFRESH_INTERVAL_SETTING,
         MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING,
         MapperService.INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING,
+        MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING,
         MapperService.INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING,
         MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING,
         MapperService.INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING,

server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java

@@ -42,6 +42,7 @@
 import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING;
 import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING;
 import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_FIELD_NAME_LENGTH_LIMIT_SETTING;
+import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING;
 import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING;
 import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING;
 import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING;
@@ -753,6 +754,7 @@ private void setRetentionLeaseMillis(final TimeValue retentionLease) {
     private volatile long mappingNestedFieldsLimit;
     private volatile long mappingNestedDocsLimit;
     private volatile long mappingTotalFieldsLimit;
+    private volatile boolean ignoreDynamicFieldsBeyondLimit;
     private volatile long mappingDepthLimit;
     private volatile long mappingFieldNameLengthLimit;
     private volatile long mappingDimensionFieldsLimit;
@@ -897,6 +899,7 @@ public IndexSettings(final IndexMetadata indexMetadata, final Settings nodeSetti
         mappingNestedFieldsLimit = scopedSettings.get(INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING);
         mappingNestedDocsLimit = scopedSettings.get(INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING);
         mappingTotalFieldsLimit = scopedSettings.get(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING);
+        ignoreDynamicFieldsBeyondLimit = scopedSettings.get(INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING);
         mappingDepthLimit = scopedSettings.get(INDEX_MAPPING_DEPTH_LIMIT_SETTING);
         mappingFieldNameLengthLimit = scopedSettings.get(INDEX_MAPPING_FIELD_NAME_LENGTH_LIMIT_SETTING);
         mappingDimensionFieldsLimit = scopedSettings.get(INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING);
@@ -976,6 +979,10 @@ public IndexSettings(final IndexMetadata indexMetadata, final Settings nodeSetti
         scopedSettings.addSettingsUpdateConsumer(INDEX_SOFT_DELETES_RETENTION_LEASE_PERIOD_SETTING, this::setRetentionLeaseMillis);
         scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING, this::setMappingNestedFieldsLimit);
         scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING, this::setMappingNestedDocsLimit);
+        scopedSettings.addSettingsUpdateConsumer(
+            INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING,
+            this::setIgnoreDynamicFieldsBeyondLimit
+        );
         scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING, this::setMappingTotalFieldsLimit);
         scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_DEPTH_LIMIT_SETTING, this::setMappingDepthLimit);
         scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_FIELD_NAME_LENGTH_LIMIT_SETTING, this::setMappingFieldNameLengthLimit);
@@ -1519,6 +1526,14 @@ private void setMappingTotalFieldsLimit(long value) {
         this.mappingTotalFieldsLimit = value;
     }
 
+    private void setIgnoreDynamicFieldsBeyondLimit(boolean ignoreDynamicFieldsBeyondLimit) {
+        this.ignoreDynamicFieldsBeyondLimit = ignoreDynamicFieldsBeyondLimit;
+    }
+
+    public boolean isIgnoreDynamicFieldsBeyondLimit() {
+        return ignoreDynamicFieldsBeyondLimit;
+    }
+
     public long getMappingDepthLimit() {
         return mappingDepthLimit;
     }

server/src/main/java/org/elasticsearch/index/IndexSettings.java

@@ -629,11 +629,6 @@ public DeleteResult(Exception failure, long version, long term, long seqNo, bool
             this.found = found;
         }
 
-        public DeleteResult(Mapping requiredMappingUpdate, String id) {
-            super(Operation.TYPE.DELETE, requiredMappingUpdate, id);
-            this.found = false;
-        }
-
         public boolean isFound() {
             return found;
         }

server/src/main/java/org/elasticsearch/index/engine/Engine.java

@@ -514,7 +514,11 @@ private static void parseObjectDynamic(DocumentParserContext context, String cur
 
             }
             if (context.dynamic() != ObjectMapper.Dynamic.RUNTIME) {
-                context.addDynamicMapper(dynamicObjectMapper);
+                if (context.addDynamicMapper(dynamicObjectMapper) == false) {
+                    failIfMatchesRoutingPath(context, currentFieldName);
+                    context.parser().skipChildren();
+                    return;
+                }
             }
             if (dynamicObjectMapper instanceof NestedObjectMapper && context.isWithinCopyTo()) {
                 throwOnCreateDynamicNestedViaCopyTo(dynamicObjectMapper, context);
@@ -556,7 +560,10 @@ private static void parseArrayDynamic(DocumentParserContext context, String curr
                 parseNonDynamicArray(context, currentFieldName, currentFieldName);
             } else {
                 if (parsesArrayValue(objectMapperFromTemplate)) {
-                    context.addDynamicMapper(objectMapperFromTemplate);
+                    if (context.addDynamicMapper(objectMapperFromTemplate) == false) {
+                        context.parser().skipChildren();
+                        return;
+                    }
                     context.path().add(currentFieldName);
                     parseObjectOrField(context, objectMapperFromTemplate);
                     context.path().remove();
@@ -674,7 +681,9 @@ private static void parseDynamicValue(final DocumentParserContext context, Strin
             failIfMatchesRoutingPath(context, currentFieldName);
             return;
         }
-        context.dynamic().getDynamicFieldsBuilder().createDynamicFieldFromValue(context, currentFieldName);
+        if (context.dynamic().getDynamicFieldsBuilder().createDynamicFieldFromValue(context, currentFieldName) == false) {
+            failIfMatchesRoutingPath(context, currentFieldName);
+        }
     }
 
     private static void ensureNotStrict(DocumentParserContext context, String currentFieldName) {