Add support for derive source feature and Implementing it for DateFieldMapper

[tanik98]

Description

• Adding interface for support of building source using stored/docvalues in FieldMapper

• Added derivedSourceSupported flag to allowlist field mapper for this feature

• Implemented these methods for DateFieldMapper

  • In DateFieldMapper, stored field are stored as Long where as docValues are stored as sorted numeric in lucene.
  • While deriving source, we are first checking for stored field and if not found we are checking docValues, only difference between these would be, in stored field order would be preserved for multi value case

• Tested the changes by integrating the flow with "doc by id" query path, cases covered:

  1. store=false, field value is getting fetched from docValues, for multi value result is returned in sorted order
  2. store=true, field value is getting fetched from stored field
  3. date format is defined, in result date will preserve the format
  4. store=false & doc_values=false, unsupported exception will be thrown

Related Issues

Resolves #17073

Part of feature #9568

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 5aa6a8c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mgodwan]

Why do we need 2 separate methods i.e. buildDerivedSource and deriveSource?

[tanik98]

Removed the buildDerivedSource method and embeded derive source feasibility checks under single method canDeriveSource only.

[mgodwan]

Since stored fields are stored in row fashion, it may turn out to be slower than doc values. Can we create an issue to benchmark what to prefer if both are available?

[tanik98]

For fetching few values, doc values would be faster for almost all cases as it can be served from OS cache(it can be very performant as multiple doc ids can be served from the same page) as well, whereas stored fields are stored in row fashion so for this serving as many doc ids as doc values might not be possible from a single memory page.

But while deriving source with multiple fields using stored field(considering for most fields, stored field is enabled) might be performant as for a single doc, all the field would be in nearby locality in memory. This we can benchmark, for now going with doc value first approach only considering wider cases where stored field would not be enabled explicitly in index field mapping and by default it's false.

[jmazanec15]

Will this support nested and/or object fields?

[tanik98]

Initially, will keep this feature disabled for nested object, later we will expand it for nested object fields as well.

[jmazanec15]

Is there a field level parameter that enables this?

[tanik98]

Yes, derivedSourceSupported flag is defined in MappedFieldType, which controls the feature at field level.

[github-actions]

❌ Gradle check result for ab3aa0a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for b0d4600: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?