Update SQL/PPL multiple value field limitation #8646
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Peng Huo <[email protected]>
penghuo
requested review from
kolchfa-aws,
Naarcha-AWS,
vagimeli,
AMoo-Miki,
natebower,
dlvenable and
epugh
as code owners
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review. |
Signed-off-by: Peng Huo <[email protected]>
kolchfa-aws
reviewed
Nov 1, 2024
There was a problem hiding this comment.
Thank you, @penghuo! Could you please clarify the plugins.query.field_type_tolerance setting?
_search-plugins/sql/limitation.md
Outdated
| @@ -97,4 +97,4 @@ Such queries are successfully executed by the `V2` engine unless they have `V1`- | |||
| Suggested change | |||
| * The `V2` query engine does not support aggregation queries such as `histogram`, `date_histogram`, `percentiles`, `topHits`, `stats`, `extended_stats`, `terms`, or `range`. | |||
| * JOINs and sub-queries are not supported. To stay up to date on the development for JOINs and sub-queries, track [GitHub issue #1441](https://github.com/opensearch-project/sql/issues/1441) and [GitHub issue #892](https://github.com/opensearch-project/sql/issues/892). | |||
| * PartiQL syntax for `nested` queries are not supported. Additionally, arrays of objects and primitive types return the first index of the array, while in `V1` they return the entire array as a JSON object. | |||
| * PartiQL syntax for `nested` queries are not supported. Additionally, OpenSearch does not natively support the ARRAY data type but does allow multi-value fields implicitly. The SQL/PPL plugin adheres strictly to the data type semantics defined in index mappings. When parsing OpenSearch responses, it expects data to match the declared type and does not account for data in array format. If the [plugins.query.field_type_tolerance](https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst#plugins-query-field-type-tolerance) setting is enabled, the SQL/PPL plugin will handle array datasets by returning scalar data types, allowing basic queries (e.g., SELECT * FROM tbl WHERE condition). However, using multi-value fields in expressions or functions will result in exceptions. If this setting is disabled or absent, only the first element of an array is returned, preserving the default behavior. | |||
There was a problem hiding this comment.
Suggested change
| * PartiQL syntax for `nested` queries are not supported. Additionally, OpenSearch does not natively support the ARRAY data type but does allow multi-value fields implicitly. The SQL/PPL plugin adheres strictly to the data type semantics defined in index mappings. When parsing OpenSearch responses, it expects data to match the declared type and does not account for data in array format. If the [plugins.query.field_type_tolerance](https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst#plugins-query-field-type-tolerance) setting is enabled, the SQL/PPL plugin will handle array datasets by returning scalar data types, allowing basic queries (e.g., SELECT * FROM tbl WHERE condition). However, using multi-value fields in expressions or functions will result in exceptions. If this setting is disabled or absent, only the first element of an array is returned, preserving the default behavior. | |
| * PartiQL syntax for `nested` queries are not supported. Additionally, OpenSearch does not natively support the array data type but does allow multi-value fields implicitly. The SQL/PPL plugin adheres strictly to the data type semantics defined in index mappings. When parsing OpenSearch responses, it expects data to match the declared type and does not interpret all data in an array. If the [plugins.query.field_type_tolerance](https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst#plugins-query-field-type-tolerance) setting is enabled, the SQL/PPL plugin handles array datasets by returning scalar data types, allowing basic queries (for example, `SELECT * FROM tbl WHERE condition`). However, using multi-value fields in expressions or functions will result in exceptions. If this setting is disabled or not set, only the first element of an array is returned, preserving the default behavior. |
_search-plugins/sql/settings.md
Outdated
| @@ -79,6 +79,7 @@ Setting | Default | Description | |||
| `plugins.query.memory_limit` | 85% | Configures the heap memory usage limit for the circuit breaker of the query engine. | |||
| `plugins.query.size_limit` | 200 | Sets the default size of index that the query engine fetches from OpenSearch. | |||
| `plugins.query.datasources.enabled` | true | Change to `false` to disable support for data sources in the plugin. | |||
| `plugins.query.field_type_tolerance` | true | set to false, then an array is reduced to the first non array value of any level of nesting. | |||
There was a problem hiding this comment.
Suggested change
| `plugins.query.field_type_tolerance` | true | set to false, then an array is reduced to the first non array value of any level of nesting. | |
| `plugins.query.field_type_tolerance` | true | If `false`, then an array is reduced to the first non-array value at any nesting level. For example, `[[1, 2], [3, 4]]` will be reduced to `1`. |
_search-plugins/sql/settings.md
Outdated
| @@ -79,6 +79,7 @@ Setting | Default | Description | |||
| `plugins.query.memory_limit` | 85% | Configures the heap memory usage limit for the circuit breaker of the query engine. | |||
| `plugins.query.size_limit` | 200 | Sets the default size of index that the query engine fetches from OpenSearch. | |||
| `plugins.query.datasources.enabled` | true | Change to `false` to disable support for data sources in the plugin. | |||
| `plugins.query.field_type_tolerance` | true | set to false, then an array is reduced to the first non array value of any level of nesting. | |||
There was a problem hiding this comment.
Can we add info about what happens if this setting is true? Does OpenSearch still accept the array without attempting to reduce it to a single element? And what is the default?
There was a problem hiding this comment.
add settingt is true description, https://github.com/opensearch-project/documentation-website/pull/8646/files#diff-9d8dd46675baec61a9c50d9e2bb1054470a532c39ae8752e1f4e48b3e49c780eR82
default value is true.
more example in https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst#plugins-query-field-type-tolerance
kolchfa-aws
added
release-notes
anasalkouz
reviewed
Nov 1, 2024
_search-plugins/sql/limitation.md
Outdated
| @@ -97,4 +97,4 @@ Such queries are successfully executed by the `V2` engine unless they have `V1`- | |||
| Suggested change | |||
| * The `V2` query engine does not support aggregation queries such as `histogram`, `date_histogram`, `percentiles`, `topHits`, `stats`, `extended_stats`, `terms`, or `range`. | |||
| * JOINs and sub-queries are not supported. To stay up to date on the development for JOINs and sub-queries, track [GitHub issue #1441](https://github.com/opensearch-project/sql/issues/1441) and [GitHub issue #892](https://github.com/opensearch-project/sql/issues/892). | |||
| * PartiQL syntax for `nested` queries are not supported. Additionally, arrays of objects and primitive types return the first index of the array, while in `V1` they return the entire array as a JSON object. | |||
| * PartiQL syntax for `nested` queries are not supported. Additionally, OpenSearch does not natively support the ARRAY data type but does allow multi-value fields implicitly. The SQL/PPL plugin adheres strictly to the data type semantics defined in index mappings. When parsing OpenSearch responses, it expects data to match the declared type and does not account for data in array format. If the [plugins.query.field_type_tolerance](https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst#plugins-query-field-type-tolerance) setting is enabled, the SQL/PPL plugin will handle array datasets by returning scalar data types, allowing basic queries (e.g., SELECT * FROM tbl WHERE condition). However, using multi-value fields in expressions or functions will result in exceptions. If this setting is disabled or absent, only the first element of an array is returned, preserving the default behavior. | |||
There was a problem hiding this comment.
Shall we move this array type limitation as separate bullet point? to make more clear instead of having it as part of nested query limitation
There was a problem hiding this comment.
Signed-off-by: Peng Huo <[email protected]>
kolchfa-aws
reviewed
Nov 4, 2024
kolchfa-aws
reviewed
Nov 4, 2024
Signed-off-by: kolchfa-aws <[email protected]>
kolchfa-aws
approved these changes
Nov 4, 2024
natebower
reviewed
Nov 4, 2024
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
epugh
pushed a commit
to o19s/documentation-website
that referenced
this pull request
Nov 23, 2024
* Update SQL/PPL multiple value field limitation Signed-off-by: Peng Huo <[email protected]> * add plugins.query.field_type_tolerance setting Signed-off-by: Peng Huo <[email protected]> * address comments Signed-off-by: Peng Huo <[email protected]> * Apply suggestions from code review Signed-off-by: kolchfa-aws <[email protected]> * Update _search-plugins/sql/limitation.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Peng Huo <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Eric Pugh <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Port change in opensearch-project/sql#3118 to document website.
Issues Resolved
n/a
Version
2.18
Frontend features
If you're submitting documentation for an OpenSearch Dashboards feature, add a video that shows how a user will interact with the UI step by step. A voiceover is optional.
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.