Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ES|QL] Support date_nanos field type #109352

Open
21 of 25 tasks
wchaparro opened this issue Jun 4, 2024 · 14 comments
Open
21 of 25 tasks

[ES|QL] Support date_nanos field type #109352

wchaparro opened this issue Jun 4, 2024 · 14 comments
Assignees
@not-napoleon
Labels
:Analytics/ES|QL AKA ESQL ES|QL-ui Impacts ES|QL UI Meta Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Comments

@wchaparro
Copy link
Member

wchaparro commented Jun 4, 2024 •
edited by not-napoleon
Loading

We want to support this field type in ES|QL, ( for filtering / sorting etc) . Not intending to support aggregations on date_nanos (current aggregations truncate the date_nanos to millisecond resolution before aggregating.

Tasks before we can remove the feature flag
Preview Give feedback
  1. [ESQL] Load and Return Date Nanos Values #109987
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  2. [ESQL] Support basic casting for date nanos #109990
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  3. [ESQL] Support date nanos on functions that take any type #109998
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  4. [ESQL] Support date nanos in binary comparisions #109992
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  5. [ESQL] create TO_DATE_NANOS function #111842
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  6. [ESQL] Support Max & Min aggregation on Date Nanos #110002
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  7. [ESQL] Support Count & Count Distinct aggregations on date nanos #110003
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  8. [ESQL] Support Values aggregation on date nanos #110005
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  9. [ESQL] Sort results by a date nanos field #109988
    :Analytics/ES|QL >enhancement Team:Analytics
  10. [ESQL] Testing for date nanos union type #112885
    :Analytics/ES|QL >test Team:Analytics
    not-napoleon

Additional work required for autocasting
Preview Give feedback
  1. [ESQL] Date Trunc function should support date nanos #110008
    :Analytics/ES|QL >enhancement Team:Analytics team-discuss
    not-napoleon
  2. [ESQL] Support addition and subtraction of date nanos #109995
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  3. Allow for mixed comparisons between millis and nanos #116281
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  4. [ESQL] Support date nanos on Bucket function #118031
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  5. Autocast Keyword to Date in Date Nanos comparisons #118476
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  6. Support Date Nanos for In function #118578
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  7. [ESQL] Support passing Date_Nanos to Date_Format function #109994
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  8. [ESQL] Add a version of DATE_PARSE that returns date nanos #109997
    :Analytics/ES|QL >enhancement Team:Analytics
  9. [ESQL] Support Date Diff function on date nanos #109999
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  10. [ESQL] Support Date Extract function on date nanos #110000
    :Analytics/ES|QL >enhancement Team:Analytics
    not-napoleon
  11. [ESQL] Extend range of supported timespan literals #111741
    :Analytics/ES|QL >enhancement Team:Analytics
  12. [ESQL] Support autocasting to resolve types between nanosecond dates and millisecond dates #110009
    :Analytics/ES|QL >enhancement Team:Analytics team-discuss
@wchaparro wchaparro added the :Analytics/ES|QL AKA ESQL label Jun 4, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jun 4, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@ruflin
Copy link
Contributor

ruflin commented Jun 5, 2024

Support for date_nanos type is critical as we eventually plan to move over to data_nanos for all logs data ( #102548) and quite a few users already use it as their default today. One of the key features is sorting. These users have more then 1 log event per millisecond and sorting by @timestamp must return the correct order where this events were created.

@ninoslavmiskovic
Copy link

@wchaparro Could you clarify this

"Not intending to support aggregations"

This could be a blocker for us since it would mean that we can not create a histogram in Discover.

If we do as current aggregations truncate the date_nano to millisecond resolution before aggregating, then we would be able to e.g. show the histogram like today in non-ES|QL environments.

cc @stratoula , @kertal , @davismcphee

@stratoula
Copy link

This could be a blocker for us since it would mean that we can not create a histogram in Discover

What do you mean Nino?

@ninoslavmiskovic
Copy link

@stratoula I talked to @kertal earlier today, and if we support the field type: date_nano but do not support the ability to do aggregations (STATS), then it would be impossible to build a histogram right?

@stratoula
Copy link

The histogram is built using a count aggregation and the @timestamp field. I can't think why not supporting aggs in the date_nano field types will break it or anything. But maybe @kertal you can explain what I am missing here.

@kertal
Copy link
Member

kertal commented Jun 6, 2024

@stratoula so when the @timestamp field is of type date_nanos (which is planned for logs data), it would work out of the box even if in theory date_nanos wouldn't support aggregations (which I assume it will but just with millisecond resolution)

@stratoula
Copy link

stratoula commented Jun 6, 2024 •
edited
Loading

@nik9000 if the @timestamp is date_nanos

this will work,right?

from kibana_sample_data_logs | limit 10 | EVAL timestamp=DATE_TRUNC(30 second, @timestamp) | stats results = count(*) by timestamp | rename timestamp as `@timestamp every 30 second`

@not-napoleon
Copy link
Member

@stratoula I'm picking this up now that I'm back from PTO. I can tell you for a fact that the query you pasted doesn't work right now with date nanos, nor would I expect it to. Supporting that type of thing is exactly what we're talking about here.

I've done some task breakdown for this already, and I'm going to flesh out this ticket (and create some sub-tickets) today.

@not-napoleon not-napoleon self-assigned this Jun 20, 2024
@stratoula
Copy link

Oh good to know. So date_trunc doesn't work with date nanos? Thanx for looking into this!

@nik9000 nik9000 added the ES|QL-ui Impacts ES|QL UI label Jun 20, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/kibana-esql (ES|QL-ui)

@nik9000
Copy link
Member

nik9000 commented Jun 20, 2024

Right now nothing works with date_nanos. Job 1 is loading them and returning them over the wire. Job 2 is plugging functions into them. After that there's a bunch of functions to look at, including aggs. Stuff like TO_STRING and TO_LONG and DATE_TRUNC and BUCKET and MAX and topn and I dunno, there are like 30 things that'll need touching.

@bpintea bpintea mentioned this issue Jul 19, 2024
Merged
@felixbarny
Copy link
Member

This is also relevant for the OTel program as we're planning to map @timestamp to date_nanos for all OTel data streams. When do we expect this to be released? I'm wondering if we should start with date and change to date_nanos later.

@nik9000
Copy link
Member

nik9000 commented Jul 24, 2024

This is also relevant for the OTel program as we're planning to map @timestamp to date_nanos for all OTel data streams. When do we expect this to be released? I'm wondering if we should start with date and change to date_nanos later.

You know we don't give dates. We're working on it now though.

@felixbarny felixbarny mentioned this issue Jul 25, 2024
Merged
3 tasks
@kkrik-es kkrik-es mentioned this issue Aug 20, 2024
Open
@not-napoleon not-napoleon mentioned this issue Sep 18, 2024
Merged
@carsonip carsonip changed the title [ES|QL] Support date_nano field type [ES|QL] Support date_nanos field type Sep 19, 2024
@felixbarny felixbarny mentioned this issue Nov 14, 2024
Draft
3 tasks
@not-napoleon not-napoleon mentioned this issue Nov 19, 2024
Merged
not-napoleon added a commit that referenced this issue Nov 20, 2024
@not-napoleon
Esql Enable Date Nanos (#117080)
c3f73d0 
This enables date nanos support as tech preview. Basic operations, like reading values, binary comparisons, and functions that don't care about type should work, but some functions are not yet supported. Most notably, Bucket is not yet supported, although Date_Trunc is and can be used for grouping. See the docs for the full list of limitations.

relates to #109352
not-napoleon added a commit to not-napoleon/elasticsearch that referenced this issue Nov 20, 2024
@not-napoleon
Esql Enable Date Nanos (elastic#117080)
570a55c 
This enables date nanos support as tech preview. Basic operations, like reading values, binary comparisons, and functions that don't care about type should work, but some functions are not yet supported. Most notably, Bucket is not yet supported, although Date_Trunc is and can be used for grouping. See the docs for the full list of limitations.

relates to elastic#109352
elasticsearchmachine pushed a commit that referenced this issue Nov 20, 2024
@not-napoleon @nik9000
[8.x] Esql Enable Date Nanos (#117080) (#117161)
cafa440 
* Esql Enable Date Nanos (#117080)

This enables date nanos support as tech preview. Basic operations, like reading values, binary comparisons, and functions that don't care about type should work, but some functions are not yet supported. Most notably, Bucket is not yet supported, although Date_Trunc is and can be used for grouping. See the docs for the full list of limitations.

relates to #109352

* Skip CATEGORIZE tests outside snapshot

---------

Co-authored-by: Nik Everett <[email protected]>
rjernst pushed a commit to rjernst/elasticsearch that referenced this issue Nov 20, 2024
@not-napoleon @rjernst
Esql Enable Date Nanos (elastic#117080)
b6cdb1f 
This enables date nanos support as tech preview. Basic operations, like reading values, binary comparisons, and functions that don't care about type should work, but some functions are not yet supported. Most notably, Bucket is not yet supported, although Date_Trunc is and can be used for grouping. See the docs for the full list of limitations.

relates to elastic#109352
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this issue Nov 28, 2024
@not-napoleon @alexey-ivanov-es
Esql Enable Date Nanos (elastic#117080)
d968898 
This enables date nanos support as tech preview. Basic operations, like reading values, binary comparisons, and functions that don't care about type should work, but some functions are not yet supported. Most notably, Bucket is not yet supported, although Date_Trunc is and can be used for grouping. See the docs for the full list of limitations.

relates to elastic#109352
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@not-napoleon not-napoleon

Labels
:Analytics/ES|QL AKA ESQL ES|QL-ui Impacts ES|QL UI Meta Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@nik9000 @ruflin @kertal @not-napoleon @felixbarny @wchaparro @stratoula @elasticsearchmachine @ninoslavmiskovic