Core: Migrating from joda to java.time. ML package #35493

pgomulka · 2018-11-13T12:50:51Z

part of the migrating joda time work. The goal is to find usages of joda
time and refactor to use java.time.

This commit moves the aggregation and mapping code from joda time to java time. This includes field mappers, root object mappers, aggregations with date histograms, query builders and a lot of changes within tests.

…m property configured

elasticmachine · 2018-11-13T12:50:53Z

Pinging @elastic/es-core-infra

dimitris-athanasiou · 2018-11-14T14:20:22Z

...in/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/fields/ExtractedField.java

@@ -112,6 +112,7 @@ public static ExtractedField newField(String alias, String name, ExtractionMetho
            }
            if (value[0] instanceof String) { // doc_value field with the epoch_millis format
                value[0] = Long.parseLong((String) value[0]);
+                //TODO confirm if can remove the script time field


OK, I had a discussion with @jpountz and got to the bottom of this. BaseDateTime was the type returned for date fields in doc values prior to allowing configuring the doc value format. Thus, we still need this for mixed 6.x clusters. The comment below // script field is actually misleading. ML never supported scripted date fields.

OK, I think the only thing left is to remove this TODO and correct the comment below (// script field). A more accurate comment would be // for handling doc value type prior to 6.4

I need to investigate what will be the behaviour for a mix cluster like that.
Is there a test in any of the old branches that I can modify and see that place being hit? We could possibly then think of creating a similar one on the master branch with a mixed cluster?

Actually we should be able to remove this from master. 7.x nodes should only talk to 6.latest which already have the doc value format specified. It's only in 6.x we need to keep this code for. Any datafeed test should be hitting this prior to 6.4.

Note that prior to 6.4 there was no if block in this method. It was directly casting to the joda time.

as we discussed, that sounds really good. Will remove that line in master

dimitris-athanasiou · 2018-11-14T14:21:01Z

...va/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/AggregationToJsonProcessor.java

@@ -179,7 +179,7 @@ private void processDateHistogram(Histogram agg) throws IOException {
     * Histograms have either a Double or Long.
     */
    private long toHistogramKeyToEpoch(Object key) {
-        if (key instanceof DateTime) {
+        if (key instanceof DateTime) {//TODO what do we use as key in aggregations


Is this a question for the ML folk?

@dimitris-athanasiou I thought initially to ask ML team, but from what I see now we are missing that migration in elasticsearch. This seems to be a planned work but not started yet.
Will have to switch to datetime histograms first before finishing ML package

That is on the java-time branch at the moment and the ZonedDateTime will be returned

dimitris-athanasiou · 2018-11-15T13:12:47Z

...n/java/org/elasticsearch/xpack/ml/job/persistence/overallbuckets/OverallBucketsProvider.java


+import java.time.Instant;


Is this import used?

unused indeed - will remove

dimitris-athanasiou · 2018-11-15T13:55:14Z

...va/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/AggregationToJsonProcessor.java

@@ -180,10 +180,10 @@ private void processDateHistogram(Histogram agg) throws IOException {
     */
    private long toHistogramKeyToEpoch(Object key) {
        if (key instanceof ZonedDateTime) {
-            return ((ZonedDateTime)key).toInstant().toEpochMilli();
+            return ((ZonedDateTime)keEy).toInstant().toEpochMilli();


I believe this introduced a typo: keEy instead of key

spinscale

left a few minor comments on how to get the current time in milliseconds, but it seems this has to wait for the java-time branch first

spinscale · 2018-11-26T14:10:52Z

...lugin/ml/src/main/java/org/elasticsearch/xpack/ml/job/retention/ExpiredForecastsRemover.java

@@ -66,7 +68,13 @@
    public ExpiredForecastsRemover(Client client, ThreadPool threadPool) {
        this.client = Objects.requireNonNull(client);
        this.threadPool = Objects.requireNonNull(threadPool);
-        this.cutoffEpochMs = DateTime.now(ISOChronology.getInstance()).getMillis();
+        this.cutoffEpochMs = getNowEpochMs();


how about Instant.now().toEpochMilli()?

I think you are right, there is no point using ISOChronology when all we need here are milliseconds since epoch

spinscale · 2018-11-26T14:11:51Z

...ml/src/main/java/org/elasticsearch/xpack/ml/job/retention/AbstractExpiredJobDataRemover.java

-        long nowEpochMs = DateTime.now(ISOChronology.getInstance()).getMillis();
+        Instant instant = ZonedDateTime.now().toInstant();
+        long nowEpochMs = IsoChronology.INSTANCE.zonedDateTime(instant, ZoneId.systemDefault())
+            .toInstant().toEpochMilli();


how about Instant.now().toEpochMilli()?

spinscale · 2018-11-26T14:12:23Z

x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/MlDailyMaintenanceService.java

-        return TimeValue.timeValueMillis(next.getMillis() - now.getMillis());
+
+        Instant instant = ZonedDateTime.now().toInstant();
+        ZonedDateTime now = IsoChronology.INSTANCE.zonedDateTime(instant, ZoneId.systemDefault());


how about Instant.now().toEpochMilli()?

actually I ZonedDatetime, might better here - more on a new review https://github.com/elastic/elasticsearch/pull/35949/files#r236714576

pgomulka · 2018-11-26T15:32:12Z

@spinscale I will rebase this branch on top of java-time branch. It will help me to integrate when java-time is merged into master

part of the migrating joda time work. The goal is to find usages of joda time and refactor to use java.time. closes: elastic#35490 confirmed histograms are returning zoneddatedate remove unused imports missing space typos removing the DateTime usage as it won't be hit removing tests no longer valid as code has been removed

pgomulka · 2018-11-29T11:47:02Z

closing, as the change was merged into java-time

spinscale added 30 commits September 20, 2018 12:57

Core: Move aggs/mapping code over to java time

19e3035

This commit moves the aggregation and mapping code from joda time to java time. This includes field mappers, root object mappers, aggregations with date histograms, query builders and a lot of changes within tests.

run test on java9 only

baf7acd

fix compilation error

efe5661

fix randomized test to use java time

b856795

silence another test for now on java8 due to BWC issues without syste…

443bdea

…m property configured

Merge branch 'master' into java-time

995855d

Merge branch 'master' into java-time

739ca1f

Merge branch 'master' into java-time

af7abc8

Merge branch 'master' into java-time

e6d61d2

Merge branch 'master' into java-time

a464a05

Merge branch 'master' into java-time

79f2ebb

fix wrong casting

a9de009

Merge branch 'master' into java-time

c9b7e22

fix import

5d34a06

Merge branch 'master' into java-time

121b208

fix compilation errors

4ab9cc2

Merge branch 'master' into java-time

13301c1

fix unit test

8f4564e

Merge branch 'master' into java-time

720caf8

Merge branch 'master' into java-time

26905c9

Merge branch 'master' into java-time

7b88806

Merge branch 'master' into java-time

d9b241c

fix compilation errors

3f8e10a

fix tests

3df911f

remove unused imports

a1987ef

Merge branch 'master' into java-time

8cd9cfa

Merge branch 'master' into java-time

d8af35d

Merge branch 'master' into java-time

5e50374

Merge branch 'master' into java-time

50e86b5

Merge branch 'master' into java-time

c58040f

pgomulka force-pushed the feature/ML-java-time-ml-migration branch 4 times, most recently from 055ec8d to d3a6e85 Compare November 13, 2018 13:41

rjernst requested a review from spinscale November 13, 2018 18:14

dimitris-athanasiou reviewed Nov 14, 2018

View reviewed changes

dimitris-athanasiou reviewed Nov 15, 2018

View reviewed changes

pgomulka force-pushed the feature/ML-java-time-ml-migration branch from 3d5696f to bb0d60b Compare November 21, 2018 14:07

pgomulka changed the base branch from master to java-time November 21, 2018 14:07

pgomulka changed the base branch from java-time to master November 21, 2018 14:51

pgomulka force-pushed the feature/ML-java-time-ml-migration branch from bb0d60b to d737fe5 Compare November 21, 2018 14:58

pgomulka added the WIP label Nov 21, 2018

pgomulka force-pushed the feature/ML-java-time-ml-migration branch from 41950c9 to 166be62 Compare November 22, 2018 09:24

Merge branch 'master' into java-time

9421ce6

pgomulka force-pushed the feature/ML-java-time-ml-migration branch from 166be62 to cef95b2 Compare November 26, 2018 11:16

spinscale added 2 commits November 26, 2018 12:21

removed import

4cf8463

Merge branch 'master' into java-time

568c1b5

spinscale suggested changes Nov 26, 2018

View reviewed changes

pgomulka force-pushed the feature/ML-java-time-ml-migration branch from cef95b2 to 06b44dc Compare November 27, 2018 10:47

pgomulka added 2 commits November 27, 2018 13:41

Merge branch 'master' into java-time

1edd15c

pgomulka force-pushed the feature/ML-java-time-ml-migration branch from 06b44dc to 3b90c7b Compare November 27, 2018 12:42

pgomulka mentioned this pull request Nov 27, 2018

Core: Migrating from joda to java.time. ML package #35949

Merged

pgomulka closed this Nov 29, 2018

ChrisHegarty unassigned pgomulka Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core: Migrating from joda to java.time. ML package #35493

Core: Migrating from joda to java.time. ML package #35493

pgomulka commented Nov 13, 2018 •

edited

Loading

elasticmachine commented Nov 13, 2018

dimitris-athanasiou Nov 14, 2018

dimitris-athanasiou Nov 15, 2018

pgomulka Nov 15, 2018

dimitris-athanasiou Nov 15, 2018

dimitris-athanasiou Nov 15, 2018

dimitris-athanasiou Nov 15, 2018

pgomulka Nov 15, 2018

dimitris-athanasiou Nov 14, 2018

pgomulka Nov 14, 2018

pgomulka Nov 15, 2018 •

edited

Loading

dimitris-athanasiou Nov 15, 2018

pgomulka Nov 15, 2018

dimitris-athanasiou Nov 15, 2018

spinscale left a comment

spinscale Nov 26, 2018

pgomulka Nov 26, 2018

spinscale Nov 26, 2018

pgomulka Nov 26, 2018

spinscale Nov 26, 2018

pgomulka Nov 26, 2018

pgomulka Nov 27, 2018

pgomulka commented Nov 26, 2018

pgomulka commented Nov 29, 2018

Core: Migrating from joda to java.time. ML package #35493

Core: Migrating from joda to java.time. ML package #35493

Conversation

pgomulka commented Nov 13, 2018 • edited Loading

elasticmachine commented Nov 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgomulka Nov 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spinscale left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgomulka commented Nov 26, 2018

pgomulka commented Nov 29, 2018

pgomulka commented Nov 13, 2018 •

edited

Loading

pgomulka Nov 15, 2018 •

edited

Loading