Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQL: day and month names tests workaround #33653

Merged
merged 7 commits into from
Sep 19, 2018
Merged

Conversation

astefan
Copy link
Contributor

@astefan astefan commented Sep 12, 2018

Because of #33621 (comment) and the fact that -Djava.locale.providers=COMPAT option is not passed along when the jvm is forked for a test, the tests use a different approach to verify the DAYNAME and MONTHNAME functions.

Fixes #33621

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search-aggs

@vladimirdolzhenko
Copy link
Contributor

We discuss with Andrei it would be good to enforce using -Djava.locale.providers=COMPAT:

  • put that system property via build.gradle
  • add comment and check that specific property is specified within NamedDateTimeProcessorTests

Copy link
Contributor

@matriv matriv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@costin costin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the locale is the canary in the mine.

  • Is the COMPAT locale used by ES itself when running (like we do for other JVM options)?

If it is then it's a bug in the way the JVMs are started. If not, then it's something that needs to be fixed in the SQL functions code as the functions need to result the same result regardless of the JVM used underneath.
Further more considering this is the default behavior in Java 9, it is likely the COMPAT flag is on its way out.

I would assume there's a formatting option that's available in Java9 (which can be detected while keeping the Java 8 compatibility in place) to get a hold of the short format - if not then returning the first 3 chars should work.

@vladimirdolzhenko
Copy link
Contributor

@costin yes - ES enforces this specific value for java.locale.providers

$ grep COMPAT config/jvm.options
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
9-:-Djava.locale.providers=COMPAT

@astefan
Copy link
Contributor Author

astefan commented Sep 18, 2018

@costin @matriv I've changed the approach with this PR and assumed the needed (and, in fact, right to be there) parameter should be there (at least for Java 9-11), otherwise the tests are skipped. Can you, please, have another look?

Copy link
Member

@costin costin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a small comment - LGTM otherwise.

}
String beforeJava9CompatibleLocale = System.getProperty("java.locale.providers");
// and COMPAT setting needs to be first on the list
boolean isBeforeJava9Compatible = beforeJava9CompatibleLocale != null && beforeJava9CompatibleLocale.split(",")[0].equals("COMPAT");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if it's not the first in the list? This piece can be more reliable with something like:
Arrays.toList(Strings.tokenizeToStringArray(beforeJava9...)).contains("COMPAT") - works with whitespace, nulls, out-of-order arguments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that the list is a preference list with the first provider being the first one checked and used. But I see your point about a more reliable check.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even If it's the case that needs to be first, I agree it would be nice to use the tokenize to trim whitespaces.

@astefan astefan merged commit c6e3231 into elastic:master Sep 19, 2018
@astefan astefan deleted the 33621_fix branch September 19, 2018 04:33
astefan added a commit to astefan/elasticsearch that referenced this pull request Sep 19, 2018
…lastic#33653)

DAYNAME and MONTHNAME functions tests will be skipped if the right JVM parameter (-Djava.locale.providers=COMPAT) is not used in unit testing environment
@astefan
Copy link
Contributor Author

astefan commented Sep 19, 2018

Fixed:
master: c6e3231
6.x: 4480718

jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Sep 19, 2018
* master: (46 commits)
  Fixing assertions in integration test (elastic#33833)
  [CCR] Rename idle_shard_retry_delay to poll_timout in auto follow patterns (elastic#33821)
  HLRC: Delete ML calendar (elastic#33775)
  Move DocsStats into Engine (elastic#33835)
  [Docs] Clarify accessing Date methods in painless (elastic#33560)
  add elasticsearch-shard tool (elastic#32281)
  Cut over to unwrap segment reader (elastic#33843)
  SQL: Fix issue with options for QUERY() and MATCH(). (elastic#33828)
  Emphasize that filesystem-level backups don't work (elastic#33102)
  Use the global doc id to generate a random score (elastic#33599)
  Add minimal sanity checks to custom/scripted similarities. (elastic#33564)
  Profiler: Don’t profile NEXTDOC for ConstantScoreQuery. (elastic#33196)
  [CCR] Change FollowIndexAction.Request class to be more user friendly (elastic#33810)
  SQL: day and month name functions tests locale providers enforcement (elastic#33653)
  TESTS: Set SO_LINGER = 0 for MockNioTransport (elastic#32560)
  Test: Relax jarhell gradle test (elastic#33787)
  [CCR] Fail with a descriptive error if leader index does not exist (elastic#33797)
  Add ES version 6.4.2 (elastic#33831)
  MINOR: Remove Some Dead Code in Scripting (elastic#33800)
  Ensure realtime `_get` and `_termvectors` don't run on the network thread (elastic#33814)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants