Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v1] Re-enable some randomized tests #1696

Merged
merged 7 commits into from
Jan 15, 2025
Merged

Conversation

alancai98
Copy link
Member

@alancai98 alancai98 commented Jan 6, 2025

Relevant Issues

Description

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copy link

github-actions bot commented Jan 6, 2025

CROSS-ENGINE-REPORT ❌

BASE (LEGACY-V0.14.8) TARGET (EVAL-15879F1) +/-
% Passing 89.67% 93.20% 3.53% ✅
Passing 5287 5495 208 ✅
Failing 609 115 -494 ✅
Ignored 0 286 286 🔶
Total Tests 5896 5896 0 ✅

Testing Details

  • Base Commit: v0.14.8
  • Base Engine: LEGACY
  • Target Commit: 15879f1
  • Target Engine: EVAL

Result Details

  • ❌ REGRESSION DETECTED. See Now Failing/Ignored Tests. ❌
  • Passing in both: 2614
  • Failing in both: 32
  • Ignored in both: 0
  • PASSING in BASE but now FAILING in TARGET: 26
  • PASSING in BASE but now IGNORED in TARGET: 114
  • FAILING in BASE but now PASSING in TARGET: 165
  • IGNORED in BASE but now PASSING in TARGET: 0

Now FAILING Tests ❌

The complete list can be found in GitHub CI summary, either from Step Summary or in the Artifact.

Now IGNORED Tests ❌

The complete list can be found in GitHub CI summary, either from Step Summary or in the Artifact.

Now Passing Tests

165 test(s) were previously failing in BASE (LEGACY-V0.14.8) but now pass in TARGET (EVAL-15879F1). Before merging, confirm they are intended to pass.

The complete list can be found in GitHub CI summary, either from Step Summary or in the Artifact.

CROSS-COMMIT-REPORT ✅

BASE (EVAL-A7B772E) TARGET (EVAL-15879F1) +/-
% Passing 93.20% 93.20% 0.00% ✅
Passing 5495 5495 0 ✅
Failing 115 115 0 ✅
Ignored 286 286 0 ✅
Total Tests 5896 5896 0 ✅

Testing Details

  • Base Commit: a7b772e
  • Base Engine: EVAL
  • Target Commit: 15879f1
  • Target Engine: EVAL

Result Details

  • Passing in both: 5495
  • Failing in both: 115
  • Ignored in both: 286
  • PASSING in BASE but now FAILING in TARGET: 0
  • PASSING in BASE but now IGNORED in TARGET: 0
  • FAILING in BASE but now PASSING in TARGET: 0
  • IGNORED in BASE but now PASSING in TARGET: 0

@@ -1,121 +0,0 @@
// package org.partiql.lang.randomized.eval.builtins
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(self-review) TimestampTemporalAccessor class was deleted in v1 branch so tests no longer relevant

@alancai98 alancai98 marked this pull request as ready for review January 13, 2025 23:09
@alancai98 alancai98 force-pushed the enable-some-randomized-tests branch from 71fa7c4 to b83cdc8 Compare January 14, 2025 00:54
@@ -33,7 +33,7 @@ internal object DateTimeUtils {
internal fun parseTime(input: String): LocalTime {
val matcher: Matcher = TIME_PATTERN.matcher(input)
if (!matcher.matches()) throw DateTimeException(
"Expect TIME Format to be in HH-mm-ss.ddd+[+|-][hh:mm|z], received $input"
"Expect TIME Format to be in HH:mm:ss.ddd+[+|-][hh:mm|z], received $input"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(self-review) these error messages were slightly off. Should use colons rather than dashes for the time part delimiters

)
}

companion object {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(self-review) had to rewrite a big chunk of this test file since the prior modeling was pretty confusing, especially for how it reused test cases. We could clean up these tests further, but given the time constraints, didn't want to spend too long.

* introduction of StaticType. The behavior described in these tests is still how we should handle integer arithmetic
* in the absence of type information.
*
* TODO these tests are not correct and the implementation is not correct. The tests and implementation need to give
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(self-review) since the implementation is currently not correct for integer overflow and these tests were not correct, chose to disable all these tests until #1697 is resolved.

@alancai98 alancai98 linked an issue Jan 14, 2025 that may be closed by this pull request
@alancai98 alancai98 self-assigned this Jan 14, 2025
@alancai98 alancai98 requested a review from johnedquinn January 14, 2025 01:18
@alancai98 alancai98 changed the title [draft][v1] Re-enable some randomized tests Re-enable some randomized tests Jan 14, 2025
@alancai98 alancai98 changed the title Re-enable some randomized tests [v1] Re-enable some randomized tests Jan 14, 2025
@alancai98 alancai98 force-pushed the enable-some-randomized-tests branch from 882472b to 1bead61 Compare January 14, 2025 23:39
Copy link
Member

@johnedquinn johnedquinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alancai98 alancai98 merged commit fc208aa into main Jan 15, 2025
14 checks passed
@alancai98 alancai98 deleted the enable-some-randomized-tests branch January 15, 2025 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[V1] Add support for randomized tests on 1.0 evaluator.
2 participants