Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v1] Test conformance report upload #1726

Merged
merged 1 commit into from
Jan 22, 2025
Merged

Conversation

alancai98
Copy link
Member

@alancai98 alancai98 commented Jan 22, 2025

Adds back default name for conformance report upload.

License Information

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copy link

CROSS-ENGINE-REPORT ❌

BASE (LEGACY-V0.14.8) TARGET (EVAL-6F2065B) +/-
% Passing 89.67% 95.00% 5.33% ✅
Passing 5287 5620 333 ✅
Failing 609 76 -533 ✅
Ignored 0 220 220 🔶
Total Tests 5896 5916 20 ✅

Testing Details

  • Base Commit: v0.14.8
  • Base Engine: LEGACY
  • Target Commit: 6f2065b
  • Target Engine: EVAL

Result Details

  • ❌ REGRESSION DETECTED. See Now Failing/Ignored Tests. ❌
  • Passing in both: 2622
  • Failing in both: 17
  • Ignored in both: 0
  • PASSING in BASE but now FAILING in TARGET: 20
  • PASSING in BASE but now IGNORED in TARGET: 98
  • FAILING in BASE but now PASSING in TARGET: 180
  • IGNORED in BASE but now PASSING in TARGET: 0

Now FAILING Tests ❌

The complete list can be found in GitHub CI summary, either from Step Summary or in the Artifact.

Now IGNORED Tests ❌

The complete list can be found in GitHub CI summary, either from Step Summary or in the Artifact.

Now Passing Tests

180 test(s) were previously failing in BASE (LEGACY-V0.14.8) but now pass in TARGET (EVAL-6F2065B). Before merging, confirm they are intended to pass.

The complete list can be found in GitHub CI summary, either from Step Summary or in the Artifact.

@alancai98 alancai98 self-assigned this Jan 22, 2025
@alancai98 alancai98 requested a review from johnedquinn January 22, 2025 22:59
@alancai98 alancai98 merged commit 470d383 into main Jan 22, 2025
14 checks passed
@alancai98 alancai98 deleted the test-conformance-report-upload branch January 22, 2025 23:00
@alancai98
Copy link
Member Author

for context, the cross-report conformance report was not appearing because the uploaded artifact name was changed in https://github.com/partiql/partiql-lang-kotlin/pull/1717/files#diff-66e2481b3cb9e39d39407d9415c75e4d26aada84b3658b8054071bee9f9b518d. changing it to the default artifact fixes the upload as seen in this other test PR -- #1727 (comment) which targets this PR's branch.

the cross-commit appears in #1727 but not this PR since this PR is targeting main which currently still uses the "broken" artifact name.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants