Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove json schema validator #1043

Merged
merged 4 commits into from
Feb 8, 2023
Merged

Conversation

windoze
Copy link
Member

@windoze windoze commented Feb 8, 2023

Description

  • Remove JSON Schema Validator dependency so we won't need additional parameter to submit Spark job.
  • Resolve the not found issue for non maven package in Azure Databricks/Synapse
  • Publish a v0.10.4-rc5 to maven and test can pass.

Does this PR introduce any user-facing changes?

  • No. You can skip the rest of this section.
  • Yes. Make sure to clarify your proposed changes.

@blrchen blrchen changed the title Windoze/remove json schema validator Remove json schema validator Feb 8, 2023
@blrchen
Copy link
Collaborator

blrchen commented Feb 8, 2023

LGTM, the main reason of this removal is curent json validator code introduce a new dependency org.everit.json.schema;1.9.1 which is not available in maven central. This will results org.everit.json.schema;1.9.1: not found job runtime error when running on cloud sparks like Azure Databricks and Azure Synpase. This is a break change as user have to apply manual workaround on spark in order to make job succeed.

Given the json schema validator is not used in current compute model, removal should be low risk if all CI test and notebook test can pass. We can revisit this when it's time to turn on vnext compute model, and explore more options to ensure critial cloud use case not break:

  1. Downgrade to the old version org.everit.json.schema available on maven
  2. Fork latest org.everit.json.schema code and submit to maven central
  3. Research how to automatically install non-maven packages on cloud sparks without manual workaround from end user side.
  4. Use a different json schema package which can be pulled from maven.

@windoze windoze merged commit ea1f399 into main Feb 8, 2023
@windoze windoze deleted the windoze/remove-json-schema-validator branch February 8, 2023 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants