Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Option to disable schema validation in targets #1904

Closed
pnadolny13 opened this issue Aug 9, 2023 · 2 comments · Fixed by #2208
Closed

feat: Option to disable schema validation in targets #1904

pnadolny13 opened this issue Aug 9, 2023 · 2 comments · Fixed by #2208

Comments

@pnadolny13
Copy link
Contributor

pnadolny13 commented Aug 9, 2023

The transferwise variant of target-snowflake has a validate_records setting that is defaulted to false. In the SDK we automatically do validation and don't provide a way to turn it off. We've seen some people struggle to transition to the new meltanolabs target-snowflake because validation is failing, its likely correctly failing but the solution for users is to fix the tap which becomes too big of a barrier for them when switching.

(Default: False) Validate every single record message to the corresponding JSON schema.
This option is disabled by default and invalid RECORD messages will fail only at load time by Snowflake.
Enabling this option will detect invalid records earlier but could cause performance degradation.

cc @tayloramurphy @edgarrmondragon

@edgarrmondragon
Copy link
Collaborator

Yeah, that's something we could certainly do.

@edgarrmondragon
Copy link
Collaborator

edgarrmondragon commented Jan 18, 2024

This was made a lot easier to implement by #2136 (currently only available in prerelease v0.35.0a2).

One could override Sink.validate_schema to read from config:

class Sink:
  @property
  def validate_schema(self) -> bool:
    """Enable JSON schema record validation."""
    return config.get("validate_records", True)  # TBD if we keep the current default of enabling validation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants