Basic Normalization: Overrule default parameter values (query_tag a.o.)

[pjleiwa6g]

Tell us about the problem you're trying to solve

When using 'basic normalization' for a Snowflake destination, the value of query_tag is set to 'normalization'. I would like to have an option to overrule this value.

Related issues, in which query_tag and other dbc parameters can be set: #9623 which also solves this issue: #9467, but it only does this for the 'raw' method.

Describe the solution you’d like

I would like to have the solution for #9623 to be extended. So when a query_tag is entered in the UI it would overrule the default query_tag which is used for normalization. And also other additional parameters that are set should overrule the default.

Describe the alternative you’ve considered or used

Use the 'raw' method and do the normalization outside of Airbyte.

┆Issue is synchronized with this Asana task by Unito

[ChristopheDuong]

In that case, for implementation, I would suggest the destination-snowflake's spec should have a specific query-tag property instead of a generic jdbc-params (it's fine to keep the generic jdbc params, but it would be preferable to have the query tag isolated too)

Then in normalization you can check if the query-tag field is populated and use it or fallback to normalization here:

airbyte/airbyte-integrations/bases/base-normalization/normalization/transform_config/transform.py

Line 207 in a518247

"query_tag": "normalization",

[pjleiwa6g]

An possibility is to extract the query_tag value from the jdbc-params. If it's not in the jdbc-params, one can fall back to the default normalization.

Something like this (in transform.py):

• parameter ->query_tag
• jdbc_url_params ->config["jdbc_url_params"]
• default -> normalization

    ....    
    import urllib.parse
    ....

    def transform_snowflake(config: Dict[str, Any]):
    ....
        query_tag=TransformConfig.get_jdbc_param_value("query_tag".upper(),config["jdbc_url_params"].upper(),"normalization")
    ....
       "query_tag": query_tag,
    ....
    
    def get_jdbc_param_value(parameter,jdbc_url_params,default_value):
        parameter_length=len(parameter) + 1 # +1 because of the = sign
        parameters=jdbc_url_params.split(',')
        parameter_value=default_value
        
        for i in parameters:
            if i.startswith(parameter):
                parameter_value=urllib.parse.unquote(i[parameter_length:])
                                
                return parameter_value

[ChristopheDuong]

Sure it seems fine, you're welcome to push a PR there!

[vishjain]

Hi @marcosmarxm , Can you please assign this task? Thanks!

[marcosmarxm]

Basic normalization is deprecatd and was removed for Snowflake.