GEN-2157: Snowflake increased cost issue

[harshsoni2024]

Describe your changes:

Fixes

Here are some of the findings:

1. Simply maintaining an open connection with engine.connect() does not keep the warehouse active.
2. Only active queries or operations (like metadata queries or table scans) trigger warehouse usage.

• Snowflake warehouse has auto suspend/resume feature which is enable by default doc

Type of change:

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

[notion-workspace]

snowflake increased cost during lineage ingestion process

[pmbrull]

this can kill the memory. Try server side cursors. example

with engine.connect() as conn:
    result = conn.execution_options(stream_results=True, max_row_buffer=100).execute(
        text("select * from public.employees")
    )
    counter = 0
    for row in result:
        counter += 1
print(counter)

[sonarqubecloud]

Quality Gate failed for 'open-metadata-ingestion'

Failed conditions
0.0% Coverage on New Code (required ≥ 20%)

See analysis details on SonarQube Cloud