Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a common DB Writer class to be used by many scripts #79

Open
rudokemper opened this issue Feb 14, 2025 · 0 comments
Open

Create a common DB Writer class to be used by many scripts #79

rudokemper opened this issue Feb 14, 2025 · 0 comments
Labels
connectors Connector scripts for ETL from upstream data sources refactor/cleanup
Milestone

Comments

@rudokemper
Copy link
Member

rudokemper commented Feb 14, 2025

Feature Request

Right now, comapeo_observations, odk_responses, kobo_responses, locusmap, and alerts_gcs have a DBWriter class to manage converting a dataset into structured SQL tables and writing them to Postgres using psycopg2.

There is a tremendous amount of duplication of code across these classes; for all but the alerts script, there is little to no difference whatsoever, sans minor differences in logging messages or some specific property like the exact name of the primary key field.

Let's come up with a common DB Writer class that all of these scripts can use as a modular import, stored in f/common_logic.

In so doing, we can upgrade to psycopg3.

@rudokemper rudokemper added connectors Connector scripts for ETL from upstream data sources refactor/cleanup labels Feb 14, 2025
@rudokemper rudokemper added this to the Nia Tero 2025 milestone Feb 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
connectors Connector scripts for ETL from upstream data sources refactor/cleanup
Projects
None yet
Development

No branches or pull requests

1 participant