You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Function prepare_table() in SQLConnector calls prepare_column(), for each column in a table.
In turn, this function queries the database to first determine whether the column exists or not. Therefore we make O(n) calls to the database.
However, in a single call we can retrieve all columns and leverage this information.
The target-postgres already implemented this optimisation by overriding the default methods prepare_table() and prepare_columns().
The way target-postgres implemented it implied modifying the signature of the functions, my implementation is different.
I take the responsibility of checking whether the column exists out of the prepare_column() function and encapsulate this in a new function called prepare_table_columns().
However, a thought: I think that in the end the underlying issue is that functions like prepare_table() and this new function prepare_table_columns() belong to the SQLSink class rather than the connector: a new sink is created every time there is a schema change. The class that knows the current schema and therefore can do these optimisations is the sink, not the connector maybe?
The text was updated successfully, but these errors were encountered:
Feature scope
Taps (catalog, state, stream maps, tests, etc.)
Description
Function
prepare_table()
inSQLConnector
callsprepare_column()
, for each column in a table.In turn, this function queries the database to first determine whether the column exists or not. Therefore we make O(n) calls to the database.
However, in a single call we can retrieve all columns and leverage this information.
The
target-postgres
already implemented this optimisation by overriding the default methodsprepare_table()
andprepare_columns()
.The way
target-postgres
implemented it implied modifying the signature of the functions, my implementation is different.I take the responsibility of checking whether the column exists out of the
prepare_column()
function and encapsulate this in a new function calledprepare_table_columns()
.However, a thought: I think that in the end the underlying issue is that functions like
prepare_table()
and this new functionprepare_table_columns()
belong to theSQLSink
class rather than the connector: a new sink is created every time there is a schema change. The class that knows the current schema and therefore can do these optimisations is the sink, not the connector maybe?The text was updated successfully, but these errors were encountered: