Support Incremental Syncs

[cgardens]

Tell us about the problem you're trying to solve

• Currently Airbyte transfers all data from source to destination on each sync.

Describe the solution you’d like

• I want to be able to send only:
  1. the records that changed
  2. new records
• As long as all of the data ends up in my warehouse I am happy. Airbyte does not need to dedupe records for me.

Describe the alternative you’ve considered or used

• We will support deduping and record merging in the future, but for our first pass at incremental.
• We will not support deletions in the first implementation. If you want to denote a record has been deleted, then it should contained a tombstone or deleted column.

Additional context

Add any other context or screenshots about the feature request here.

Tasks

• Plan doc
• Support passing persisting and passing state to sources (Support passing persisting and passing state to sources #990)
• Model incremental in AirbyteCatalog (Configuration Data model for Incremental in AirbyteCatalog #988)
• Source configuration of comparable:
  • Display configured comparable in the UI. (tbd)
  • Source-defined comparable (case where the source knows what the comparable is, the user need not provide input. e.g. stripe api.) (Source-defined comparable  #989)
  • User-defined comparable - (case where, for each stream, the user needs to define what the comparable is. e.g. database). (User-Defined Comparable #1019)
• Migrate destination-postgres (Add incremental support to Postgres (destination) #1018)
• Migrate destination-bigquery or destination-snowflake (Add incremental support to BQ #995)
• Migrate source-postgres (tbd)
• Migrate source- (1 of 3) (tbd)
• Migrate source- (2 of 3) (tbd)
• Migrate source- (3 of 3) (tbd)
• UI (Add configuring incremental to the UI #1074)

[cgardens]

this was completed in decemeber.