Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added info about normalization costs to docs #22182

Merged
merged 2 commits into from
Feb 1, 2023
Merged

Added info about normalization costs to docs #22182

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
58a0a85
added info about normalization costs
sophia-wiley Jan 31, 2023
4e4df09
edited wording
sophia-wiley Feb 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/cloud/core-concepts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,13 @@ After a sync is complete, Airbyte normalizes the data. When setting up a connect

* Raw data (no normalization): Airbyte places the JSON blob version of your data in a table called `_airbyte_raw_<stream name>`
* Basic Normalization: Airbyte converts the raw JSON blob version of your data to the format of your destination. *Note: Not all destinations support normalization.*
* [dbt Cloud integration](https://docs.airbyte.com/cloud/dbt-cloud-integration): Airbyte integrates dbt Cloud, allowing you to use dbt Cloud for transforming and cleaning your data during the normalization process.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mfsiega-airbyte Is this accurate?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Phrasing suggestion: Airbyte's dbt Cloud integration allows you to use dbt Cloud for transforming and cleaning your data during the normalization process.


*Note: Custom normalization through dbt is not yet available for Airbyte Cloud.*
:::note

Normalizing data may cause an increase in your destination's compute cost. This cost will vary depending on the amount of data that is normalized and is not related to Airbyte credit usage.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andyjih is there any other info you think users should know about normalization costs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good to me!


:::

## Workspace

Expand Down
14 changes: 10 additions & 4 deletions docs/understanding-airbyte/connections/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,19 @@ A sync mode is therefore, a combination of a source and destination mode togethe

### Airbyte basic normalization

As described by the [Airbyte Protocol from the Airbyte Specifications](../airbyte-protocol.md), a replication is composed of source connectors that are transmitting data in a JSON format. It is then written as such by the destination connectors.
As described by the [Airbyte Protocol from the Airbyte Specifications](../airbyte-protocol.md), replication is composed of source connectors that are transmitting data in a JSON format. It is then written as such by the destination connectors.

On top of this replication, Airbyte provides the option to enable or disable an additional transformation step at the end of the sync called [basic normalization](../basic-normalization.md). This operation is:

* only available for destinations that support dbt execution.
* responsible for automatically generating a pipeline or a DAG of dbt transformation models to convert JSON blob objects into normalized tables.
* responsible for running and applying these dbt models to the data written in the destination.
* Only available for destinations that support dbt execution
* Automatically generates a pipeline or DAG of dbt transformation models to convert JSON blob objects into normalized tables
* Runs and applies these dbt models to the data written in the destination

:::note

Normalizing data may cause an increase in your destination's compute cost. This cost will vary depending on the amount of data that is normalized and is not related to Airbyte credit usage.

:::

### Custom sync operations

Expand Down