Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-cloud documentation #18872

Merged
merged 4 commits into from
Nov 4, 2022
Merged

Multi-cloud documentation #18872

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b38f8f5
added multicloud info
sophia-wiley Nov 1, 2022
bd2cee0
Update docs/cloud/getting-started-with-airbyte-cloud.md
sophia-wiley Nov 2, 2022
8023698
edited for clarity
sophia-wiley Nov 3, 2022
d9c7b48
incorporated suggestions
sophia-wiley Nov 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 24 additions & 42 deletions docs/cloud/getting-started-with-airbyte-cloud.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,44 +60,14 @@ A connection is an automated data pipeline that replicates data from a source to

Setting up a connection involves configuring the following parameters:

<table>
<tr>
<td><strong>Parameter</strong>
</td>
<td><strong>Description</strong>
</td>
</tr>
<tr>
<td>Replication frequency
</td>
<td>How often should the data sync?
</td>
</tr>
<tr>
<td>Destination Namespace and stream names
</td>
<td>Where should the replicated data be written?
</td>
</tr>
<tr>
<td>Catalog selection
</td>
<td>Which streams and fields should be replicated from the source to the destination?
</td>
</tr>
<tr>
<td>Sync mode
</td>
<td>How should the streams be replicated (read and written)?
</td>
</tr>
<tr>
<td>Optional transformations
</td>
<td>How should Airbyte protocol messages (raw JSON blob) data be converted into other data representations?
</td>
</tr>
</table>
| Parameter | Description |
|----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Replication frequency | How often should the data sync? |
| Data residency | Where should the data be processed? To choose the preferred data processing location for all of your connections, set your default [data residency](https://docs.airbyte.com/cloud/managing-airbyte-cloud#choose-your-default-data-residency). |
| Destination Namespace and stream names | Where should the replicated data be written? |
| Catalog selection | Which streams and fields should be replicated from the source to the destination? |
| Sync mode | How should the streams be replicated (read and written)? |
| Optional transformations | How should Airbyte protocol messages (raw JSON blob) data be converted into other data representations? |

For more information, see [Connections and Sync Modes](../understanding-airbyte/connections/README.md) and [Namespaces](../understanding-airbyte/namespaces.md)

Expand Down Expand Up @@ -214,7 +184,19 @@ Verify the sync by checking the logs:

## Allowlist IP address

You may need to allowlist one of our IP addresses to enable access to Airbyte:
- 34.106.109.131
- 34.106.196.165
- 34.106.60.246
Depending on your data residency location, you may need to allowlist the following IP addresses to enable access to Airbyte:

### United States and Airbyte Default
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should be explicit about saying that the US/Default IP addresses are GCP us-west-2
and the EU ip addresses are are both GCP us-west-2 and AWS eu-west-3 (Paris).

wdyt @davinchia and @malikdiarra?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree. I think it only helps us to be more specific with the info.

#### GCP region: us-west2
* 34.106.109.131
* 34.106.196.165
* 34.106.60.246

### European Union
#### AWS region: eu-west-3
* 34.106.109.131
* 34.106.196.165
* 34.106.60.246
* 13.37.4.46
* 13.37.142.60
* 35.181.124.238
65 changes: 55 additions & 10 deletions docs/cloud/managing-airbyte-cloud.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,6 @@ To remove a user from your workspace:

4. The **Remove user** dialog displays. Click **Remove**.

### Switch between multiple workspaces

To switch between workspaces:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click the current workspace name under the Airbyte logo in the navigation bar.

2. Click **View all workspaces**.

3. Click the name of the workspace you want to switch to.

### Rename a workspace

To rename a workspace:
Expand Down Expand Up @@ -84,6 +74,44 @@ You can use one or multiple workspaces with Airbyte Cloud.
| Single | You can use the same payment method for all purchases. | Credits pay for the use of resources in a workspace when you run a sync. Resource usage cannot be divided and paid for separately (for example, you cannot bill different departments in your organization for the usage of some credits in one workspace). |
| Multiple | Workspaces are independent of each other, so you can use a different payment method card for each workspace (for example, different credit cards per department in your organization). | You can use the same payment method for different workspaces, but each workspace is billed separately. Managing billing for each workspace can become complicated if you have many workspaces. |

### Switch between multiple workspaces

To switch between workspaces:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click the current workspace name under the Airbyte logo in the navigation bar.

2. Click **View all workspaces**.

3. Click the name of the workspace you want to switch to.

### Choose your default data residency
Default data residency allows you to choose where your data is processed. When you set the default data residency, it is applied to all new connections, but it does not affect existing connections.

For individual connections, you can choose a data residency that is different from the default. You can do this in the [connection settings](#choose-the-data-residency-for-a-connection) or when you create a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection).

:::note

Your data is processed in the chosen residency, but some data is still stored in the US.

:::


To choose your default data residency:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Settings** in the navigation bar.

2. In the Workspace settings sidebar, click **Default Data Residency**.

3. Click the dropdown and choose the location for your default data residency.

4. Click **Save changes**.

:::info

Depending on your network configuration, you may need to add [IP addresses](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#allowlist-ip-address) to your allowlist.

:::

## Manage Airbyte Cloud notifications

To set up Slack notifications:
Expand Down Expand Up @@ -259,6 +287,23 @@ To display **Connection State**:

The **Connection State** displays.

## Choose the data residency for a connection
To choose the data residency for an existing connection:

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Connections** in the navigation bar and then click the connection that you want to change.

The Connection page displays.

2. Click the **Settings** tab.

3. Click the **Data residency** dropdown and choose the location for your default data residency.
Copy link
Contributor

@josephkmh josephkmh Nov 4, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a note here about whether changing a connection's data residency will affect currently running syncs? @timroes mentioned this when reviewing the UI PR. I don't think currently running syncs will be affected, but I'm not 100% sure. Maybe @davinchia has an idea?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reminding me Joey - yeah updating the workspace doesn't update existing connections so we should call that out too.

@sophia-wiley one more follow up change!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay so just to make sure I understand this correctly -- If you choose a different data residency in the connection settings while a sync is running, it will not affect the currently running sync, but it will apply the new data residency to future syncs for that connection?

Copy link
Contributor

@davinchia davinchia Nov 4, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Data residency changes on a connection will only affect the next sync, and not any currently running sync.
  • Changing the workspace default residency only affects newly created connections. Already created connections have to be manually edited.

Does ^ make the behaviour clearer?


:::note

You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for all of your connections.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for all of your connections.
You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for your workspace.

Just a suggestion: this could be interpreted such that changing your default data residency affects all of your connections, which it does not.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I'll edit this in another PR


:::

## Buy credits

This section guides you through purchasing credits on Airbyte Cloud. An Airbyte [credit](https://airbyte.com/pricing) is a unit of measure used to pay for Airbyte resources when you run a sync.
Expand Down