Multi-cloud documentation #18872

sophia-wiley · 2022-11-02T19:51:09Z

Users are able to:

Choose a default data residency (settings).
Choose data residency when creating a new connection.
Choose data residency in connection settings.
See IP addresses for EU, US, and Airbyte Default.

The documentation in this PR includes:

Set up a connection (Getting Started with Airbyte Cloud)
Allowlist IP addresses (Getting Started with Airbyte Cloud)
Choose your default data residency (Managing Airbyte Cloud)
Choose data residency for a connection (Managing Airbyte Cloud)

docs/cloud/getting-started-with-airbyte-cloud.md

Co-authored-by: Joey Marshment-Howell <[email protected]>

andyjih · 2022-11-02T21:13:31Z

@sophia-wiley One thought here: I think it might be useful to discuss the split between the Control Plane and the Data Plane. Currently the Data Residency option only changes where the data is processed e.g. the Data Plane, so all other data will still live in the Control Plane. Here's a Whimsical that lays out roughly what the split looks like. We can ask @Upmitt to make a prettier version of the visuals as well.

andyjih · 2022-11-02T21:14:21Z

I also wonder if perhaps the Architecture Overview page could use an update.

sophia-wiley · 2022-11-03T20:32:42Z

@andyjih Sounds good! I will add the info about the split between the control plane and the data plane to the Architecture Overview Page.

malikdiarra

I agree with Andy that the architecture diagram could use an update but I think we should address that separately.

malikdiarra · 2022-11-04T01:05:06Z

docs/cloud/getting-started-with-airbyte-cloud.md

+| Parameter                              | Description                                                                                                                                                                                                                                                           |
+|----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Replication frequency                  | How often should the data sync?                                                                                                                                                                                                                                       |
+| Data residency                         | Where is the data processing location for this connection? To choose the preferred data processing location for all of your connections, set your default [data residency](https://docs.airbyte.com/cloud/managing-airbyte-cloud#choose-your-default-data-residency). |


That might be a bit of a nit but I find the wording here a bit odd: "Where is the data processing location for this connection". I'm not sure I would understand what this means if I were a customer (I haven't checked our documentation for a while, so my opinion might be off.) Every other entry uses the word should which really conveys that those options are directive for Airbyte Cloud.

I would go with something like this instead: "Where should Airbyte Cloud data synchronization workflow run"

I agree this sounds weird.

Maybe something even simpler - "Where do I want Airbyte to process my data?"

Great points! I will change the wording here

malikdiarra · 2022-11-04T01:37:11Z

@andyjih Sounds good! I will add the info about the split between the control plane and the data plane to the Architecture Overview Page.

I do not think this is the best page to update, the concept of data residency is important mainly for Airbyte Cloud. While nothing is blocking someone from using it in OSS, there is additional documentation we probably need to provide to fully support that. I would avoid making that diagram more complicated just for that.

davinchia

One minor comment!

andyjih · 2022-11-04T03:49:39Z

docs/cloud/getting-started-with-airbyte-cloud.md

- 34.106.60.246
+Depending on your data residency location, you may need to allowlist the following IP addresses to enable access to Airbyte:
+
+### United States and Airbyte Default


I wonder if we should be explicit about saying that the US/Default IP addresses are GCP us-west-2
and the EU ip addresses are are both GCP us-west-2 and AWS eu-west-3 (Paris).

wdyt @davinchia and @malikdiarra?

agree. I think it only helps us to be more specific with the info.

andyjih · 2022-11-04T03:51:19Z

docs/cloud/managing-airbyte-cloud.md

+### Choose your default data residency
+Default data residency allows you to choose where your data is processed. When you set the default data residency, it applies that data residency to all new connections, but it will not affect existing connections. 
+
+For individual connections, you can choose a data residency that is different from the default. You can do this in the [connection settings](#choose-the-data-residency-for-a-connection) or when you create a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection). Your data will not leave the chosen data residency for your connection.


I think this is an area where we might want to talk about the control plane vs. the data plane. Because this setting doesn't necessarily mean where your data is stored, but it's about where your data is processed.

that's a good point - we don't highlight the difference between the control/data plane today i.e. while data is processed in the specific region, configuration information is still stored in the US.

this means that users need to be careful to make sure they don't choose a sensitive cursor.

Andy, do you think this is worth calling out? Or is it too 'scary' to do so?

I think it's worth mentioning!

I agree that the fact that data is processed in one region and stored in another needs to be mentioned. I'm wondering if I should delete the last line "Your data will not leave the chosen data residency for your connection" and add a note like "Your data is processed in the chosen residency, but some data is still stored in the US." We could also list out types of data (like configuration information) we store in the US if that would help the users.

Any thoughts on that?

should delete the last line "Your data will not leave the chosen data residency for your connection" and add a note like "Your data is processed in the chosen residency, but some data is still stored in the US."

I like that. I would like the note to be a markdown note so it jumps out.

One tactical point, we don't have to wordsmith too much in this PR. We can merge this in for now and iterate on subsequent PRs.

@andyjih @davinchia Do you think that a note like this would have enough info about data plane and control plane? Is there any other info you think the users will need to know? (I'll edit wording after we know what info we want to include):

Your data is processed on a data plane in the chosen residency, but some data, like data associated with the sync mode, cursor, and primary key, is still stored in the US on our control plane.

That's pretty good. I would also add: "Because of this, please do not configure uses columns whose data must stay within chosen residency as cursor and primary keys configuration."

andyjih · 2022-11-04T03:52:59Z

@andyjih Sounds good! I will add the info about the split between the control plane and the data plane to the Architecture Overview Page.

I do not think this is the best page to update, the concept of data residency is important mainly for Airbyte Cloud. While nothing is blocking someone from using it in OSS, there is additional documentation we probably need to provide to fully support that. I would avoid making that diagram more complicated just for that.

Do you think we should talk about the control plane/data plane split on this multi-cloud doc? I agree that it's Cloud-specific, but I do think it's important for people to know that some data is still stored in the control plane and where the data is actually processed is in the data plane.

josephkmh

Looks good to me! Made a couple minor comments.

josephkmh · 2022-11-04T15:19:08Z

docs/cloud/managing-airbyte-cloud.md

+
+2. Click the **Settings** tab. 
+
+3. Click the **Data residency** dropdown and choose the location for your default data residency.


Should we add a note here about whether changing a connection's data residency will affect currently running syncs? @timroes mentioned this when reviewing the UI PR. I don't think currently running syncs will be affected, but I'm not 100% sure. Maybe @davinchia has an idea?

Thanks for reminding me Joey - yeah updating the workspace doesn't update existing connections so we should call that out too.

@sophia-wiley one more follow up change!

Okay so just to make sure I understand this correctly -- If you choose a different data residency in the connection settings while a sync is running, it will not affect the currently running sync, but it will apply the new data residency to future syncs for that connection?

Data residency changes on a connection will only affect the next sync, and not any currently running sync.

Changing the workspace default residency only affects newly created connections. Already created connections have to be manually edited.

Does ^ make the behaviour clearer?

josephkmh · 2022-11-04T15:21:49Z

docs/cloud/managing-airbyte-cloud.md

+
+:::note 
+
+You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for all of your connections. 


Suggested change

You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for all of your connections.

You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for your workspace.

Just a suggestion: this could be interpreted such that changing your default data residency affects all of your connections, which it does not.

Good catch! I'll edit this in another PR

added multicloud info

b38f8f5

github-actions bot added the area/documentation Improvements or additions to documentation label Nov 2, 2022

josephkmh reviewed Nov 2, 2022

View reviewed changes

docs/cloud/getting-started-with-airbyte-cloud.md Outdated Show resolved Hide resolved

Update docs/cloud/getting-started-with-airbyte-cloud.md

bd2cee0

Co-authored-by: Joey Marshment-Howell <[email protected]>

edited for clarity

8023698

sophia-wiley requested review from andyjih, malikdiarra and davinchia November 3, 2022 20:30

malikdiarra approved these changes Nov 4, 2022

View reviewed changes

davinchia approved these changes Nov 4, 2022

View reviewed changes

andyjih reviewed Nov 4, 2022

View reviewed changes

incorporated suggestions

d9c7b48

sophia-wiley merged commit 77f3e3b into master Nov 4, 2022

sophia-wiley deleted the multicloud_documentation branch November 4, 2022 06:18

josephkmh mentioned this pull request Nov 4, 2022

🪟 🎉 Multi-cloud: edit connection & workspace data geographies in the UI #18611

Merged

josephkmh reviewed Nov 4, 2022

View reviewed changes

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-cloud documentation #18872

Multi-cloud documentation #18872

sophia-wiley commented Nov 2, 2022

andyjih commented Nov 2, 2022

andyjih commented Nov 2, 2022

sophia-wiley commented Nov 3, 2022

malikdiarra left a comment

malikdiarra Nov 4, 2022

davinchia Nov 4, 2022

sophia-wiley Nov 4, 2022

malikdiarra commented Nov 4, 2022

davinchia left a comment

andyjih Nov 4, 2022

davinchia Nov 4, 2022

andyjih Nov 4, 2022

davinchia Nov 4, 2022

andyjih Nov 4, 2022

sophia-wiley Nov 4, 2022

davinchia Nov 4, 2022

sophia-wiley Nov 4, 2022

davinchia Nov 4, 2022 •

edited

Loading

andyjih commented Nov 4, 2022

josephkmh left a comment •

edited

Loading

josephkmh Nov 4, 2022 •

edited

Loading

davinchia Nov 4, 2022

sophia-wiley Nov 4, 2022

davinchia Nov 4, 2022 •

edited

Loading

josephkmh Nov 4, 2022

sophia-wiley Nov 4, 2022


		2. Click the Settings tab.

		3. Click the Data residency dropdown and choose the location for your default data residency.


		:::note

		You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for all of your connections.

Multi-cloud documentation #18872

Multi-cloud documentation #18872

Conversation

sophia-wiley commented Nov 2, 2022

andyjih commented Nov 2, 2022

andyjih commented Nov 2, 2022

sophia-wiley commented Nov 3, 2022

malikdiarra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

malikdiarra commented Nov 4, 2022

davinchia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davinchia Nov 4, 2022 • edited Loading

Choose a reason for hiding this comment

andyjih commented Nov 4, 2022

josephkmh left a comment • edited Loading

Choose a reason for hiding this comment

josephkmh Nov 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davinchia Nov 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davinchia Nov 4, 2022 •

edited

Loading

josephkmh left a comment •

edited

Loading

josephkmh Nov 4, 2022 •

edited

Loading

davinchia Nov 4, 2022 •

edited

Loading