Skip to content

Commit

Permalink
restore images
Browse files Browse the repository at this point in the history
  • Loading branch information
michel-tricot committed Dec 14, 2020
1 parent 2be5615 commit 3c048aa
Show file tree
Hide file tree
Showing 8 changed files with 50 additions and 3 deletions.
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,14 @@

![GitHub Workflow Status](https://img.shields.io/github/workflow/status/airbytehq/airbyte/Airbyte%20CI) ![License](https://img.shields.io/github/license/airbytehq/airbyte)

![](docs/.gitbook/assets/airbyte_horizontal_color_white-background.svg)

**Data integration made simple, secure and extensible.**
The new open-source standard to sync data from applications, APIs & databases to warehouses.

[](docs/deploying-airbyte/on-your-workstation.md) [](docs/deploying-airbyte/on-aws-ec2.md) [](docs/deploying-airbyte/on-gcp-compute-engine.md)
[![](docs/.gitbook/assets/deploy-locally.svg)](docs/deploying-airbyte/on-your-workstation.md) [![](docs/.gitbook/assets/deploy-on-aws.svg)](docs/deploying-airbyte/on-aws-ec2.md) [![](docs/.gitbook/assets/deploy-on-gcp.svg)](docs/deploying-airbyte/on-gcp-compute-engine.md)

![](docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines.png)

Airbyte is on a mission to make data integration pipelines a commodity.

Expand All @@ -26,7 +30,7 @@ docker-compose up

Now visit [http://localhost:8000](http://localhost:8000)

Here is a [step-by-step guide]() showing you how to load data from a sample Postgres database into another database using Airbyte, all on your computer.
Here is a [step-by-step guide](docs/tutorials/getting-started.md) showing you how to load data from a sample Postgres database into another database using Airbyte, all on your computer.

## Features

Expand Down
2 changes: 2 additions & 0 deletions docs/architecture/high-level-view.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ description: A high level view of Airbyte's components.

# High-level View

![3.048-Kilometer view](../.gitbook/assets/10-000-feet-view.png)

* `UI`: Acts as the control center for Airbyte. From the UI, you can configure new integration connections. You can also track the different syncing jobs and view logs.
* `Config Store`: Stores all the connections information \(credentials, frequency...\).
* `Scheduler Store`: Stores statuses and job information for the scheduler bookkeeping.
Expand Down
12 changes: 12 additions & 0 deletions docs/deploying-airbyte/on-aws-ec2.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,18 @@ The instructions have been tested on `Amazon Linux 2 AMI (HVM)`

* Launch a new instance

![](../.gitbook/assets/aws_ec2_launch.png)

* Select instance AMI

![](../.gitbook/assets/aws_ec2_ami.png)

* Select instance type
* For testing out Airbyte, a `t2.medium` instance is likely sufficient.
* For long-running Airbyte installations, we recommend a `t2.large` instance.

![](../.gitbook/assets/aws_ec2_instance_type.png)

* `Next: Configure Instance Details`
* You can tune parameters or keep the defaults
* `Next: Add Storage`
Expand All @@ -23,13 +29,19 @@ The instructions have been tested on `Amazon Linux 2 AMI (HVM)`
* `Next: Configure Security Groups`
* We are going to allow network for `ssh`

![](../.gitbook/assets/aws_ec2_security_group.png)

* `Review and Launch`
* `Launch`
* Create a ssh key so you can connect to the instance
* Download the key \(and don't lose it or you won't be able to connect to the instance\)

![](../.gitbook/assets/aws_ec2_ssh_key.png)

* `Launch Instances`

![](../.gitbook/assets/aws_ec2_instance_view.png)

* Wait for the instance to become `Running`

## Install environment
Expand Down
4 changes: 4 additions & 0 deletions docs/deploying-airbyte/on-gcp-compute-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,14 @@ The instructions have been tested on `Debian GNU/Linux 10 (buster)`

* Launch a new instance

![](../.gitbook/assets/gcp_ce_launch.png)

* Configure new instance
* For testing out Airbyte, an `e2.medium` instance is likely sufficient.
* For long-running Airbyte installations, we recommend a `n1-standard-2` instance.

![](../.gitbook/assets/gcp_ce_configure.png)

* `Create`

## Install environment
Expand Down
5 changes: 4 additions & 1 deletion docs/integrations/sources/facebook-marketing-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,12 @@ Visit the [Facebook Developers App hub](https://developers.facebook.com/apps/) a

From the App's Dashboard screen \(seen in the screenshot below\) enable the Marketing API for your app if it is not already setup.

![](../../.gitbook/assets/screen-shot-2020-11-03-at-9.25.21-pm.png)

### API Access Token

In the App Dashboard screen, click Marketing API --> Tools on the left sidebar. Then highlight all the available token permissions \(`ads_management`, `ads_read`, `read_insights`\) and click "Get token". A long string of characters should appear in front of you; **this is the access token.** Copy this string for use in the Airbyte UI later.

With the Ad Account ID and API access token, you should be ready to start pulling data from the Facebook Marketing API. Head to the Airbyte UI to setup your source connector!
![](../../.gitbook/assets/screen-shot-2020-11-03-at-9.35.40-pm.png)

With the Ad Account ID and API access token, you should be ready to start pulling data from the Facebook Marketing API. Head to the Airbyte UI to setup your source connector!
2 changes: 2 additions & 0 deletions docs/integrations/sources/google-sheets.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ Once you've created the Service Account, you need to explicitly give it access t

Finally, you'll need the ID of the Spreadsheet you'd like to sync. To get it, navigate to the spreadsheet in your browser, then copy the portion of the URL which comes after "/d" and before "/edit" or "/view". This is the highlighted portion of the screenshot below:

![](../../.gitbook/assets/screen-shot-2020-10-30-at-2.44.55-pm.png)

### Setting up in the Airbyte UI

The Airbyte UI will ask for two things:
Expand Down
12 changes: 12 additions & 0 deletions docs/tutorials/airbyte-config-persistence.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,15 +151,21 @@ docker cp airbyte-server:/data $TUTORIAL_DIR/my-setup

Head back to [http://localhost:8000/](http://localhost:8000/) and add more connectors. Here is an example of configuration from an API source:

![](../.gitbook/assets/airbyte_config_persistence_1.png)

and a local file destination:

![](../.gitbook/assets/airbyte_config_persistence_2.png)

## Run a Sync job

* once the source and destination are created
* the catalog and frequency can be configured
* then run the "Sync Now" button
* finally inspect logs in the UI

![](../.gitbook/assets/airbyte_config_persistence_3.png)

## Exploring Logs folders

We can read from the lines reported in the logs the working directory that is being used to run the synchronization process from.
Expand Down Expand Up @@ -356,10 +362,16 @@ Here are some examples of public API CSV:
https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv
```

![](../.gitbook/assets/airbyte_config_persistence_4.png)

And a local Postgres Database:

![](../.gitbook/assets/airbyte_config_persistence_5.png)

After setting up the connectors, we can trigger the sync and study the logs:

![](../.gitbook/assets/airbyte_config_persistence_6.png)

Since we wiped the workspace volume and restarted the Airbyte Server, notice that the process ran in the `/tmp/workspace/5/0` as well but the logs for ExchangeRate are gone...

## Export and customize Normalization step with DBT
Expand Down
8 changes: 8 additions & 0 deletions docs/tutorials/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,14 @@ Once you see an Airbyte banner, the UI is ready to go at [http://localhost:8000/

You should see an onboarding page. Enter your email if you want updates about Airbyte and continue.

![](../.gitbook/assets/airbyte_get-started.png)

## 2. Set up your first connection

Now you will see a wizard that allows you choose the data you want to send through Airbyte.

![](../.gitbook/assets/02_set-up-sources.png)

As of our alpha launch, we have one database source \(Postgres\) and two API sources \(an exchange rate API and the Stripe API\). We're currently building an integration framework that makes it easy to create sources and destinations, so you should expect many more soon. Please reach out to us if you need a specific connector or would like to help build one.

For now, we will start out with a Postgres source and destination.
Expand Down Expand Up @@ -72,12 +76,16 @@ DB Name: postgres

After adding the destination, you can choose what tables and columns you want to sync.

![](../.gitbook/assets/03_set-up-connection.png)

For this demo, we recommend leaving the defaults and selecting "Every 5 Minutes" as the frequency. Click `Set Up Connection` to finish setting up the sync.

## 3. Check the logs of your first sync

You should now see a list of sources with the source you just added. Click on it to find more information about your connection. This is the page where you can update any settings about this source and how it syncs. There should be a `Completed` job under the history section. If you click on that run, it will show logs from that run.

![](../.gitbook/assets/04_source-details.png)

One of biggest problems we've seen in tools like Fivetran is the lack of visibility when debugging. In Airbyte, allowing full log access and the ability to debug and fix connector problems is one of our highest priorities. We'll be working hard to make these logs accessible and understandable.

## 4. Check if the syncing actually worked
Expand Down

0 comments on commit 3c048aa

Please sign in to comment.