Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes docs: external logs with S3 #34621

Merged
merged 2 commits into from
Feb 5, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 122 additions & 11 deletions docs/deploying-airbyte/on-kubernetes-via-helm.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,28 +167,139 @@ Before upgrading the chart update values.yaml as stated above and then run:
- Perform upgrade of chart by running `helm upgrade %release_name% airbyte/airbyte --set auth.rootPassword=$ROOT_PASSWORD`
- If you get an error about setting the auth.rootPassword, then you forgot to update the `values.yaml` file

### Custom logging and jobs configuration
### External Logs

Starting from `0.39.37-alpha` if you've configured logging yourself using `logging or jobs` section of `values.yaml` file, you need to update your configuration so you can continue to use your custom logging and jobs configuration.
::info
This was tested using [Airbyte Helm Chart Version 0.50.13](https://artifacthub.io/packages/helm/airbyte/airbyte/0.50.13) and S3 logs only.
Previous or newer version can change how to setup the external logs.
:::

Simply declare global value in `values.yaml` file and move everything related to logging and jobs under that section like in the example bellow:
Create a file called `airbyte-logs-secrets.yaml` to store the AWS Keys and other informations:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: airbyte-logs-secrets
type: Opaque
stringData:
AWS_KEY: <AWS_KEY>
AWS_SECRET_KEY: <AWS_SECRET_KEY>
S3_LOG_BUCKET: <BUCKET_NAME>
S3_LOG_BUCKET_REGION: <REGION>
```
Run `kubectl apply -f airbyte-logs-secrets.yaml -n <NAMESPACE>` to create the secret in the namespace you're using Airbyte.
This file contains more than just the keys but it needs for now. Future updates will make the configuration easier.

```text
Change the global section to use `S3` external logs.
```yaml
global:
logging:
%your_logging_options_here%
jobs:
%your_jobs_options_here%
# <...>
state:
# -- Determines which state storage will be utilized; "MINIO", "S3", or "GCS"
storage:
type: "S3"
# <...>
logs:
accessKey:
password: ""
existingSecret: "airbyte-logs-secrets"
existingSecretKey: "AWS_KEY"
secretKey:
password: ""
existingSecret: "airbyte-logs-secrets"
existingSecretKey: "AWS_SECRET_KEY"
# <...>
storage:
type: "S3"

minio:
# Change from true to false
enabled: false
nodeSelector: {}
tolerations: []
affinity: {}
```
You can try to use `GCS` or `External Minio` but both weren't tested yet. Feel free to run tests and update the documentation.

After updating `values.yaml` simply upgrade your chart by running command:
Add extra env variables to the following blocks:
```yaml
worker:
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_KEY
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_SECRET_KEY
- name: STATE_STORAGE_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_KEY
- name: STATE_STORAGE_S3_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_SECRET_KEY
- name: STATE_STORAGE_S3_BUCKET_NAME
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: S3_LOG_BUCKET
- name: STATE_STORAGE_S3_REGION
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: S3_LOG_BUCKET_REGION
```

```shell
helm upgrade -f path/to/values.yaml %release_name% airbyte/airbyte
and also edit the server block:

```yaml
server:
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_KEY
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_SECRET_KEY
- name: STATE_STORAGE_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_KEY
- name: STATE_STORAGE_S3_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_SECRET_KEY
- name: STATE_STORAGE_S3_BUCKET_NAME
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: S3_LOG_BUCKET
- name: STATE_STORAGE_S3_REGION
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: S3_LOG_BUCKET_REGION
```

Than run:
`helm upgrade --install %RELEASE_NAME% airbyte/airbyte -n <NAMESPACE> --values /path/to/values.yaml --version 0.50.13`

### External Airbyte Database


::info
This was tested using [Airbyte Helm Chart Version 0.50.13](https://artifacthub.io/packages/helm/airbyte/airbyte/0.50.13).
Previous or newer version can change how the external database can be configured.
Expand Down
Loading