Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated WAL-G related docs #414

Merged
merged 25 commits into from
Mar 23, 2019
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/concepts/snapshot.md
Original file line number Diff line number Diff line change
Expand Up @@ -434,7 +434,7 @@ secret "swift-secret" created
```

```yaml
$ kubectl get secret azure-secret -o yaml
$ kubectl get secret swift-secret -o yaml
apiVersion: v1
data:
OS_AUTH_URL: PHlvdXItYXV0aC11cmw+
Expand Down
20 changes: 20 additions & 0 deletions docs/examples/postgres/snapshot/wal-postgres-azure.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: wal-postgres
namespace: demo
spec:
version: "11.1-v1"
replicas: 2
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
archiver:
storage:
storageSecretName: azure-secret
azure:
container: kubedb
20 changes: 20 additions & 0 deletions docs/examples/postgres/snapshot/wal-postgres-gcs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: wal-postgres
namespace: demo
spec:
version: "11.1-v1"
replicas: 2
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
archiver:
storage:
storageSecretName: gcs-secret
gcs:
bucket: kubedb
20 changes: 20 additions & 0 deletions docs/examples/postgres/snapshot/wal-postgres-swift.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: wal-postgres
namespace: demo
spec:
version: "11.1-v1"
replicas: 2
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
archiver:
storage:
storageSecretName: swift-secret
swift:
container: kubedb
125 changes: 115 additions & 10 deletions docs/guides/postgres/initialization/wal_source.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ namespace/demo created

## Prepare WAL Archive

We need a WAL archive to perform initialization. If you already don't have a WAL archive ready, create one by following the tutorial [here](/docs/guides/postgres/snapshot/continuous_archiving.md).
We need a WAL archive to perform initialization. If you don't have a WAL archive ready, create one by following the tutorial [here](/docs/guides/postgres/snapshot/continuous_archiving.md).

Let's populate the database so that we can verify that the initialized database has the same data. We will `exec` into the database pod and use `psql` command-line tool to create a table.

Expand Down Expand Up @@ -94,9 +94,9 @@ Now, we are ready to proceed for rest of the tutorial.

## Create Postgres with WAL source

We can initialize a new database from this archived WAL files. We have to specify the archive backend in the `spec.init.postgresWAL` field of Postgres object.
We can initialize a new database from this archived WAL files. We have to specify the archive backend in the `spec.init.postgresWAL` field of Postgres object. User can use any of S3, GCS, Azure, or Swift WAL archiver for init.

Here, the YAML of Postgres object that we are going to create in this tutorial,
Here, the YAML of Postgres object that we are going to create in this tutorial, uses S3 as the cloud backup provider.

```yaml
apiVersion: kubedb.com/v1alpha1
Expand All @@ -105,7 +105,7 @@ metadata:
name: replay-postgres
namespace: demo
spec:
version: "9.6-v2"
version: "11.1-v1"
replicas: 2
databaseSecret:
secretName: wal-postgres-auth
Expand All @@ -126,18 +126,123 @@ spec:

Here,

- `spec.init.postgresWAL` specifies storage information that will be used by `wal-g`
- `spec.init.postgresWAL` specifies storage information that will be used by `WAL-G`
- `storageSecretName` points to the Secret containing the credentials for cloud storage destination.
- `s3` points to s3 storage configuration.
- `s3` points to S3 storage configuration.
- `s3.bucket` points to the bucket name where archived WAL data is stored.
- `s3.prefix` points to the path of archived WAL data.
- `gcs` points to GCS storage configuration.

Here is a YAML of Postgres object that uses GCS as the cloud backup provider.

```yaml
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: replay-postgres
namespace: demo
spec:
version: "11.1-v1"
replicas: 2
databaseSecret:
secretName: wal-postgres-auth
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
init:
postgresWAL:
storageSecretName: azure-secret
azure:
iamrz1 marked this conversation as resolved.
Show resolved Hide resolved
bucket: kubedb
prefix: 'kubedb/demo/wal-postgres/archive'
```

Here,

- `spec.init.postgresWAL` specifies storage information that will be used by `WAL-G`
- `storageSecretName` points to the Secret containing the credentials for cloud storage destination.
- `gcs` points to storage configuration of GCS.
- `gcs.bucket` points to the bucket name where archived WAL data is stored.
- `gcs.prefix` points to the path of archived WAL data..
- `gcs.prefix` points to the path of archived WAL data.

Here is another YAML of Postgres object that uses Azure as the cloud backup provider.

```yaml
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: replay-postgres
namespace: demo
spec:
version: "11.1-v1"
replicas: 2
databaseSecret:
secretName: wal-postgres-auth
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
init:
postgresWAL:
storageSecretName: azure-secret
azure:
container: kubedb
prefix: 'kubedb/demo/wal-postgres/archive'
```

Here,

- `spec.init.postgresWAL` specifies storage information that will be used by`WAL-G`
- `storageSecretName` points to the Secret containing the credentials for cloud storage destination.
- `azure` points to Azure storage configuration.
- `azure.container` points to the container/bucket name where archived WAL data is stored.
- `azure.prefix` points to the path of archived WAL data.

And this yet another YAML of Postgres object that uses Swift as it's cloud backup provider.

```yaml
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: replay-postgres
namespace: demo
spec:
version: "11.1-v1"
replicas: 2
databaseSecret:
secretName: wal-postgres-auth
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
init:
postgresWAL:
storageSecretName: swift-secret
swift:
container: kubedb
prefix: 'kubedb/demo/wal-postgres/archive'
```

Here,

- `spec.init.postgresWAL` specifies storage information that will be used by `WAL-G`
- `storageSecretName` points to the Secret containing the credentials for cloud storage destination.
- `swift` points to Swift storage configuration.
- `swift.container` points to the container/bucket name where archived WAL data is stored.
- `swift.prefix` points to the path of archived WAL data.


User can use either s3 or gcs WAL archiver for init.

**wal-g** receives archived WAL data from a folder called `/kubedb/{namespace}/{postgres-name}/archive/`.
**wal-g** receives archived WAL data from a directory inside the bucket/container called `/kubedb/{namespace}/{postgres-name}/archive/`.

Here, `{namespace}` & `{postgres-name}` indicates Postgres object whose WAL archived data will be replayed.

Expand Down
138 changes: 138 additions & 0 deletions docs/guides/postgres/snapshot/archiving_to_azure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# Continuous Archiving to Azure
the-redback marked this conversation as resolved.
Show resolved Hide resolved

**WAL-G** is used to handle continuous archiving mechanism. Please refer to [continuous archiving in kubeDB](/docs/guides/postgres/snapshot/continuous_archiving.md) to know more about it.

## Before You Begin

To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial.

```console
$ kubectl create ns demo
namespace/demo created
```

## Create PostgreSQL with Continuous Archiving

For archiving, we need storage Secret, and storage backend information. Below is a Postgres object created with Continuous Archiving support to backup WAL files to Azure Storage.

```yaml
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: wal-postgres
namespace: demo
spec:
version: "11.1"
iamrz1 marked this conversation as resolved.
Show resolved Hide resolved
replicas: 2
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
archiver:
storage:
storageSecretName: azure-secret
gcs:
iamrz1 marked this conversation as resolved.
Show resolved Hide resolved
bucket: kubedb
```

Here,

- `spec.archiver.storage` specifies storage information that will be used by `WAL-G`
- `storage.storageSecretName` points to the Secret containing the credentials for cloud storage destination.
- `storage.azure` points to Azure storage configuration.
- `storage.azure.container` points to the bucket name used to store continuous archiving data.

**Archiver Storage Secret**

Storage Secret should contain credentials that will be used to access storage destination.

Storage Secret for **WAL-G** is needed with the following 2 keys:

| Key | Description |
| -------------------- | -------------------------------------- |
| `AZURE_ACCOUNT_NAME` | `Required`. Azure Storage account name |
| `AZURE_ACCOUNT_KEY` | `Required`. Azure Storage account key |

```console
$ echo -n '<your-azure-storage-account-name>' > AZURE_ACCOUNT_NAME
$ echo -n '<your-azure-storage-account-key>' > AZURE_ACCOUNT_KEY
$ kubectl create secret generic azure-secret \
--from-file=./AZURE_ACCOUNT_NAME \
--from-file=./AZURE_ACCOUNT_KEY
secret "azure-secret" created
```

```yaml
$ kubectl get secret azure-secret -o yaml
apiVersion: v1
data:
AZURE_ACCOUNT_KEY: PHlvdXItYXp1cmUtc3RvcmFnZS1hY2NvdW50LWtleT4=
AZURE_ACCOUNT_NAME: PHlvdXItYXp1cmUtc3RvcmFnZS1hY2NvdW50LW5hbWU+
kind: Secret
metadata:
creationTimestamp: 2017-06-28T13:27:16Z
name: azure-secret
namespace: default
resourceVersion: "6809"
selfLink: /api/v1/namespaces/default/secrets/azure-secret
uid: 80f658d1-5c05-11e7-bb52-08002711f4aa
type: Opaque
```

**Archiver Storage Backend**

To configure GCS backend, following parameters are available:

| Parameter | Description |
| ---------------------- | ------------------------------------------------------------ |
| `spec.azure.container` | `Required`. Name of Storage container |
| `spec.azure.prefix` | `Optional`. Path prefix into bucket where snapshot will be stored |

Now create this Postgres object with Continuous Archiving support.

```console
$ kubectl create -f https://raw.githubusercontent.com/kubedb/cli/0.11.0/docs/examples/postgres/snapshot/wal-postgres-azure.yaml
postgres.kubedb.com/wal-postgres created
```

When database is ready, **WAL-G** takes a base backup and uploads it to the cloud storage defined by storage backend.

Archived data is stored in a folder called `{container}/{prefix}/kubedb/{namespace}/{postgres-name}/archive/`.

You can see continuous archiving data stored in azure container.

<p align="center">
<kbd>
<img alt="continuous-archiving" src="/docs/images/postgres/wal-postgres-azure.png">
</kbd>
</p>



From the above image, you can see that the archived data is stored in a folder `kubedb/kubedb/demo/wal-postgres/archive`.

## Termination Policy

If termination policy of this `wal-postgres` is set to `WipeOut` or, If `Spec.WipeOut` of dormant database is set to `true`, then the data in cloud backend will be deleted.

The data will be intact in other scenarios.

## Cleaning up

To cleanup the Kubernetes resources created by this tutorial, run:

```console
kubectl patch -n demo pg/wal-postgres -p '{"spec":{"terminationPolicy":"WipeOut"}}' --type="merge"
kubectl delete -n demo pg/wal-postgres

kubectl delete -n demo secret/azure-secret
kubectl delete ns demo
```

## Next Steps

- Learn about initializing [PostgreSQL from WAL](/docs/guides/postgres/initialization/script_source.md) files stored in cloud.

Loading