From 09de9010c63460607f03856b8d2c832d7d5c450d Mon Sep 17 00:00:00 2001 From: Trent Fowler Date: Wed, 26 Feb 2025 10:05:35 -0700 Subject: [PATCH 1/6] Adding the document. --- .../north-eap-private-deployment.mdx | 606 ++++++++++++++++++ 1 file changed, 606 insertions(+) create mode 100644 fern/pages/deployment-options/north-eap-private-deployment.mdx diff --git a/fern/pages/deployment-options/north-eap-private-deployment.mdx b/fern/pages/deployment-options/north-eap-private-deployment.mdx new file mode 100644 index 000000000..1152ed5d9 --- /dev/null +++ b/fern/pages/deployment-options/north-eap-private-deployment.mdx @@ -0,0 +1,606 @@ +--- +title: "North EAP - Private Deployments" +slug: "docs/north-eap-private-deployments" + +hidden: true + +description: "This document describes the steps required for a POC install of North." +image: "../../assets/images/f1cc130-cohere_meta_image.jpg" +keywords: "generative AI, large language models, knowledge management, enterprise AI" + +createdAt: "Wed Feb 26 2025 09:54:00 (MST)" +updatedAt: "" +--- + + +This document describes the steps taken during a POC install. Where North is in Early Access, the install process is meant to be done live with a Cohere engineer directly available. This is a rapidly evolving product and requirements are subject to change. + + +## Requirements + +### Cluster + +- Kubernetes Cluster (tested on Kubernetes v1.30.x) + +### Hardware + +- CPU compute + - 24 vCPU cores + - 72 Gb Memory + - OpenSearch nodes should have a `sysctl` config with `vm_max_map_count=262144` +- GPU Compute + - 1x NVidia A100 or 1x NVidia H100 + - 1x NVidia A10 or better + - 2x NVidia T4 or better +- Storage (PVC) + - A minimum of 300Gb of persistent storage + +### Cluster Dependencies + +- [NVidia Device Plugin](https://github.com/NVIDIA/k8s-device-plugin) installed in the cluster +- [Stakater Reloader](https://github.com/stakater/Reloader) installed in the cluster + - We rely on `reloader` to restart pods based on modifications of certain `ConfigMap` and `Secret` resources. + +### Permissions + +- Kubernetes Cluster Admin permissions + +### External Resources / Infrastructure + +- Postgres Instance + - With a user with permission to create and manage databases for North and Compass. + - The user can create these databases beforehand to minimize the set of permissions. The required databases are `north` and `compass`. +- Redis + +### Firewall & Network Communication + +Access to the following domains: + +| Domain | Required | Purpose | +| ------ | -------- | ------- | +| `helm.cohere.com` | no* | Pulling the helm chart. * This is only required during installation, but not required in the cluster’s firewall. | +| `registry.cohere.com` | yes | Pulling Cohere’s container images. | +| `ghcr.io` | yes | Pulling public container images hosted on GitHub’s container registry. | +| `docker.io` | yes | Pulling public container images hosted on the Docker Hub container registry. | +| `api.cohere.com` | no** | Making model API calls. ** Required if using the Cohere platform as the model provider. | + +## Installation + + +The installation process documented below assumes direct access to the Kubernetes cluster with the permissions and requirements described in the Requirements section below. The same operations could be done with a GitOps tool instead of the kubectl create and kubectl apply operations. + + +1. **Confirm connection to cluster** + + ```bash + kubectl config current-context + ``` + +2. **Create a namespace for the installation** + + ```bash + kubectl create namespace cohere + + kubectl config set-context --current --namespace cohere + ``` + +3. **Install cluster dependencies** + + ```bash + helm repo add stakater https://stakater.github.io/stakater-charts + helm repo add opensearch-operator https://opensearch-project.github.io/opensearch-k8s-operator/ + helm repo update + + helm install reloader stakater/reloader --create-namespace -n reloader + helm install opensearch-operator opensearch-operator/opensearch-operator \ + -n cohere --create-namespace + ``` + +4. **Create a secret with the credentials to your external databases** +The defaults are set up to expect a secret named `credentials` in your installation namespace with the keys `postgresPassword` and `redisPassword` + + ```bash + (cat < \ + --password + + # Install the helm chart, making sure the required values are provided + helm install north oci://helm.cohere.com/north/stable/cohere-eno -n cohere \ + -f values.yaml --timeout 10m \ + --set global.config.postgres.host="" \ + --set global.config.postgres.user="" \ + --set global.config.postgres.tls.caCerts.secretName="" \ + --set global.config.postgres.tls.caCerts.secretKey="" \ + --set global.config.redis.host="" \ + --set global.config.redis.tls.caCerts.secretName="" \ + --set global.config.redis.tls.caCerts.secretKey="" \ + --set toolkit.config.publicFrontendURL="https://" \ + --set toolkit.config.publicBackendURL="https:///api" + ``` + +7. **Create the ingress routes.** + + This installation excludes setting up any ingress to the cluster, and it won’t be complete without it. + + The following routes should be defined in the `cohere` namespace: + + - `/` → `toolkit-frontend.cohere.svc.cluster.local:80` + - `/api/v1` → `toolkit-backend.cohere.svc.cluster.local:80/v1` + - `/api/internal/v1` →`toolkit-backend.cohere.svc.cluster.local:80/internal/v1` + + These routes can be set up with any desired ingress controller, allowing users to integrate their Ingress and certificate management solutions. + +8. **Validate the installation** + - Check that all pods in the `cohere` namespace are up and `Ready`. + + ```bash + kubectl get pods -n cohere --watch + ``` + + + + - Check if the UI is accessible using the hostname defined by the ingress routes in the previous step. + +## Configuration Options + +This section provides advanced configuration options to modify or harden the base installation. + +## OIDC Authentication + +The base installation described above ships with basic authentication as the login option. This is the simplest authentication option, but it is not recommended for production use cases. +We recommend setting up Open-ID Connect (OIDC) as the authentication strategy. + +It’s up to the user to create an OIDC application with an identity provider (e.g., Okta, Google, Azure, etc). Once the OIDC application is created, you must store the OIDC credentials in the cluster and configure the North Helm installation. + +Start by creating a secret in Kubernetes with the client ID and secret: + +```yaml +(cat <" + OIDC_CLIENT_SECRET: "" +EOF +) | kubectl apply -f - +``` + +Once the secret is created, you have two options: modifying the `values.yaml` directly or adding `--set` flags to the Helm installation command. Both options are described below. + +**Option 1): Edit the `values.yaml` file** + +Add these values to your `values.yaml`file. This option is preferred if you’re working with GitOps deployment tools. + +```yaml +toolkit: + config: + auth: + basic: + enabled: false + oidc: + enabled: true + clientID: + secretKeyRef: + name: "north-oidc" + key: "OIDC_CLIENT_ID" + clientSecret: + secretKeyRef: + name: "north-oidc" + key: "OIDC_CLIENT_SECRET" + wellKnownEndpoint: "" +``` + + + +**Option 2): Use Helm `--set` flags** + +Alternatively, you can add these flags to the helm installation command to configure OIDC. + +```bash +--set toolkit.config.auth.basic.enabled=false \ +--set toolkit.config.auth.oidc.enabled=true \ +--set toolkit.config.auth.oidc.clientID.secretKeyRef.name="north-oidc" \ +--set toolkit.config.auth.oidc.clientID.secretKeyRef.key="OIDC_CLIENT_ID" \ +--set toolkit.config.auth.oidc.clientSecret.secretKeyRef.name="north-oidc" \ +--set toolkit.config.auth.oidc.clientSecret.secretKeyRef.key="OIDC_CLIENT_SECRET" \ +--set toolkit.config.auth.oidc.wellKnownEndpoint="" +``` + + + +## Postgres Certificates + +**Option 1): Edit the `values.yaml` file** + +```yaml +global: + config: + postgres: + sslMode: "verify-ca" + tls: + caCerts: + secretName: "" + secretKey: "" +``` + +**Option 2): Use Helm `--set` flags** + +```yaml + --set global.config.postgres.sslMode="verify-ca" \ + --set global.config.postgres.tls.caCerts.secretName="" \ + --set global.config.postgres.tls.caCerts.secretKey="" +``` + +## Redis / Valkey Certificates + +**Option 1): Edit the `values.yaml` file** + +```yaml +global: + config: + redis: + scheme: "rediss" + connectionOptions: "" # comma separated list of connection options + tls: + caCerts: + secretName: "" + secretKey: "" +``` + +**Option 2): Use Helm `--set` flags** + +```yaml + --set global.config.redis.scheme="rediss" \ + --set global.config.redis.connectionOptions="" \ + --set global.config.redis.tls.caCerts.secretName="" \ + --set global.config.redis.tls.caCerts.secretKey="" +``` + +## External Redis / Valkey + +**Option 1): Edit the `values.yaml` file** + +```yaml +valkey: + enabled: false # disable bundled valkey instance +global: + config: + redis: + scheme: "" # redis or rediss + host: "" # redis host + port: "" # redis port + connectionOptions: "" # comma separated string with connection options + password: + secretKeyRef: + name: "" # name of the secret containing the password + key: "" # identifier of the key containing the password within the secret +``` + +**Option 2): Use Helm `--set` flags** + +```bash +--set valkey.enabled=false \ +--set global.config.redis.scheme="" \ +--set global.config.redis.host="" \ +--set global.config.redis.port="" \ +--set global.config.redis.connectionOptions="" \ +--set global.config.redis.password.secretKeyRef.name="" \ +--set global.config.redis.password.secretKeyRef.key="" +``` + +## Alternative Model Hosting Platforms + +### Cohere Platform + +**Option 1): Edit the `values.yaml` file** + +```yaml +global: + config: + cohere: + apiKey: + secretKeyRef: + name: "" + key: "" + +models: + enabled: false + +toolkit: + config: + modelDeploymentType: cohere_platform +``` + +**Option 2): Use Helm `--set` flags** + +```yaml +--set models.enabled=false \ +--set toolkit.config.modelDeploymentType=cohere_platform \ +--set global.config.cohere.apiKey.secretKeyRef.name="" \ +--set global.config.cohere.apiKey.secretKeyRef.key="" +``` + From 6fba618dca12c48ac007649469d95cf3068d89bc Mon Sep 17 00:00:00 2001 From: Trent Fowler Date: Wed, 26 Feb 2025 10:16:03 -0700 Subject: [PATCH 2/6] adding the v1 and v2 paths. --- .../north-eap-private-deployment.mdx | 18 ++++++------------ fern/v1.yml | 2 ++ fern/v2.yml | 2 ++ 3 files changed, 10 insertions(+), 12 deletions(-) diff --git a/fern/pages/deployment-options/north-eap-private-deployment.mdx b/fern/pages/deployment-options/north-eap-private-deployment.mdx index 1152ed5d9..cc446f315 100644 --- a/fern/pages/deployment-options/north-eap-private-deployment.mdx +++ b/fern/pages/deployment-options/north-eap-private-deployment.mdx @@ -415,7 +415,7 @@ The defaults are set up to expect a secret named `credentials` in your installat This section provides advanced configuration options to modify or harden the base installation. -## OIDC Authentication +### OIDC Authentication The base installation described above ships with basic authentication as the login option. This is the simplest authentication option, but it is not recommended for production use cases. We recommend setting up Open-ID Connect (OIDC) as the authentication strategy. @@ -465,10 +465,7 @@ toolkit: ``` **Option 2): Use Helm `--set` flags** @@ -486,13 +483,10 @@ Alternatively, you can add these flags to the helm installation command to confi ``` -## Postgres Certificates +### Postgres Certificates **Option 1): Edit the `values.yaml` file** @@ -515,7 +509,7 @@ global: --set global.config.postgres.tls.caCerts.secretKey="" ``` -## Redis / Valkey Certificates +### Redis / Valkey Certificates **Option 1): Edit the `values.yaml` file** @@ -540,7 +534,7 @@ global: --set global.config.redis.tls.caCerts.secretKey="" ``` -## External Redis / Valkey +### External Redis / Valkey **Option 1): Edit the `values.yaml` file** diff --git a/fern/v1.yml b/fern/v1.yml index 4c04fb860..d629026b2 100644 --- a/fern/v1.yml +++ b/fern/v1.yml @@ -286,6 +286,8 @@ navigation: path: pages/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx + - page: North Private Deployment + path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks diff --git a/fern/v2.yml b/fern/v2.yml index 174815075..e2945bf43 100644 --- a/fern/v2.yml +++ b/fern/v2.yml @@ -273,6 +273,8 @@ navigation: path: pages/v2/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx + - page: North Private Deployment + path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks From 0eea2f46001f5765af8222ea6df2070f7b107989 Mon Sep 17 00:00:00 2001 From: Trent Fowler Date: Wed, 26 Feb 2025 10:22:13 -0700 Subject: [PATCH 3/6] Unnesting. --- fern/v1.yml | 4 ++-- fern/v2.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/fern/v1.yml b/fern/v1.yml index d629026b2..2a1c23a3a 100644 --- a/fern/v1.yml +++ b/fern/v1.yml @@ -286,8 +286,8 @@ navigation: path: pages/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx - - page: North Private Deployment - path: pages/deployment-options/north-eap-private-deployment.mdx + - page: North Private Deployment + path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks diff --git a/fern/v2.yml b/fern/v2.yml index e2945bf43..5928d6333 100644 --- a/fern/v2.yml +++ b/fern/v2.yml @@ -273,8 +273,8 @@ navigation: path: pages/v2/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx - - page: North Private Deployment - path: pages/deployment-options/north-eap-private-deployment.mdx + - page: North Private Deployment + path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks From db6f6911fdbb3f78fd9e164a7f58748eaea8bf40 Mon Sep 17 00:00:00 2001 From: trentfowlercohere <141260477+trentfowlercohere@users.noreply.github.com> Date: Wed, 26 Feb 2025 15:25:37 -0700 Subject: [PATCH 4/6] Update north-eap-private-deployment.mdx :) Signed-off-by: trentfowlercohere <141260477+trentfowlercohere@users.noreply.github.com> --- fern/pages/deployment-options/north-eap-private-deployment.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fern/pages/deployment-options/north-eap-private-deployment.mdx b/fern/pages/deployment-options/north-eap-private-deployment.mdx index cc446f315..b618dc873 100644 --- a/fern/pages/deployment-options/north-eap-private-deployment.mdx +++ b/fern/pages/deployment-options/north-eap-private-deployment.mdx @@ -8,7 +8,7 @@ description: "This document describes the steps required for a POC install of No image: "../../assets/images/f1cc130-cohere_meta_image.jpg" keywords: "generative AI, large language models, knowledge management, enterprise AI" -createdAt: "Wed Feb 26 2025 09:54:00 (MST)" +createdAt: "Wed Feb 26 2025 10:54:00 (MST)" updatedAt: "" --- From 4c235b804ea7628436157392cf2d93298e692dd7 Mon Sep 17 00:00:00 2001 From: Trent Fowler Date: Thu, 27 Feb 2025 07:17:10 -0700 Subject: [PATCH 5/6] Removing the v1/v2 paths, so the hidden page doesn't display. --- fern/v1.yml | 2 -- fern/v2.yml | 3 --- 2 files changed, 5 deletions(-) diff --git a/fern/v1.yml b/fern/v1.yml index 2a1c23a3a..4c04fb860 100644 --- a/fern/v1.yml +++ b/fern/v1.yml @@ -286,8 +286,6 @@ navigation: path: pages/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx - - page: North Private Deployment - path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks diff --git a/fern/v2.yml b/fern/v2.yml index 5928d6333..f9e10adbd 100644 --- a/fern/v2.yml +++ b/fern/v2.yml @@ -272,9 +272,6 @@ navigation: - page: Cohere on Azure path: pages/v2/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) - path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx - - page: North Private Deployment - path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks From 1d28bcadaa435d026eba711ade5ffed065419561 Mon Sep 17 00:00:00 2001 From: Trent Fowler Date: Thu, 27 Feb 2025 07:27:20 -0700 Subject: [PATCH 6/6] Marking the files as hidden. --- fern/v1.yml | 3 +++ fern/v2.yml | 4 ++++ 2 files changed, 7 insertions(+) diff --git a/fern/v1.yml b/fern/v1.yml index 4c04fb860..717f6c704 100644 --- a/fern/v1.yml +++ b/fern/v1.yml @@ -286,6 +286,9 @@ navigation: path: pages/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx + - page: North Private Deployment + hidden: true + path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks diff --git a/fern/v2.yml b/fern/v2.yml index f9e10adbd..aedf185c3 100644 --- a/fern/v2.yml +++ b/fern/v2.yml @@ -272,6 +272,10 @@ navigation: - page: Cohere on Azure path: pages/v2/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) + path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx + - page: North Private Deployment + hidden: true + path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks