diff --git a/fern/pages/deployment-options/north-eap-private-deployment.mdx b/fern/pages/deployment-options/north-eap-private-deployment.mdx new file mode 100644 index 000000000..b618dc873 --- /dev/null +++ b/fern/pages/deployment-options/north-eap-private-deployment.mdx @@ -0,0 +1,600 @@ +--- +title: "North EAP - Private Deployments" +slug: "docs/north-eap-private-deployments" + +hidden: true + +description: "This document describes the steps required for a POC install of North." +image: "../../assets/images/f1cc130-cohere_meta_image.jpg" +keywords: "generative AI, large language models, knowledge management, enterprise AI" + +createdAt: "Wed Feb 26 2025 10:54:00 (MST)" +updatedAt: "" +--- + + +This document describes the steps taken during a POC install. Where North is in Early Access, the install process is meant to be done live with a Cohere engineer directly available. This is a rapidly evolving product and requirements are subject to change. + + +## Requirements + +### Cluster + +- Kubernetes Cluster (tested on Kubernetes v1.30.x) + +### Hardware + +- CPU compute + - 24 vCPU cores + - 72 Gb Memory + - OpenSearch nodes should have a `sysctl` config with `vm_max_map_count=262144` +- GPU Compute + - 1x NVidia A100 or 1x NVidia H100 + - 1x NVidia A10 or better + - 2x NVidia T4 or better +- Storage (PVC) + - A minimum of 300Gb of persistent storage + +### Cluster Dependencies + +- [NVidia Device Plugin](https://github.com/NVIDIA/k8s-device-plugin) installed in the cluster +- [Stakater Reloader](https://github.com/stakater/Reloader) installed in the cluster + - We rely on `reloader` to restart pods based on modifications of certain `ConfigMap` and `Secret` resources. + +### Permissions + +- Kubernetes Cluster Admin permissions + +### External Resources / Infrastructure + +- Postgres Instance + - With a user with permission to create and manage databases for North and Compass. + - The user can create these databases beforehand to minimize the set of permissions. The required databases are `north` and `compass`. +- Redis + +### Firewall & Network Communication + +Access to the following domains: + +| Domain | Required | Purpose | +| ------ | -------- | ------- | +| `helm.cohere.com` | no* | Pulling the helm chart. * This is only required during installation, but not required in the cluster’s firewall. | +| `registry.cohere.com` | yes | Pulling Cohere’s container images. | +| `ghcr.io` | yes | Pulling public container images hosted on GitHub’s container registry. | +| `docker.io` | yes | Pulling public container images hosted on the Docker Hub container registry. | +| `api.cohere.com` | no** | Making model API calls. ** Required if using the Cohere platform as the model provider. | + +## Installation + + +The installation process documented below assumes direct access to the Kubernetes cluster with the permissions and requirements described in the Requirements section below. The same operations could be done with a GitOps tool instead of the kubectl create and kubectl apply operations. + + +1. **Confirm connection to cluster** + + ```bash + kubectl config current-context + ``` + +2. **Create a namespace for the installation** + + ```bash + kubectl create namespace cohere + + kubectl config set-context --current --namespace cohere + ``` + +3. **Install cluster dependencies** + + ```bash + helm repo add stakater https://stakater.github.io/stakater-charts + helm repo add opensearch-operator https://opensearch-project.github.io/opensearch-k8s-operator/ + helm repo update + + helm install reloader stakater/reloader --create-namespace -n reloader + helm install opensearch-operator opensearch-operator/opensearch-operator \ + -n cohere --create-namespace + ``` + +4. **Create a secret with the credentials to your external databases** +The defaults are set up to expect a secret named `credentials` in your installation namespace with the keys `postgresPassword` and `redisPassword` + + ```bash + (cat < \ + --password + + # Install the helm chart, making sure the required values are provided + helm install north oci://helm.cohere.com/north/stable/cohere-eno -n cohere \ + -f values.yaml --timeout 10m \ + --set global.config.postgres.host="" \ + --set global.config.postgres.user="" \ + --set global.config.postgres.tls.caCerts.secretName="" \ + --set global.config.postgres.tls.caCerts.secretKey="" \ + --set global.config.redis.host="" \ + --set global.config.redis.tls.caCerts.secretName="" \ + --set global.config.redis.tls.caCerts.secretKey="" \ + --set toolkit.config.publicFrontendURL="https://" \ + --set toolkit.config.publicBackendURL="https:///api" + ``` + +7. **Create the ingress routes.** + + This installation excludes setting up any ingress to the cluster, and it won’t be complete without it. + + The following routes should be defined in the `cohere` namespace: + + - `/` → `toolkit-frontend.cohere.svc.cluster.local:80` + - `/api/v1` → `toolkit-backend.cohere.svc.cluster.local:80/v1` + - `/api/internal/v1` →`toolkit-backend.cohere.svc.cluster.local:80/internal/v1` + + These routes can be set up with any desired ingress controller, allowing users to integrate their Ingress and certificate management solutions. + +8. **Validate the installation** + - Check that all pods in the `cohere` namespace are up and `Ready`. + + ```bash + kubectl get pods -n cohere --watch + ``` + + + + - Check if the UI is accessible using the hostname defined by the ingress routes in the previous step. + +## Configuration Options + +This section provides advanced configuration options to modify or harden the base installation. + +### OIDC Authentication + +The base installation described above ships with basic authentication as the login option. This is the simplest authentication option, but it is not recommended for production use cases. +We recommend setting up Open-ID Connect (OIDC) as the authentication strategy. + +It’s up to the user to create an OIDC application with an identity provider (e.g., Okta, Google, Azure, etc). Once the OIDC application is created, you must store the OIDC credentials in the cluster and configure the North Helm installation. + +Start by creating a secret in Kubernetes with the client ID and secret: + +```yaml +(cat <" + OIDC_CLIENT_SECRET: "" +EOF +) | kubectl apply -f - +``` + +Once the secret is created, you have two options: modifying the `values.yaml` directly or adding `--set` flags to the Helm installation command. Both options are described below. + +**Option 1): Edit the `values.yaml` file** + +Add these values to your `values.yaml`file. This option is preferred if you’re working with GitOps deployment tools. + +```yaml +toolkit: + config: + auth: + basic: + enabled: false + oidc: + enabled: true + clientID: + secretKeyRef: + name: "north-oidc" + key: "OIDC_CLIENT_ID" + clientSecret: + secretKeyRef: + name: "north-oidc" + key: "OIDC_CLIENT_SECRET" + wellKnownEndpoint: "" +``` + + + +**Option 2): Use Helm `--set` flags** + +Alternatively, you can add these flags to the helm installation command to configure OIDC. + +```bash +--set toolkit.config.auth.basic.enabled=false \ +--set toolkit.config.auth.oidc.enabled=true \ +--set toolkit.config.auth.oidc.clientID.secretKeyRef.name="north-oidc" \ +--set toolkit.config.auth.oidc.clientID.secretKeyRef.key="OIDC_CLIENT_ID" \ +--set toolkit.config.auth.oidc.clientSecret.secretKeyRef.name="north-oidc" \ +--set toolkit.config.auth.oidc.clientSecret.secretKeyRef.key="OIDC_CLIENT_SECRET" \ +--set toolkit.config.auth.oidc.wellKnownEndpoint="" +``` + + + +### Postgres Certificates + +**Option 1): Edit the `values.yaml` file** + +```yaml +global: + config: + postgres: + sslMode: "verify-ca" + tls: + caCerts: + secretName: "" + secretKey: "" +``` + +**Option 2): Use Helm `--set` flags** + +```yaml + --set global.config.postgres.sslMode="verify-ca" \ + --set global.config.postgres.tls.caCerts.secretName="" \ + --set global.config.postgres.tls.caCerts.secretKey="" +``` + +### Redis / Valkey Certificates + +**Option 1): Edit the `values.yaml` file** + +```yaml +global: + config: + redis: + scheme: "rediss" + connectionOptions: "" # comma separated list of connection options + tls: + caCerts: + secretName: "" + secretKey: "" +``` + +**Option 2): Use Helm `--set` flags** + +```yaml + --set global.config.redis.scheme="rediss" \ + --set global.config.redis.connectionOptions="" \ + --set global.config.redis.tls.caCerts.secretName="" \ + --set global.config.redis.tls.caCerts.secretKey="" +``` + +### External Redis / Valkey + +**Option 1): Edit the `values.yaml` file** + +```yaml +valkey: + enabled: false # disable bundled valkey instance +global: + config: + redis: + scheme: "" # redis or rediss + host: "" # redis host + port: "" # redis port + connectionOptions: "" # comma separated string with connection options + password: + secretKeyRef: + name: "" # name of the secret containing the password + key: "" # identifier of the key containing the password within the secret +``` + +**Option 2): Use Helm `--set` flags** + +```bash +--set valkey.enabled=false \ +--set global.config.redis.scheme="" \ +--set global.config.redis.host="" \ +--set global.config.redis.port="" \ +--set global.config.redis.connectionOptions="" \ +--set global.config.redis.password.secretKeyRef.name="" \ +--set global.config.redis.password.secretKeyRef.key="" +``` + +## Alternative Model Hosting Platforms + +### Cohere Platform + +**Option 1): Edit the `values.yaml` file** + +```yaml +global: + config: + cohere: + apiKey: + secretKeyRef: + name: "" + key: "" + +models: + enabled: false + +toolkit: + config: + modelDeploymentType: cohere_platform +``` + +**Option 2): Use Helm `--set` flags** + +```yaml +--set models.enabled=false \ +--set toolkit.config.modelDeploymentType=cohere_platform \ +--set global.config.cohere.apiKey.secretKeyRef.name="" \ +--set global.config.cohere.apiKey.secretKeyRef.key="" +``` + diff --git a/fern/v1.yml b/fern/v1.yml index 4c04fb860..717f6c704 100644 --- a/fern/v1.yml +++ b/fern/v1.yml @@ -286,6 +286,9 @@ navigation: path: pages/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx + - page: North Private Deployment + hidden: true + path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks diff --git a/fern/v2.yml b/fern/v2.yml index 174815075..aedf185c3 100644 --- a/fern/v2.yml +++ b/fern/v2.yml @@ -273,6 +273,9 @@ navigation: path: pages/v2/deployment-options/cohere-on-microsoft-azure.mdx - page: Cohere on Oracle Cloud Infrastructure (OCI) path: pages/deployment-options/oracle-cloud-infrastructure-oci.mdx + - page: North Private Deployment + hidden: true + path: pages/deployment-options/north-eap-private-deployment.mdx - section: Tutorials contents: - page: Cookbooks