feat: Documentation for starting standalone schema manager

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-privileges.md

@@ -7,6 +7,10 @@ If you implement Camunda 8 with Elasticsearch as a service provider, you must co
 
 ## Cluster privileges
 
+:::note Running Elasticsearch with limited cluster privileges
+In case the application cannot be granted cluster privileges, it is possible to run the schema manager as a stand-alone application separate from the main application. With this setup, the single application does not need to have cluster privileges. For more details please refer to [running application without cluster privileges](./elasticsearch-without-cluster-privileges.md).
+:::
+
 - `monitor` - necessary for health check
 - `manage_index_templates` to create and manage index schema on start up, if they don't already exist in Elasticsearch.
 - _Optional_ `manage_ilm` - required only when ILM is enabled

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

@@ -0,0 +1,282 @@
+---
+id: elasticsearch-without-cluster-privileges
+title: "Elasticsearch without cluster privileges"
+---
+
+In case the Camunda single application cannot access Elasticsearch with cluster privileges, it is possible to run the schema manager as a stand-alone application separate from the main application. The schema manager solely creates all necessary Elasticsearch Indices. In this case, elevated privileges are only required for the schema creation, the single application does not need to have cluster privileges to work any more.
+
+:::note Database Support
+This feature is only available from 8.6.10 on and is also only supported for Elasticsearch installations (no OpenSearch support).
+:::
+
+:::note Essential privileges required by the single application
+Index level privilege of at least `manage` is still required for the camunda single application to work properly
+:::
+
+## Setup
+
+For this setup to work, the database schema needs to be initialized first (step 1). This requires cluster level privileges for the database. This only needs to be executed once.
+Once the schema is initialized, the application can be started without cluster level privileges (step 2). The steps are described in detail below.
+
+### 1. Initialize the schema manager
+
+The schema manager is started as a separate standalone java application and is responsible for creating and managing the database schema and applying database settings, such as e.g. retention policies.
+This step requires cluster level privileges for the database and only needs to be executed once per installation.
+
+#### 1.1 Configure Schema Manager
+
+Create an additional custom configuration for the schema manager with the following values
+
+```
+zeebe.broker.exporters.elasticsearch:
+  className: io.camunda.zeebe.exporter.ElasticsearchExporter
+  args:
+    index:
+      createTemplate: true
+    retention:
+      enabled: true
+    # Example assuming an existing user called 'camunda-admin' who has 'superuser' privileges
+    authentication:
+      username: camunda-admin
+      password: camunda123
+camunda:
+  operate:
+    elasticsearch:
+      # Example assuming an existing user called 'camunda-admin' who has 'superuser' privileges
+      username: camunda-admin
+      password: camunda123
+      healthCheckEnabled: false
+    archiver:
+      ilmEnabled: true
+  tasklist:
+    elasticsearch:
+      # Example assuming an existing user called 'camunda-admin' who has 'superuser' privileges
+      username: camunda-admin
+      password: camunda123
+      healthCheckEnabled: false
+    archiver:
+      ilmEnabled: true
+
+```
+
+#### 1.2 Start the Schema Manager with the config above
+
+Start the java application `schema` (or `schema.bat` for Windows) provided in the `bin` folder of the delivered jar file. The schema manager will create the necessary indices and templates in the database and apply the respective settings.
+Assuming the configuration above was saved in a file named `schema-manager.yaml`, you can start the application with the following command:
+
+```
+SPRING_CONFIG_ADDITIONALLOCATION=/path/to/schema-manager.yaml ./bin/schema
+```
+
+Verify that the application executed successfully.
+
+### 2. Start the camunda single application
+
+The camunda single application can now be started without cluster level privileges. The application will connect to the database and use the schema created by the schema manager.
+
+#### 2.1 Ensure that an Elasticsearch user with sufficient privileges exists
+
+The application requires a database user with at least `manage` privileges on the indices it is supposed to work with.
+
+Feel free to use an existing user with the required privileges. Alternatively the required privileges can be assigned to an example user called `camunda-app` with the following request to the Elasticsearch REST API:
+
+```
+PUT _security/role/read_write_role
+{
+  "indices": [
+    {
+      "names": [
+        "*"
+      ],
+      "privileges": [
+        "read",
+        "write",
+        "view_index_metadata"
+      ],
+      "allow_restricted_indices": false
+    },
+    {
+      "names": [
+        "operate-*",
+        "tasklist-*",
+        "zeebe-*"
+      ],
+      "privileges": [
+        "manage"
+      ],
+      "allow_restricted_indices": false
+    }
+  ],
+  "applications": [],
+  "run_as": [],
+  "metadata": {},
+  "transient_metadata": {
+    "enabled": true
+  }
+}
+```
+
+Then assign the user to the role defined above (e.g. if Elasticsearch is running on docker, this can be achieved with the following command):
+
+```
+docker exec -t elasticsearch elasticsearch-users useradd camunda-app -p camunda123
+docker exec -t elasticsearch elasticsearch-users roles camunda-app -a read_write_role
+```
+
+#### 2.2 Configure the Camunda Single Application
+
+Create a configuration for the Camunda single application with the values below. This essentially disables schema creation for the app.
+
+```
+zeebe.broker.exporters.elasticsearch:
+  className: io.camunda.zeebe.exporter.ElasticsearchExporter
+  args:
+    index:
+      createTemplate: false
+    retention:
+      enabled: false
+      managePolicy: false
+    # Example assuming an existing user called 'camunda-app' with the privileges described in 2.1
+    authentication:
+      username: camunda-app
+      password: camunda123
+camunda:
+  tasklist:
+    elasticsearch:
+      createSchema: false
+      username: camunda-app
+      password: camunda123
+      healthCheckEnabled: false
+    zeebeElasticsearch:
+      # Example assuming an existing user called 'camunda-app' with the privileges described in 2.1
+      username: camunda-app
+      password: camunda123
+    archiver:
+      ilmEnabled: false
+      ilmManagePolicy: false
+    migration:
+      migrationEnabled: false
+  operate:
+    elasticsearch:
+      createSchema: false
+      username: camunda-app
+      password: camunda123
+      healthCheckEnabled: false
+    zeebeElasticsearch:
+      # Example assuming an existing user called 'camunda-app' with the privileges described in 2.1
+      username: camunda-app
+      password: camunda123
+    archiver:
+      ilmEnabled: false
+    migration:
+      migrationEnabled: false
+```
+
+#### 2.3 Start the application with the config above
+
+#### Starting the application from the jar file
+
+Start the java application `camunda` (or `camunda.bat` for Windows) provided in the `bin` folder of the delivered jar file.
+Assuming the configuration above was saved in a file named `application-custom.yaml`, you can start the application with the following command:
+
+```
+SPRING_CONFIG_ADDITIONALLOCATION=/path/to/application-custom.yaml ./bin/camunda
+```
+
+#### Starting the application using Helm charts
+
+##### Case 1: Auto-generated app config by Helm chart
+
+Using [Spring Boot convention](https://docs.spring.io/spring-boot/reference/features/external-config.html#features.external-config.typesafe-configuration-properties.relaxed-binding.environment-variables) environment variables can be used to override configuration.
+
+These are the Helm values needed to disable the schema manager in the Camunda apps.
+
+```
+# Helm chart values file.
+zeebe:
+  env:
+    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_CREATETEMPLATE
+      value: "false"
+    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_RETENTION_ENABLED
+      value: "false"
+    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_RETENTION_MANAGEPOLICY
+      value: "false"
+
+tasklist:
+  env:
+    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_CREATESCHEMA
+      value: "false"
+    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_HEALTHCHECKENABLED
+      value: "false"
+    - name: CAMUNDA_TASKLIST_ARCHIVER_ILMENABLED
+      value: "false"
+    - name: CAMUNDA_TASKLIST_ARCHIVER_ILMMANAGEPOLICY
+      value: "false"
+    - name: CAMUNDA_TASKLIST_MIGRATION_MIGRATIONENABLED
+      value: "false"
+
+operate:
+  env:
+    - name: CAMUNDA_OPERATE_ELASTICSEARCH_CREATESCHEMA
+      value: "false"
+    - name: CAMUNDA_OPERATE_ELASTICSEARCH_HEALTHCHECKENABLED
+      value: "false"
+    - name: CAMUNDA_OPERATE_ARCHIVER_ILMENABLED
+      value: "false"
+    - name: CAMUNDA_OPERATE_MIGRATION_MIGRATIONENABLED
+      value: "false"
+```
+
+##### Case 2: - Manually-managed app config by the user
+
+In case the application configurations are managed directly and don't rely on the Helm chart auto-generated configuration.
+
+```
+# Helm chart values file.
+
+zeebe:
+  configuration |
+    [...] # Any other custom config.
+    zeebe.broker.exporters.elasticsearch:
+      className: io.camunda.zeebe.exporter.ElasticsearchExporter
+      args:
+        index:
+          createTemplate: false
+        retention:
+          enabled: false
+          managePolicy: false
+    [...] # Any other custom config.
+
+tasklist:
+  configuration |
+    [...] # Any other custom config.
+    camunda.tasklist:
+      elasticsearch:
+        createSchema: false
+        healthCheckEnabled: false
+      archiver:
+        ilmEnabled: false
+        ilmManagePolicy: false
+      migration:
+        migrationEnabled: false
+    [...] # Any other custom config.
+
+operate:
+  configuration |
+    [...] # Any other custom config.
+    camunda.operate:
+      elasticsearch:
+        createSchema: false
+        healthCheckEnabled: false
+      archiver:
+        ilmEnabled: false
+      migration:
+        migrationEnabled: false
+    [...] # Any other custom config.
+```
+
+## Limitations
+
+- Please note that this feature is only available for the Camunda `8.6.10` version and above.
+- It only works for installations using Elasticsearch.
+- Camunda Optimize cannot be executed with this setup.

versioned_sidebars/version-8.6-sidebars.json

@@ -1929,6 +1929,7 @@
         "self-managed/concepts/multi-tenancy",
         "self-managed/concepts/mapping-rules",
         "self-managed/concepts/elasticsearch-privileges",
+        "self-managed/concepts/elasticsearch-without-cluster-privileges",
         "self-managed/concepts/opensearch-privileges"
       ]
     },