feat: Documentation for starting standalone schema manager #5101

grlimacan · 2025-02-27T11:06:50Z

Description

Adding documentation for stand alone schema manager
Parent epic: camunda/camunda#27883

When should this change go live?

This is a bug fix, security concern, or something that needs urgent release support. (add bug or support label)
This is already available but undocumented and should be released within a week. (add available & undocumented label)
This is on a specific schedule and the assignee will coordinate a release with the DevEx team. (create draft PR and/or add hold label)
This is part of a scheduled alpha or minor. (add alpha or minor label)
There is no urgency with this change (add low prio label)

PR Checklist

My changes are for an upcoming minor release and:
- are in the /docs directory (version 8.8).
- are in the /versioned_docs/version-8.7/ directory (version 8.7).
My changes are for an already released minor and are in a /versioned_docs directory.

I added a DRI, team, or delegate as a reviewer for technical accuracy and grammar/style:
- Engineering team review
- Technical writer review via @camunda/tech-writers unless working with an embedded writer.

github-actions · 2025-02-27T11:07:11Z

👋 🤖 🤔 Hello, @mesellings! Did you make your changes in all the right places?

These files were changed only in versioned_docs/version-8.6/. You might want to duplicate these changes in docs/.

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-privileges.md
versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

These files were changed only in versioned_docs/version-8.6/. You might want to duplicate these changes in versioned_docs/version-8.7/.

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-privileges.md
versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

You may have done this intentionally, but we wanted to point it out in case you didn't. You can read more about the versioning within our docs in our documentation guidelines.

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-privileges.md

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

grlimacan · 2025-02-27T11:24:30Z

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

+      username: camunda-app
+      password: camunda123
+    archiver:
+      ilmEnabled: false


Please note that I removed these 3 params that were present in our test-config so that standard values are used:

elsRolloverDateFormat: "yyyy-MM-dd-HH-mm" rolloverInterval: 1m waitPeriodBeforeArchiving: 1m

grlimacan · 2025-02-27T11:25:10Z

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

+      username: camunda-app
+      password: camunda123
+    archiver:
+      ilmEnabled: false


Please note that I removed these 3 params that were present in our test-config so that standard values are used:

elsRolloverDateFormat: "yyyy-MM-dd-HH-mm" rolloverInterval: 1m waitPeriodBeforeArchiving: 1m

aabouzaid · 2025-02-27T13:55:05Z

@grlimacan Here are the Helm chart values to achieve the same result (disable Schema Manager in the Helm deployment).

There are 2 cases, depending on how the user manages the customers:

Auto-generated app config by Helm chart.
Manually-managed app config by the user.

Case 1: Auto-generated app config by Helm chart.

Using Spring Boot convention using env vars to override config.

Here, we rely on the Helm auto-generated config and only provide the config needed to disable Schema Manager in the apps.

# Helm chart values file.
zeebe:
  env:
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_CREATETEMPLATE
      value: "false"
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_RETENTION_ENABLED
      value: "false"
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_RETENTION_MANAGEPOLICY
      value: "false"

tasklist:
  env:
    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_CREATESCHEMA
      value: "false"
    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_HEALTHCHECKENABLED
      value: "false"
    - name: CAMUNDA_TASKLIST_ARCHIVER_ILMENABLED
      value: "false"
    - name: CAMUNDA_TASKLIST_ARCHIVER_ILMMANAGEPOLICY
      value: "false"
    - name: CAMUNDA_TASKLIST_MIGRATION_MIGRATIONENABLED
      value: "false"

operate:
  env:
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_CREATESCHEMA
      value: "false"
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_HEALTHCHECKENABLED
      value: "false"
    - name: CAMUNDA_OPERATE_ARCHIVER_ILMENABLED
      value: "false"
    - name: CAMUNDA_OPERATE_ARCHIVER_ILMMANAGEPOLICY
      value: "false"
    - name: CAMUNDA_OPERATE_MIGRATION_MIGRATIONENABLED
      value: "false"

Case 2: - Manually-managed app config by the user.

Only if the user manages the apps config directly and don't rely on the Helm chart auto-generated config.

# Helm chart values file.

zeebe:
  configuration |
    [...] # Any other custom config.
    zeebe.broker.exporters.elasticsearch:
      className: io.camunda.zeebe.exporter.ElasticsearchExporter
      args:
        index:
          createTemplate: false
        retention:
          enabled: false
          managePolicy: false
        authentication:
          username: camunda-app
          password: camunda123
    [...] # Any other custom config. 

tasklist:
  configuration |
    [...] # Any other custom config.
    camunda.tasklist:
        elasticsearch:
          createSchema: false
          username: camunda-app
          password: camunda123
          healthCheckEnabled: false
        zeebeElasticsearch:
          username: camunda-app
          password: camunda123
        archiver:
          ilmEnabled: false
          elsRolloverDateFormat: "yyyy-MM-dd-HH-mm"
          rolloverInterval: 1m
          waitPeriodBeforeArchiving: 1m
          ilmManagePolicy: false
        migration:
          migrationEnabled: false
    [...] # Any other custom config.

operate:
  configuration |
    [...] # Any other custom config.
    camunda.operate:
        elasticsearch:
          createSchema: false
          username: camunda-app
          password: camunda123
          healthCheckEnabled: false
        zeebeElasticsearch:
          username: camunda-app
          password: camunda123
        archiver:
          ilmEnabled: false
          elsRolloverDateFormat: "yyyy-MM-dd-HH-mm"
          rolloverInterval: 1m
          waitPeriodBeforeArchiving: 1m
        migration:
          migrationEnabled: false
    [...] # Any other custom config.

grlimacan · 2025-02-27T14:31:16Z

@grlimacan Here are the Helm chart values to achieve the same result (disable Schema Manager in the Helm deployment).

There are 2 cases, depending on how the user manages the customers:

Auto-generated app config by Helm chart.
Manually-managed app config by the user.

Case 1: Auto-generated app config by Helm chart.

Using Spring Boot convention using env vars to override config.

Here, we rely on the Helm auto-generated config and only provide the config needed to disable Schema Manager in the apps.

# Helm chart values file.
zeebe:
  env:
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_CREATETEMPLATE
      value: "false"
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_RETENTION_ENABLED
      value: "false"
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_RETENTION_MANAGEPOLICY
      value: "false"

tasklist:
  env:
    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_CREATESCHEMA
      value: "false"
    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_HEALTHCHECKENABLED
      value: "false"
    - name: CAMUNDA_TASKLIST_ARCHIVER_ILMENABLED
      value: "false"
    - name: CAMUNDA_TASKLIST_ARCHIVER_ILMMANAGEPOLICY
      value: "false"
    - name: CAMUNDA_TASKLIST_MIGRATION_MIGRATIONENABLED
      value: "false"

operate:
  env:
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_CREATESCHEMA
      value: "false"
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_HEALTHCHECKENABLED
      value: "false"
    - name: CAMUNDA_OPERATE_ARCHIVER_ILMENABLED
      value: "false"
    - name: CAMUNDA_OPERATE_ARCHIVER_ILMMANAGEPOLICY
      value: "false"
    - name: CAMUNDA_OPERATE_MIGRATION_MIGRATIONENABLED
      value: "false"

Case 2: - Manually-managed app config by the user.

Only if the user manages the apps config directly and don't rely on the Helm chart auto-generated config.

# Helm chart values file.

zeebe:
  configuration |
    [...] # Any other custom config.
    zeebe.broker.exporters.elasticsearch:
      className: io.camunda.zeebe.exporter.ElasticsearchExporter
      args:
        index:
          createTemplate: false
        retention:
          enabled: false
          managePolicy: false
        authentication:
          username: camunda-app
          password: camunda123
    [...] # Any other custom config. 

tasklist:
  configuration |
    [...] # Any other custom config.
    camunda.tasklist:
        elasticsearch:
          createSchema: false
          username: camunda-app
          password: camunda123
          healthCheckEnabled: false
        zeebeElasticsearch:
          username: camunda-app
          password: camunda123
        archiver:
          ilmEnabled: false
          elsRolloverDateFormat: "yyyy-MM-dd-HH-mm"
          rolloverInterval: 1m
          waitPeriodBeforeArchiving: 1m
          ilmManagePolicy: false
        migration:
          migrationEnabled: false
    [...] # Any other custom config.

operate:
  configuration |
    [...] # Any other custom config.
    camunda.operate:
        elasticsearch:
          createSchema: false
          username: camunda-app
          password: camunda123
          healthCheckEnabled: false
        zeebeElasticsearch:
          username: camunda-app
          password: camunda123
        archiver:
          ilmEnabled: false
          elsRolloverDateFormat: "yyyy-MM-dd-HH-mm"
          rolloverInterval: 1m
          waitPeriodBeforeArchiving: 1m
        migration:
          migrationEnabled: false
    [...] # Any other custom config.

Great stuff! Thanks!! :-)

grlimacan · 2025-02-27T15:53:52Z

Hi @camunda/tech-writers ! This is an urgent PR for a change that only applies to 8.6 (to be more exact the to-be-released 8.6.10 onwards). This needs to be merged by tomorrow. Can you please make sure it gets reviewed today in the evening so that I can merge it tomorrow (European) morning?

mesellings · 2025-02-27T16:12:08Z

Hi @camunda/tech-writers ! This is an urgent PR for a change that only applies to 8.6 (to be more exact the to-be-released 8.6.10 onwards). This needs to be merged by tomorrow. Can you please make sure it gets reviewed today in the evening so that I can merge it tomorrow (European) morning?

@grlimacan I'll review this for you, and I can then merge and release for you in the morning as well seeing as I'm in the same timezone 👍 I will make changes directly to the branch considering the tight timescale if that's okay?

grlimacan · 2025-02-27T16:20:01Z

Hi @camunda/tech-writers ! This is an urgent PR for a change that only applies to 8.6 (to be more exact the to-be-released 8.6.10 onwards). This needs to be merged by tomorrow. Can you please make sure it gets reviewed today in the evening so that I can merge it tomorrow (European) morning?

@grlimacan I'll review this for you, and I can then merge and release for you in the morning as well seeing as I'm in the same timezone 👍 I will make changes directly to the branch considering the tight timescale if that's okay?

Thanks @mesellings! Yes please do make the changes straight away :-) I am just going to apply some minor changes quickly now, so just make sure to re-sync the branch. Please do not merge straight away since a technical review from Engineering is still pending

aabouzaid

I've left a couple of tiny comments in the Helm section.

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

grlimacan · 2025-02-27T16:40:20Z

@mesellings I'm done with my changes for today at least, please make your changes directly on the PR now. Thanks :-)

mesellings · 2025-02-27T16:56:26Z

@grlimacan I've completed my review for you in the comments, I haven't made them directly. None of them are blocking if you wanted us to apply them in a retrospective PR instead so we don't hold up the release tomorrow 👍

grlimacan · 2025-02-27T17:03:00Z

@grlimacan I've completed my review for you in the comments, I haven't made them directly. None of them are blocking if you wanted us to apply them in a retrospective PR instead so we don't hold up the release tomorrow 👍

Thanks @mesellings ! Could it be that you forgot to submit the comments? I don't see anything in the PR?

mesellings

Great work @grlimacan - approving so we don't block tomorrow, but I've added some readability improvements as suggestions 👍

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-privileges.md

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

mesellings · 2025-02-27T17:16:30Z

FYI @camunda/tech-writers I'm liaising/reviewing this with @grlimacan and plan is for us to get this merged and released by lunchtime (Europe) tomorrow - it might benefit from a @conceptualshark review at some point as an SME, but not urgently 👍

Szik · 2025-02-27T20:59:57Z

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

+    authentication:
+      username: camunda-admin
+      password: camunda123
+camunda:


this configuration requires tasklist and operate to know the elasticsearchs url in the form of this example:

camunda.operate.elasticsearch.url: "http://camunda-elasticsearch:9200"

and

camunda.tasklist.elasticsearch.url: "http://camunda-elasticsearch:9200"

I left this undefined on purpose, because I am only documenting the necessary changes additionally to the settings the customer already has. When left unset, in our example, all tools will use the default http://localhost:9200, however if customers have a running setup where they use different URLs, if I add this field here, this would possibly confuse them

Co-authored-by: Mark Sellings <[email protected]>

mesellings

Approved for release - great job 🚀 (I might add some non-blocking edits in a follow-up PR).

npepinpe

🚀 Some comments, but nothing major :) Please address them (accept or reject them) via comments before merging

npepinpe · 2025-02-28T06:58:18Z

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

+title: "Elasticsearch without cluster privileges"
+---
+
+In case the Camunda single application cannot access Elasticsearch with cluster privileges, it is possible to run the schema manager as a stand-alone application separate from the main application. The schema manager solely creates all necessary Elasticsearch Indices. In this case, elevated privileges are only required for the schema creation, the single application does not need to have cluster privileges to work any more.


🔧 I would omit the sentence The schema manager solely creates all necessary Elasticsearch Indices. I think it just leads to confusion (since it actually does more than this anyway), and gives unnecessary details. We can leave it at that the application creates the necessary schema for C8 to run afterwards without cluster privileges.

🔧 Similarly, we use elevated privileges and cluster privileges interchangeably, and I don;'t know if that's true? I would pick one, likely cluster level privileges, and just stick to it.

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

npepinpe · 2025-02-28T09:09:06Z

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

+
+## Setup
+
+For this setup to work, the database schema must be initialized first (step 1). This requires cluster level privileges for the database. This only needs to be executed once.


❓

This only needs to be executed once.

Wouldn't this be, this needs to be executed once on the first installation, and then every time they perform an update? Or are we not discussing updates?

Because they will eventually update, and they will need to run it again at that point.

Following this discussion it is better to leave as is to avoid confusion. We will update this text before 8.6.11

Do we have a way to ensure it will be done before the next update? Note that the next patch is coming out this Tuesday 😄

I mean, I don't expect we have schema changes until then, so it's fine. But you get the idea ;)

npepinpe · 2025-02-28T09:10:21Z

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

+### 1. Initialize the Schema Manager
+
+The schema manager is started as a separate standalone Java application and is responsible for creating and managing the database schema and applying database settings, such as retention policies for example.
+This step requires cluster level privileges for the database and only needs to be executed once per installation.


❓ Do we need to specify the required permissions? Or is there an ES equivalent, i.e. saying superuser or something (is that a thing for ES?).

npepinpe · 2025-02-28T09:11:39Z

versioned_docs/version-8.6/self-managed/concepts/elasticsearch-without-cluster-privileges.md

+
+#### 1.1 Configure Schema Manager
+
+Create an additional custom configuration for the schema manager with the following values:


🔧 The configuration below implies they want ILM. Maybe we can add a note that the configuration can be tweaked based on existing configuration (link to the respective components configuration), but that config here would be a minimal sample configuration.

I've added a note on the config file about ILM. About the existing configuration, that is what is implied with additional custom configuration

npepinpe · 2025-02-28T09:32:58Z

Oops, it was merged already 😄 Oh well, we can always iterate

github-actions · 2025-02-28T09:35:46Z

The preview environment relating to the commit d2b7730 has successfully been deployed. You can access it at https://preview.docs.camunda.cloud/pr-5101/index.html

grlimacan self-assigned this Feb 27, 2025

grlimacan added the deploy Stand up a temporary docs site with this PR label Feb 27, 2025

github-actions bot reviewed Feb 27, 2025

View reviewed changes

feat: Documentation for starting standalone schema manager

8f35dc0

grlimacan force-pushed the es-without-cluster-privileges branch from 4f97750 to 8f35dc0 Compare February 27, 2025 11:10

fix: Apply suggestions from automated code review

1472c61

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

grlimacan marked this pull request as draft February 27, 2025 11:17

grlimacan added the kind/bug Issues related with bugs in the documentation label Feb 27, 2025

grlimacan commented Feb 27, 2025

View reviewed changes

github-actions bot temporarily deployed to camunda-docs February 27, 2025 11:26 Destroyed

fix: Fixing some formatting issues

0c08cee

github-actions bot temporarily deployed to camunda-docs February 27, 2025 11:45 Destroyed

grlimacan requested review from npepinpe, lenaschoenburg and a team February 27, 2025 11:54

github-actions bot temporarily deployed to camunda-docs February 27, 2025 15:01 Destroyed

feat: Adding Helm Chart documentation

4c8b223

grlimacan force-pushed the es-without-cluster-privileges branch from d8983be to 4c8b223 Compare February 27, 2025 15:28

github-actions bot temporarily deployed to camunda-docs February 27, 2025 15:36 Destroyed

grlimacan marked this pull request as ready for review February 27, 2025 15:48

mesellings requested review from mesellings and removed request for a team February 27, 2025 16:10

Merge branch 'main' into es-without-cluster-privileges

def8773

github-actions bot temporarily deployed to camunda-docs February 27, 2025 16:22 Destroyed

aabouzaid requested changes Feb 27, 2025

View reviewed changes

feat: Implementing findings from review

d6c9bba

grlimacan requested a review from aabouzaid February 27, 2025 16:37

github-actions bot temporarily deployed to camunda-docs February 27, 2025 16:45 Destroyed

mesellings previously approved these changes Feb 27, 2025

View reviewed changes

aabouzaid previously approved these changes Feb 27, 2025

View reviewed changes

Szik reviewed Feb 27, 2025

View reviewed changes

fix: Apply suggestions from code review

ec198b3

Co-authored-by: Mark Sellings <[email protected]>

grlimacan dismissed stale reviews from aabouzaid and mesellings via ec198b3 February 28, 2025 08:28

grlimacan removed the hold This issue is parked, do not merge. label Feb 28, 2025

grlimacan added 2 commits February 28, 2025 10:34

feat: Implementing findings from review

6a78531

feat: Implementing findings from review

21430f7

github-actions bot temporarily deployed to camunda-docs February 28, 2025 08:51 Destroyed

grlimacan requested a review from mesellings February 28, 2025 09:11

Merge branch 'main' into es-without-cluster-privileges

1760e61

mesellings approved these changes Feb 28, 2025

View reviewed changes

mesellings removed the deploy Stand up a temporary docs site with this PR label Feb 28, 2025

github-actions bot temporarily deployed to camunda-docs February 28, 2025 09:30 Destroyed

npepinpe approved these changes Feb 28, 2025

View reviewed changes

mesellings merged commit 9653c0a into main Feb 28, 2025
10 checks passed

mesellings deleted the es-without-cluster-privileges branch February 28, 2025 09:31

This was referenced Feb 28, 2025

Backport StandaloneSchemaManager to 8.6/.7 camunda/camunda#27883

Closed

Take over changes made in 8.6 for the Standalone Schema Manager to the main (8.8) branch camunda/camunda#28989

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Documentation for starting standalone schema manager #5101

feat: Documentation for starting standalone schema manager #5101

grlimacan commented Feb 27, 2025 •

edited

Loading

github-actions bot commented Feb 27, 2025 •

edited

Loading

grlimacan Feb 27, 2025

grlimacan Feb 27, 2025

aabouzaid commented Feb 27, 2025

grlimacan commented Feb 27, 2025

Case 1: Auto-generated app config by Helm chart.

Case 2: - Manually-managed app config by the user.

grlimacan commented Feb 27, 2025

mesellings commented Feb 27, 2025

grlimacan commented Feb 27, 2025

aabouzaid left a comment

grlimacan commented Feb 27, 2025

mesellings commented Feb 27, 2025

grlimacan commented Feb 27, 2025

mesellings left a comment

mesellings commented Feb 27, 2025

Szik Feb 27, 2025

grlimacan Feb 28, 2025

mesellings left a comment

npepinpe left a comment

npepinpe Feb 28, 2025

npepinpe Feb 28, 2025

grlimacan Feb 28, 2025

npepinpe Feb 28, 2025

npepinpe Feb 28, 2025

npepinpe Feb 28, 2025

grlimacan Feb 28, 2025

npepinpe commented Feb 28, 2025 •

edited

Loading

github-actions bot commented Feb 28, 2025


		## Setup

		For this setup to work, the database schema must be initialized first (step 1). This requires cluster level privileges for the database. This only needs to be executed once.


		#### 1.1 Configure Schema Manager

		Create an additional custom configuration for the schema manager with the following values:

feat: Documentation for starting standalone schema manager #5101

feat: Documentation for starting standalone schema manager #5101

Conversation

grlimacan commented Feb 27, 2025 • edited Loading

Description

When should this change go live?

PR Checklist

github-actions bot commented Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aabouzaid commented Feb 27, 2025

Case 1: Auto-generated app config by Helm chart.

Case 2: - Manually-managed app config by the user.

grlimacan commented Feb 27, 2025

Case 1: Auto-generated app config by Helm chart.

Case 2: - Manually-managed app config by the user.

grlimacan commented Feb 27, 2025

mesellings commented Feb 27, 2025

grlimacan commented Feb 27, 2025

aabouzaid left a comment

Choose a reason for hiding this comment

grlimacan commented Feb 27, 2025

mesellings commented Feb 27, 2025

grlimacan commented Feb 27, 2025

mesellings left a comment

Choose a reason for hiding this comment

mesellings commented Feb 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mesellings left a comment

Choose a reason for hiding this comment

npepinpe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

npepinpe commented Feb 28, 2025 • edited Loading

github-actions bot commented Feb 28, 2025

grlimacan commented Feb 27, 2025 •

edited

Loading

github-actions bot commented Feb 27, 2025 •

edited

Loading

npepinpe commented Feb 28, 2025 •

edited

Loading