Cluster configuration updates

assembly/guide-configs/concepts/clusters.json

@@ -43,10 +43,22 @@
       "type": "h3",
       "id": "concepts/what-is-a-retention-period"
     },
+    {
+      "type": "h2",
+      "id": "concepts/what-is-persistence"
+    },
+    {
+      "type": "h2",
+      "id": "concepts/what-is-visibility"
+    },
     {
       "type": "h2",
       "id": "concepts/what-is-archival"
     },
+    {
+      "type": "h2",
+      "id": "concepts/what-is-cluster-configuration"
+    },
     {
       "type": "h2",
       "id": "concepts/what-is-multi-cluster-replication"

docs-src/clusters/how-to-set-up-archival.md

@@ -89,6 +89,8 @@ The following table showcases acceptable values for each configuration and what
 | `namespaceDefaults.archival.history.state`     | `enabled`, `disabled`                                                              | Default state of the Archival feature whenever a new Namespace is created without specifying the Archival state.             |
 | `namespaceDefaults.archival.history.URI`       | Valid URI                                                                          | Must be a URI of the file store location and match a schema that correlates to a provider.                                   |
 
+Additional resources: [Cluster configuration reference](/references/configuration).
+
 #### Namespace creation
 
 Although Archival is configured at the cluster level, it operates independently within each Namespace.

docs-src/concepts/what-is-a-temporal-cluster.md

@@ -2,128 +2,27 @@
 id: what-is-a-temporal-cluster
 title: What is a Temporal Cluster?
 sidebar_label: Temporal Cluster
-description: A Temporal Cluster is the Temporal Server paired with persistence.
+description: A Temporal Cluster is a Temporal Server paired with a Persistence store.
 tags:
   - term
   - explanation
 ---
 
-A Temporal Cluster is the group of services, known as the [Temporal Server](/concepts/what-is-the-temporal-server), combined with persistence stores, that together act as a component of the Temporal Platform.
+A Temporal Cluster is the group of services, known as the [Temporal Server](/concepts/what-is-the-temporal-server), combined with [Persistence stores](/concepts/what-is-persistence), that together act as a component of the Temporal Platform.
 
 - [How to quickly install a Temporal Cluster for testing and development](/kb/all-the-ways-to-run-a-cluster)
 - [Cluster deployment guide](/cluster-deployment-guide)
 
 ![A Temporal Cluster (Server + persistence)](/diagrams/temporal-cluster.svg)
 
-### Persistence
-
-A Temporal Cluster's only required dependency for basic operation is a database.
-Multiple types of databases are supported.
-
-![Persistence](/diagrams/temporal-database.svg)
-
-The database stores the following types of data:
-
-- Tasks: Tasks to be dispatched.
-- State of Workflow Executions:
-  - Execution table: A capture of the mutable state of Workflow Executions.
-  - History table: An append only log of Workflow Execution History Events.
-- Namespace metadata: Metadata of each Namespace in the Cluster.
-- Visibility data: Enables operations like "show all running Workflow Executions".
-  For production environments, we recommend using Elasticsearch.
-
-An Elasticsearch database must be added to enable [Advanced Visibility](/concepts/what-is-advanced-visibility) on Temporal Server versions 1.19.1 and earlier.
-
-With Temporal Server version 1.20 and later, Advanced Visibility features are available on SQL databases like MySQL (version 8.0.17 and later), PostgreSQL (version 12 and later), SQLite (v3.31.0 and later) and Elasticsearch.
-
-#### Dependency versions
-
-Temporal tests compatibility by spanning the **minimum** and **maximum** stable non-EOL major versions for each supported database.
-As of time of writing, these specific versions are used in our test pipelines and actively tested before we release any version of Temporal:
-
-- **Cassandra v3.11 and v4.0**
-- **PostgreSQL v10.18 and v13.4**
-- **MySQL v5.7 and v8.0** (specifically 8.0.19+ due to a bug)
-
-We update these support ranges once a year.
-The release notes of each Temporal Server declare when we plan to drop support for database versions reaching End of Life.
-
-- Because Temporal Server primarily relies on core database functionality, we do not expect compatibility to break often.
-  Temporal has no opinions on database upgrade paths; as long as you can upgrade your database according to each project's specifications, Temporal should work with any version within supported ranges.
-- We do not run tests with vendors like Vitess and CockroachDB, so you rely on their compatibility claims if you use them.
-  Feel free to discuss them with fellow users [in our forum](https://community.temporal.io/).
-- Temporal also supports SQLite v3.x persistence, but this is meant only for development and testing, not production usage.
-
-### Monitoring and observation
-
-Temporal emits metrics by default in a format that is supported by Prometheus.
-Monitoring and observing those metrics is optional.
-Any metrics software that supports the same format can be used. Currently, we test with the following Prometheus and Grafana versions:
-
-- **Prometheus >= v2.0**
-- **Grafana >= v2.5**
-
-### Visibility
+<!-- ### Visibility
+Commenting this out because it is out of place. Using the what is visibility concept topic in the guide instead.
+Also these details are covered in the Visibility store setup under cluster deployment.
 
 Temporal has built-in [Visibility](/concepts/what-is-visibility) features.
 To enhance this feature, Temporal supports an [integration with Elasticsearch](/clusters/how-to-integrate-elasticsearch-into-a-temporal-cluster).
 
 - Elasticsearch v8 is supported from Temporal version 1.18.0 onwards
 - Elasticsearch v7.10 is supported from Temporal version 1.7.0 onwards
 - Elasticsearch v6.8 is supported up to Temporal version 1.17.x
-- Elasticsearch v6.8 and v7.10 versions are explicitly supported with AWS Elasticsearch
-
-### mTLS encryption
-
-Temporal supports Mutual Transport Layer Security (mTLS) as a method of encrypting network traffic between services within a Temporal Cluster, or between application processes and a Cluster.
-
-Mutual TLS can be enabled in Temporal’s [TLS configuration](/references/configuration#tls).
-This configuration can be passed through `WithConfig` or `WithConfigLoader`.
-
-This configuration includes two sections that serve to separate intra-cluster and external traffic. That way, different certificates and settings can be used to encrypt each section of traffic:
-
-- `internode`: configuration for encrypting communication between nodes within the Cluster.
-- `frontend`: configuration for encrypting the Frontend's public endpoints
-
-### Temporal Client connections
-
-A client's network access can be limited by using certificates issued by a specific Certificate Authority (CA).
-
-To restrict access to Temporal Cluster endpoints, use the `clientCAFiles` or `clientCAData` property and the `requireClientAuth` property.
-These properties can be specified in both the `internode` and `frontend` sections of the [mTLS configuration](/references/configuration#tls).
-
-#### Server name specification
-
-Specify the `serverName` in the `client` section of your mTLS configuration to prevent spoofing and [MITM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack).
-
-Entering a value for `serverName` enables established connections to authenticate the endpoint.
-This ensures that the server certificate presented to any connected client has the specified server name in its CN property.
-
-This measure can be used for `internode` and `frontend` endpoints.
-
-For more information on mTLS configuration, refer to our [TLS configuration guide](/references/configuration#tls).
-
-### Auth
-
-**Authentication** is the process of verifying users who want to access your application are actually the users you want accessing it.
-**Authorization** is the verification of applications and data that a user on your Cluster or application has access to.
-
-Temporal has several authentication protocols that can be set to restrict access to your data.
-These protocols address three areas: servers, client connections, and users.
-
-Server attacks can be prevented by specifying `serverName` in the `client` section of your mTLS configuration.
-This can be done for both `frontend` and `internode` endpoints.
-
-Client connections can be restricted to certain endpoints by requiring certificates from a specific CA.
-Modify the `clientCaFiles`, `clientCaData`, and `requireClientAuth` properties in the `internode` and `frontend` sections of the mTLS configuration.
-
-User access can be restricted through extensibility points and plugins.
-When implemented, the `frontend` invokes the plugin before executing the requested operation.
-
-Temporal offers two plugin interfaces for API call authentication and authorization.
-
-- [`ClaimMapper`](/concepts/what-is-a-claimmapper-plugin)
-- [`Authorizer`](/concepts/what-is-an-authorizer-plugin)
-
-The logic of both plugins can be customized to fit a variety of use cases.
-When provided, the frontend invokes the implementation of the plugins before running the requested operation.
+- Elasticsearch v6.8 and v7.10 versions are explicitly supported with AWS Elasticsearch -->

docs-src/concepts/what-is-cluster-configuration.md

@@ -0,0 +1,127 @@
+---
+id: what-is-cluster-configuration
+title: What is Cluster configuration?
+sidebar_label: Cluster configuration
+description: Cluster Configuration is the setup and configuration details of your Temporal Cluster, defined using YAML.
+tags:
+  - term
+  - explanation
+---
+
+Cluster configuration is the setup and configuration details of your self-hosted Temporal Cluster, defined using YAML.
+You must define your Cluster configuration when setting up your Temporal Cluster.
+
+Cluster configuration is composed of two types of configurations: Static configuration and Dynamic configuration.
+
+### Static configuration
+
+Static configuration contains details of how the Cluster should be set up. The static configuration is read just once and used to configure service nodes container/pod at startup.
+Depending on how you want to deploy your self-hosted Temporal Cluster, your static configuration must contain details for setting up:
+
+- Temporal Services - Frontend, History, Matching, Worker
+- Membership ports for the Temporal Services
+- Persistence (including Shard count), Visibility and Advanced Visibility, Archival store setups.
+- TLS, Auth
+- Server log level
+- Metrics
+- Cluster Metadata
+- Dynamic Config Client
+
+Static configuration values cannot be changed at runtime.
+Some values, such as Metrics configuration or Server log level can be changed in the static configuration but require restarting the Cluster for the changes to take effect.
+
+For details on static configuration keys, see [Cluster configuration reference](/references/configuration).
+
+### Dynamic configuration
+
+Dynamic configuration is the ability to change the behavior of Temporal Server without having to redeploy the server images.
+
+Some configuration fields are truly dynamic; that is, changes to those values take effect immediately.
+Other configuration fields require a server restart because they are checked only at server startup (such as thread pool size).
+
+For details on dynamic configuration keys, see [Dynamic configuration reference](/references/dynamic-configuration).
+
+### Temporal Cluster security configuration
+
+Secure your Temporal Cluster (self-hosted and Temporal Cloud) by encrypting your network communication and setting authentication and authorization protocols for API calls.
+
+For details on setting up your Temporal Cluster security, see [Temporal Platform security features](/security).
+
+#### mTLS encryption
+
+Temporal supports Mutual Transport Layer Security (mTLS) to encrypt network traffic between services within a Temporal Cluster, or between application processes and a Cluster.
+
+On self-hosted Temporal Clusters, configure mTLS in the `tls` section of the [Cluster configuration](/references/configuration#tls).
+mTLS configuration is a [static configuration](#static-configuration) property.
+
+You can then use either [`WithConfig`](/references/server-options#withconfig) or [`WithConfigLoader`](/references/server-options#withconfigloader) server options to start your Temporal Cluster with this configuration.
+
+The mTLS configuration includes two sections that serve to separate communication within a Temporal Cluster and client calls made from your application the Cluster.
+
+- `internode`: configuration for encrypting communication between nodes within the Cluster.
+- `frontend`: configuration for encrypting the Frontend's public endpoints
+
+Setting mTLS for `internode` and `frontend` separately allows you to use different certificates and settings to encrypt each section of traffic.
+
+#### Using certificates for Client connections
+
+Use CA certificates to authenticate client connections to your Temporal Cluster.
+
+On Temporal Cloud, you can [set your CA certificates in your Temporal Cloud settings](/cloud-context/get-started-certificates) and use the end-entity certificates in your client calls.
+
+On self-hosted Temporal Clusters, you can restrict access to Temporal Cluster endpoints by using the `clientCAFiles` or `clientCAData` property and the [`requireClientAuth`](/references/configuration#tls) property in your Cluster configuration.
+These properties can be specified in both the `internode` and `frontend` sections of the [mTLS configuration](/references/configuration#tls).
+For details, see the [tls confiuration reference](/references/configuration#tls).
+
+##### Server name specification
+
+On self-hosted Temporal Clusters, you can specify the `serverName` in the `client` section of your mTLS configuration to prevent spoofing and [MITM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack).
+
+Entering a value for `serverName` enables established connections to authenticate the endpoint.
+This ensures that the server certificate presented to any connected client has the specified server name in its CN property.
+
+This measure can be used for `internode` and `frontend` endpoints.
+
+For more information on mTLS configuration, see [tls configuration reference](/references/configuration#tls).
+
+#### Auth
+
+<!-- commenting this very generic explanation out. Can include it back in if everyone feels strongly.
+ **Authentication** is the process of verifying users who want to access your application are actually the users you want accessing it.
+**Authorization** is the verification of applications and data that a user on your Cluster or application has access to. -->
+
+Temporal provides authentication protocols that can be set to restrict access to your data.
+These protocols address three areas: servers, client connections, and users.
+
+Temporal offers two plugin interfaces for API call authentication and authorization.
+
+- [`ClaimMapper`](/concepts/what-is-a-claimmapper-plugin)
+- [`Authorizer`](/concepts/what-is-an-authorizer-plugin)
+
+The logic of both plugins can be customized to fit a variety of use cases.
+When provided, the Frontend Service invokes the implementation of the plugins before running the requested operation.
+
+## Monitoring and observation
+
+You can monitor and observe performance by using the metrics emitted by your self-hosted Temporal Cluster or by Temporal Cloud.
+
+Temporal emits metrics by default in a format that is supported by Prometheus.
+Any metrics software that supports the same format can be used.
+Currently, we test with the following Prometheus and Grafana versions:
+
+- **Prometheus >= v2.0**
+- **Grafana >= v2.5**
+
+Temporal Cloud emits metrics through a Prometheus HTTP API endpoint which can be directly used as a Prometheus data source in Grafana or to query and export Cloud metrics to any observability platform.
+
+On self-hosted Temporal Clusters, expose Prometheus endpoints in your Cluster configuration and configure Prometheus to scrape metrics from the endpoints. You can then set up your observability platform (such as Grafana) to use this as a datasource.
+
+Metrics reference:
+
+- [Temporal Cloud metrics reference](/cloud/how-to-monitor-temporal-cloud-metrics)
+- [Temporal Cluster OSS metrics reference](/references/cluster-metrics)
+
+For details on setting up metrics on Temporal Cloud and self-hosted Clusters, see:
+
+- [Set up Grafana with Temporal Cloud observability to view metrics](/kb/prometheus-grafana-setup-cloud#data-sources-configuration-for-temporal-cloud-and-sdk-metrics-in-grafana)
+- [Set up Prometheus and Grafana to view SDK and self-hosted Cluster metrics](/kb/prometheus-grafana-setup)