[#21963] YSQL: Configure clockbound agent (#24434)

1. Changes to manual deployment configuration with additional details added to clock sync setup.
2. Changes to database configuration after the system is setup with clockbound systemd service.
3. Changes to read restart error doc on additional recommendation about using the new clock.

docs/content/preview/architecture/transactions/read-restart-error.md

@@ -74,6 +74,7 @@ How does YugabyteDB prevent this clock skew anomaly?
 
 You can handle and mitigate read restart errors using the following techniques:
 
+- {{<tags/feature/tp>}} Configure [highly accurate clocks](../../../deploy/manual-deployment/system-config#set-up-time-synchronization).
 - Implement retry logic in the application. Application retries can help mitigate read restart errors. Moreover, a statement or a transaction may fail in other ways such as transaction conflicts or infrastructure failures. Therefore, a retry mechanism is strongly recommended for a cloud-native, distributed database such as YugabyteDB.
 
   While implementing application retries is the best long-term approach, there are a few short-term solutions you can use in the interim.

docs/content/preview/deploy/checklist.md

@@ -17,8 +17,8 @@ A YugabyteDB cluster consists of two distributed services - the [YB-TServer](../
 
 - YugabyteDB supports both x86 and ARM (aarch64) CPU architectures.
 - YugabyteDB is supported on a variety of [operating systems](../../reference/configuration/operating-systems/). For production workloads, the recommended operating systems are AlmaLinux 8 and RHEL 8.
-- The appropriate system limits should be set using [`ulimit`](../manual-deployment/system-config/#ulimits) on each node running a YugabyteDB server.
-- [NTP or chrony](../manual-deployment/system-config/#ntp) should be used to synchronize time among the machines.
+- The appropriate system limits should be set using [`ulimit`](../manual-deployment/system-config/#set-ulimits) on each node running a YugabyteDB server.
+- [Chrony](../manual-deployment/system-config#set-up-time-synchronization) should be used to synchronize time among the machines.
 
 ## Replication
 

docs/content/preview/deploy/kubernetes/multi-cluster/gke/helm-chart.md

@@ -36,7 +36,7 @@ The YugabyteDB Helm chart has been tested with the following software versions:
 - GKE running Kubernetes 1.20 or later with nodes such that a total of 12 CPU cores and 45 GB RAM can be allocated to YugabyteDB. This can be three nodes with 4 CPU core and 15 GB RAM allocated to YugabyteDB. `n1-standard-8` is the minimum instance type that meets these criteria.
 - Helm 3.4 or later.
 - YugabyteDB Docker image (yugabytedb/yugabyte) 2.1.0 or later.
-- For optimal performance, ensure you have set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#ulimits) on each node in your Kubernetes cluster.
+- For optimal performance, ensure you have set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#set-ulimits) on each node in your Kubernetes cluster.
 
 The following steps show how to meet these prerequisites:
 

docs/content/preview/deploy/kubernetes/multi-zone/eks/helm-chart.md

@@ -32,7 +32,7 @@ The YugabyteDB Helm chart has been tested with the following software versions:
 - Amazon EKS running Kubernetes 1.18 (or later) with nodes such that a total of 12 CPU cores and 45 GB RAM can be allocated to YugabyteDB. This can be three nodes with 4 CPU core and 15 GB RAM allocated to YugabyteDB. `m5.2xlarge` is the minimum AWS EC2 instance type that meets these criteria.
 - Helm 3.4 or later
 - YugabyteDB docker image (yugabytedb/yugabyte) 2.1.0 or later
-- For optimal performance, ensure you've set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#ulimits) on each node in your Kubernetes cluster.
+- For optimal performance, ensure you've set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#set-ulimits) on each node in your Kubernetes cluster.
 
 The following steps show how to meet these prerequisites.
 

docs/content/preview/deploy/kubernetes/multi-zone/gke/helm-chart.md

@@ -32,7 +32,7 @@ The YugabyteDB Helm Chart has been tested with the following software versions:
 - GKE running Kubernetes 1.18 (or later) with nodes such that a total of 12 CPU cores and 45 GB RAM can be allocated to YugabyteDB. This can be three nodes with 4 CPU core and 15 GB RAM allocated to YugabyteDB. `n1-standard-8` is the minimum instance type that meets these criteria.
 - Helm 3.4 or later
 - YugabyteDB docker image (`yugabytedb/yugabyte`) 2.1.0 or later
-- For optimal performance, ensure you've set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#ulimits) on each node in your Kubernetes cluster.
+- For optimal performance, ensure you've set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#set-ulimits) on each node in your Kubernetes cluster.
 
 The following steps show how to meet these prerequisites.
 

docs/content/preview/deploy/kubernetes/single-zone/gke/helm-chart.md

@@ -45,7 +45,7 @@ The YugabyteDB Helm chart has been tested with the following software versions:
 - GKE running Kubernetes 1.20 or later. The Helm chart you use to install YugabyteDB creates three YB-Master and three YB-TServers, each with 2 CPU cores, for a total of 12 CPU cores. This means you need a Kubernetes cluster with more than 12 CPU cores. If the cluster contains three nodes, then each node should have more than 4 cores.
 
 - Helm 3.4 or later.
-- For optimal performance, ensure you set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#ulimits) on each node in your Kubernetes cluster.
+- For optimal performance, ensure you set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#set-ulimits) on each node in your Kubernetes cluster.
 
 The following steps show how to meet these prerequisites:
 

docs/content/preview/deploy/kubernetes/single-zone/oss/helm-chart.md

@@ -44,7 +44,7 @@ The YugabyteDB Helm chart has been tested with the following software versions:
 - Kubernetes 1.20 or later with nodes such that a total of 12 CPU cores and 18 GB RAM can be allocated to YugabyteDB. This can be three nodes with 4 CPU core and 6 GB RAM allocated to YugabyteDB.
 - Helm 3.4 or later.
 - YugabyteDB Docker image (yugabytedb/yugabyte) 2.1.0 or later
-- For optimal performance, ensure you have set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#ulimits) on each node in your Kubernetes cluster.
+- For optimal performance, ensure you have set the appropriate [system limits using `ulimit`](../../../../manual-deployment/system-config/#set-ulimits) on each node in your Kubernetes cluster.
 
 Confirm that `helm` and `kubectl` are configured correctly, as follows:
 

docs/content/preview/deploy/manual-deployment/start-masters.md

@@ -46,6 +46,8 @@ The number of comma-separated addresses in `--master_addresses` should equal the
 
 You can specify multiple directories using the [`--fs_data_dirs`](../../../reference/configuration/yb-master/#fs-data-dirs) flag. Replace the [`--rpc_bind_addresses`](../../../reference/configuration/yb-master/#rpc-bind-addresses) value with the private IP address of the host, and set the `placement_cloud`, `placement_region`, and `placement_zone` values appropriately. For single zone deployment, use the same value for the `placement_zone` flag.
 
+{{<tags/feature/tp>}} Highly accurate clocks can be configured by specifying `--time_source=clockbound`. Requires [system configuration](../system-config#set-up-time-synchronization).
+
 For the full list of configuration flags, see the [YB-Master reference](../../../reference/configuration/yb-master/).
 
 ## Run YB-Master servers with configuration file

docs/content/preview/deploy/manual-deployment/start-tservers.md

@@ -48,6 +48,8 @@ $ ./bin/yb-tserver \
 
 Provide all of the master addresses using the [`--tserver_master_addrs`](../../../reference/configuration/yb-tserver/#tserver-master-addrs) flag. Replace the [`--rpc_bind_addresses`](../../../reference/configuration/yb-tserver/#rpc-bind-addresses) value with the private IP address of the host, and set the `placement_cloud`, `placement_region`, and `placement_zone` values appropriately. For single zone deployment, use the same value for the `--placement_zone` flag.
 
+{{<tags/feature/tp>}} Highly accurate clocks can be configured by specifying `--time_source=clockbound`. Requires [system configuration](../system-config#set-up-time-synchronization).
+
 For the full list of configuration flags, see the [YB-TServer reference](../../../reference/configuration/yb-tserver/).
 
 ## Run YB-TServer with configuration file

docs/content/preview/deploy/manual-deployment/system-config.md

@@ -13,29 +13,84 @@ type: docs
 
 Perform the following configuration on each node in the cluster:
 
-- ntp or chrony
-- ulimits
-- transparent hugepages
+- set up time synchronization
+- set ulimits
+- enable transparent hugepages
 
 Keep in mind that, although YugabyteDB is PostgreSQL compatible and runs a postgres process, it is not a PostgreSQL distribution. The PostgreSQL it runs doesn't need the same OS and system resources that open source PostgreSQL requires. For this reason, the kernel configuration requirements are different.
 
 In particular, the main YugabyteDB process, the YB-TServer, is multi-threaded. As a result, you don't need to modify settings for shared memory and inter-process communication (IPC), because there is no inter-process communication or shared memory in a multi-threaded process model (all memory is shared by the same process).
 
-## ntp
+## Set up time synchronization
 
-If your instance does not have public Internet access, make sure the ntp package is installed:
+YugabyteDB relies on clock synchronization to guarantee consistency in distributed transactions. Chrony is the prefered NTP implementation for clock synchronization.
+
+### Install Chrony
+
+To install chrony, run:
 
 ```sh
-$ sudo yum install -y ntp
+$ sudo yum install -y chrony
 ```
 
-As of CentOS 8, `ntp` is no longer available and has been replaced by `chrony`. To install, run:
+### Configure Precision Time Protocol
+
+{{<tags/feature/tp>}} Precision Time Protocol (PTP) is a network protocol designed for highly accurate time synchronization across devices in a network. PTP provides microsecond-level accuracy. PTP relies on a PTP Hardware Clock (PHC), a dedicated physical clock device that enhances time synchronization accuracy.
+
+Currently, PTP is only available for AWS. To check if your AWS instance supports PTP and PHC, see [AWS PTP Hardware Clock](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-ec2-ntp.html#connect-to-the-ptp-hardware-clock).
+
+Configure PTP using the `configure_ptp.sh` script in the bin directory of your YugabyteDB home directory as follows:
 
 ```sh
-$ sudo yum install -y chrony
+sudo bash ./bin/configure_ptp.sh
+```
+
+### Configure ClockBound
+
+{{<tags/feature/tp>}} [ClockBound](https://github.com/aws/clock-bound) is an open source daemon that allows you to compare timestamps to determine order for events and transactions, independent of an instance's geographic location. ClockBound provides a strict interval in which the reference time (true time) exists.
+
+Although optional, configuring ClockBound improves clock accuracy by several orders of magnitude. ClockBound requires chrony and can be used in conjunction with PTP.
+
+Configure ClockBound using the `configure_clockbound.sh` script in the bin directory of your YugabyteDB home directory as follows:
+
+```sh
+sudo bash ./bin/configure_clockbound.sh
 ```
 
-## ulimits
+After configuring ClockBound, you must configure the [YB-TServer](../start-tservers/) and [YB-Master](../start-masters/) servers with the `time_source=clockbound` flag.
+
+If the ClockBound agent is configured with PTP, use a more aggressive clock error estimate such as `clockbound_clock_error_estimate_usec=100`.
+
+### Verify ClockBound configuration
+
+Verify that ClockBound is configured properly using the following command:
+
+```sh
+systemctl status clockbound
+```
+
+A correctly configured ClockBound service reports no errors.  The following shows example output with PTP enabled:
+
+```sh
+● clockbound.service - ClockBound
+     Loaded: loaded (/usr/lib/systemd/system/clockbound.service; enabled; preset: disabled)
+     Active: active (running) since Wed 2024-10-16 23:49:38 UTC; 53s ago
+   Main PID: 92765 (clockbound)
+      Tasks: 3 (limit: 22143)
+     Memory: 4.1M
+        CPU: 18ms
+     CGroup: /system.slice/clockbound.service
+             └─92765 /usr/local/bin/clockbound --max-drift-rate 50 -r PHC0 -i eth0
+
+Oct 16 23:49:38 ip-172-199-76-70.ec2.internal systemd[1]: Started ClockBound.
+Oct 16 23:49:38 ip-172-199-76-70.ec2.internal clockbound[92765]: 2024-10-16T23:49:38.629593Z  INFO main ThreadId(01) /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/c>
+Oct 16 23:49:38 ip-172-199-76-70.ec2.internal clockbound[92765]: 2024-10-16T23:49:38.629874Z  INFO ThreadId(02) /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clock->
+Oct 16 23:49:38 ip-172-199-76-70.ec2.internal clockbound[92765]: 2024-10-16T23:49:38.630045Z  INFO ThreadId(03) /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clock->
+```
+
+</details>
+
+## Set ulimits
 
 In Linux, `ulimit` is used to limit and control the usage of system resources (threads, files, and network connections) on a per-process or per-user basis.
 
@@ -195,7 +250,7 @@ LimitRTPRIO=    | ulimit -r            | 0 |
 
 If a ulimit is set to `unlimited`, set it to `infinity` in the systemd configuration file.
 
-## transparent hugepages
+## Enable transparent hugepages
 
 Transparent hugepages should be enabled for optimal performance. By default, they are enabled.
 

docs/content/preview/explore/going-beyond-sql/connection-mgr-ysql.md

@@ -63,7 +63,7 @@ Because `enable_ysql_conn_mgr` is a preview flag only, to use it, add the flag t
 
 {{< note >}}
 
-To create a large number of client connections, ensure that "SHMMNI" (the maximum number of concurrent shared memory segments an OS allows) as well as [ulimit](../../../deploy/manual-deployment/system-config/#ulimits) is set correctly as follows:
+To create a large number of client connections, ensure that "SHMMNI" (the maximum number of concurrent shared memory segments an OS allows) as well as [ulimit](../../../deploy/manual-deployment/system-config/#set-ulimits) is set correctly as follows:
 
 1. Open the file `/etc/sysctl.conf`.
 1. Add `kernel.shmmni = 32768` (support for 30000 clients) at the end of the file.

docs/content/preview/quick-start/linux.md

@@ -66,7 +66,7 @@ Installing YugabyteDB involves completing [prerequisites](#prerequisites) and [d
 
 #### ulimits
 
-Because each tablet maps to its own file, you can create a very large number of files in the current shell by experimenting with several hundred tables and several tablets per table. You need to [configure ulimit values](../../deploy/manual-deployment/system-config/#ulimits).
+Because each tablet maps to its own file, you can create a very large number of files in the current shell by experimenting with several hundred tables and several tablets per table. You need to [configure ulimit values](../../deploy/manual-deployment/system-config/#set-ulimits).
 
 ### Download
 

docs/content/preview/reference/configuration/yb-master.md

@@ -177,6 +177,12 @@ If enabled, indexes on the same (YCQL) table may be batched together during back
 
 Default: `true`
 
+##### --time_source
+
+Specifies the time source used by the database. {{<tags/feature/tp>}} Set this to `clockbound` for configuring a highly accurate time source. Using `clockbound` requires [system configuration](../../../deploy/manual-deployment/system-config/#set-up-time-synchronization).
+
+Default: `""`
+
 ## YSQL flags
 
 ##### --enable_ysql

docs/content/preview/reference/configuration/yb-tserver.md

@@ -175,6 +175,12 @@ Location of the `.htpasswd` file containing usernames and hashed passwords, for
 
 Default: `""`
 
+##### --time_source
+
+Specifies the time source used by the database. {{<tags/feature/tp>}} Set this to `clockbound` for configuring a highly accurate time source. Using `clockbound` requires [system configuration](../../../deploy/manual-deployment/system-config/#set-up-time-synchronization).
+
+Default: `""`
+
 ## Logging flags
 
 ##### --log_dir

scripts/k8s_preflight.py

@@ -19,7 +19,7 @@
 # POSIX message queues, threshold - 819200 (RLIMIT_MSGQUEUE)
 # real-time priority, threshold - 0 (RLIMIT_RTPRIO)
 # As of now, necessary ulimit resources for YB are in verification RLIMIT_NOFILE and RLIMIT_NPROC
-# Ref - https://docs.yugabyte.com/preview/deploy/manual-deployment/system-config/#ulimits
+# Ref - https://docs.yugabyte.com/preview/deploy/manual-deployment/system-config/#set-ulimits
 
 REQUIRED_RESOURCES = {
     "RLIMIT_NOFILE": (1048576, "open files"),