diff --git a/docs/content/preview/architecture/transactions/read-restart-error.md b/docs/content/preview/architecture/transactions/read-restart-error.md index ca0d5c77467..e4637c2e13e 100644 --- a/docs/content/preview/architecture/transactions/read-restart-error.md +++ b/docs/content/preview/architecture/transactions/read-restart-error.md @@ -74,6 +74,7 @@ How does YugabyteDB prevent this clock skew anomaly? You can handle and mitigate read restart errors using the following techniques: +- {{}} Configure [highly accurate clocks](../../../deploy/manual-deployment/system-config#set-up-time-synchronization). - Implement retry logic in the application. Application retries can help mitigate read restart errors. Moreover, a statement or a transaction may fail in other ways such as transaction conflicts or infrastructure failures. Therefore, a retry mechanism is strongly recommended for a cloud-native, distributed database such as YugabyteDB. While implementing application retries is the best long-term approach, there are a few short-term solutions you can use in the interim. diff --git a/docs/content/preview/deploy/checklist.md b/docs/content/preview/deploy/checklist.md index b7dcfdaa810..6009bd7d7d3 100644 --- a/docs/content/preview/deploy/checklist.md +++ b/docs/content/preview/deploy/checklist.md @@ -18,7 +18,7 @@ A YugabyteDB cluster consists of two distributed services - the [YB-TServer](../ - YugabyteDB supports both x86 and ARM (aarch64) CPU architectures. - YugabyteDB is supported on a variety of [operating systems](../../reference/configuration/operating-systems/). For production workloads, the recommended operating systems are AlmaLinux 8 and RHEL 8. - The appropriate system limits should be set using [`ulimit`](../manual-deployment/system-config/#ulimits) on each node running a YugabyteDB server. -- [NTP or chrony](../manual-deployment/system-config/#ntp) should be used to synchronize time among the machines. +- [Chrony](../manual-deployment/system-config#set-up-time-synchronization) should be used to synchronize time among the machines. ## Replication diff --git a/docs/content/preview/deploy/manual-deployment/start-masters.md b/docs/content/preview/deploy/manual-deployment/start-masters.md index e934818b09f..62982a95d3e 100644 --- a/docs/content/preview/deploy/manual-deployment/start-masters.md +++ b/docs/content/preview/deploy/manual-deployment/start-masters.md @@ -46,6 +46,8 @@ The number of comma-separated addresses in `--master_addresses` should equal the You can specify multiple directories using the [`--fs_data_dirs`](../../../reference/configuration/yb-master/#fs-data-dirs) flag. Replace the [`--rpc_bind_addresses`](../../../reference/configuration/yb-master/#rpc-bind-addresses) value with the private IP address of the host, and set the `placement_cloud`, `placement_region`, and `placement_zone` values appropriately. For single zone deployment, use the same value for the `placement_zone` flag. +{{}} Highly accurate clocks can be configured by specifying `--time_source=clockbound`. Requires [system configuration](../system-config#set-up-time-synchronization). + For the full list of configuration flags, see the [YB-Master reference](../../../reference/configuration/yb-master/). ## Run YB-Master servers with configuration file diff --git a/docs/content/preview/deploy/manual-deployment/start-tservers.md b/docs/content/preview/deploy/manual-deployment/start-tservers.md index 4e01aab9f0e..750cf355ac3 100644 --- a/docs/content/preview/deploy/manual-deployment/start-tservers.md +++ b/docs/content/preview/deploy/manual-deployment/start-tservers.md @@ -48,6 +48,8 @@ $ ./bin/yb-tserver \ Provide all of the master addresses using the [`--tserver_master_addrs`](../../../reference/configuration/yb-tserver/#tserver-master-addrs) flag. Replace the [`--rpc_bind_addresses`](../../../reference/configuration/yb-tserver/#rpc-bind-addresses) value with the private IP address of the host, and set the `placement_cloud`, `placement_region`, and `placement_zone` values appropriately. For single zone deployment, use the same value for the `--placement_zone` flag. +{{}} Highly accurate clocks can be configured by specifying `--time_source=clockbound`. Requires [system configuration](../system-config#set-up-time-synchronization). + For the full list of configuration flags, see the [YB-TServer reference](../../../reference/configuration/yb-tserver/). ## Run YB-TServer with configuration file diff --git a/docs/content/preview/deploy/manual-deployment/system-config.md b/docs/content/preview/deploy/manual-deployment/system-config.md index 28a0a83a35f..d2c10d8e099 100644 --- a/docs/content/preview/deploy/manual-deployment/system-config.md +++ b/docs/content/preview/deploy/manual-deployment/system-config.md @@ -13,29 +13,84 @@ type: docs Perform the following configuration on each node in the cluster: -- ntp or chrony -- ulimits -- transparent hugepages +- set up time synchronization +- set ulimits +- enable transparent hugepages Keep in mind that, although YugabyteDB is PostgreSQL compatible and runs a postgres process, it is not a PostgreSQL distribution. The PostgreSQL it runs doesn't need the same OS and system resources that open source PostgreSQL requires. For this reason, the kernel configuration requirements are different. In particular, the main YugabyteDB process, the YB-TServer, is multi-threaded. As a result, you don't need to modify settings for shared memory and inter-process communication (IPC), because there is no inter-process communication or shared memory in a multi-threaded process model (all memory is shared by the same process). -## ntp +## Set up time synchronization -If your instance does not have public Internet access, make sure the ntp package is installed: +YugabyteDB relies on clock synchronization to guarantee consistency in distributed transactions. Chrony is the prefered NTP implementation for clock synchronization. + +### Install Chrony + +To install chrony, run: ```sh -$ sudo yum install -y ntp +$ sudo yum install -y chrony ``` -As of CentOS 8, `ntp` is no longer available and has been replaced by `chrony`. To install, run: +### Configure Precision Time Protocol + +{{}} Precision Time Protocol (PTP) is a network protocol designed for highly accurate time synchronization across devices in a network. PTP provides microsecond-level accuracy. PTP relies on a PTP Hardware Clock (PHC), a dedicated physical clock device that enhances time synchronization accuracy. + +Currently, PTP is only available for AWS. To check if your AWS instance supports PTP and PHC, see [AWS PTP Hardware Clock](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-ec2-ntp.html#connect-to-the-ptp-hardware-clock). + +Configure PTP using the `configure_ptp.sh` script in the bin directory of your YugabyteDB home directory as follows: ```sh -$ sudo yum install -y chrony +sudo bash ./bin/configure_ptp.sh +``` + +### Configure ClockBound + +{{}} [ClockBound](https://github.com/aws/clock-bound) is an open source daemon that allows you to compare timestamps to determine order for events and transactions, independent of an instance's geographic location. ClockBound provides a strict interval in which the reference time (true time) exists. + +Although optional, configuring ClockBound improves clock accuracy by several orders of magnitude. ClockBound requires chrony and can be used in conjunction with PTP. + +Configure ClockBound using the `configure_clockbound.sh` script in the bin directory of your YugabyteDB home directory as follows: + +```sh +sudo bash ./bin/configure_clockbound.sh ``` -## ulimits +After configuring ClockBound, you must configure the [YB-TServer](../start-tservers/) and [YB-Master](../start-masters/) servers with the `time_source=clockbound` flag. + +If the ClockBound agent is configured with PTP, use a more aggressive clock error estimate such as `clockbound_clock_error_estimate_usec=100`. + +### Verify ClockBound configuration + +Verify that ClockBound is configured properly using the following command: + +```sh +systemctl status clockbound +``` + +A correctly configured ClockBound service reports no errors. The following shows example output with PTP enabled: + +```sh +● clockbound.service - ClockBound + Loaded: loaded (/usr/lib/systemd/system/clockbound.service; enabled; preset: disabled) + Active: active (running) since Wed 2024-10-16 23:49:38 UTC; 53s ago + Main PID: 92765 (clockbound) + Tasks: 3 (limit: 22143) + Memory: 4.1M + CPU: 18ms + CGroup: /system.slice/clockbound.service + └─92765 /usr/local/bin/clockbound --max-drift-rate 50 -r PHC0 -i eth0 + +Oct 16 23:49:38 ip-172-199-76-70.ec2.internal systemd[1]: Started ClockBound. +Oct 16 23:49:38 ip-172-199-76-70.ec2.internal clockbound[92765]: 2024-10-16T23:49:38.629593Z INFO main ThreadId(01) /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/c> +Oct 16 23:49:38 ip-172-199-76-70.ec2.internal clockbound[92765]: 2024-10-16T23:49:38.629874Z INFO ThreadId(02) /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clock-> +Oct 16 23:49:38 ip-172-199-76-70.ec2.internal clockbound[92765]: 2024-10-16T23:49:38.630045Z INFO ThreadId(03) /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clock-> +``` + + + +## Set ulimits In Linux, `ulimit` is used to limit and control the usage of system resources (threads, files, and network connections) on a per-process or per-user basis. @@ -195,7 +250,7 @@ LimitRTPRIO= | ulimit -r | 0 | If a ulimit is set to `unlimited`, set it to `infinity` in the systemd configuration file. -## transparent hugepages +## Enable transparent hugepages Transparent hugepages should be enabled for optimal performance. By default, they are enabled. diff --git a/docs/content/preview/reference/configuration/yb-master.md b/docs/content/preview/reference/configuration/yb-master.md index 1cfc88164e4..296ceb610e5 100644 --- a/docs/content/preview/reference/configuration/yb-master.md +++ b/docs/content/preview/reference/configuration/yb-master.md @@ -177,6 +177,12 @@ If enabled, indexes on the same (YCQL) table may be batched together during back Default: `true` +##### --time_source + +Specifies the time source used by the database. {{}} Set this to `clockbound` for configuring a highly accurate time source. Using `clockbound` requires [system configuration](../../../deploy/manual-deployment/system-config/#set-up-time-synchronization). + +Default: `""` + ## YSQL flags ##### --enable_ysql diff --git a/docs/content/preview/reference/configuration/yb-tserver.md b/docs/content/preview/reference/configuration/yb-tserver.md index b59f230c924..3d5b8f007ce 100644 --- a/docs/content/preview/reference/configuration/yb-tserver.md +++ b/docs/content/preview/reference/configuration/yb-tserver.md @@ -175,6 +175,12 @@ Location of the `.htpasswd` file containing usernames and hashed passwords, for Default: `""` +##### --time_source + +Specifies the time source used by the database. {{}} Set this to `clockbound` for configuring a highly accurate time source. Using `clockbound` requires [system configuration](../../../deploy/manual-deployment/system-config/#set-up-time-synchronization). + +Default: `""` + ## Logging flags ##### --log_dir