Skip to content

Commit

Permalink
[yugabyte#21963] YSQL: Configure clockbound agent
Browse files Browse the repository at this point in the history
  • Loading branch information
pao214 committed Oct 17, 2024
1 parent 3f7333a commit cffc1af
Showing 1 changed file with 65 additions and 10 deletions.
75 changes: 65 additions & 10 deletions docs/content/preview/deploy/manual-deployment/system-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,29 +13,84 @@ type: docs

Perform the following configuration on each node in the cluster:

- ntp or chrony
- ulimits
- transparent hugepages
- set up time synchronization
- set ulimits
- enable transparent hugepages

Keep in mind that, although YugabyteDB is PostgreSQL compatible and runs a postgres process, it is not a PostgreSQL distribution. The PostgreSQL it runs doesn't need the same OS and system resources that open source PostgreSQL requires. For this reason, the kernel configuration requirements are different.

In particular, the main YugabyteDB process, the YB-TServer, is multi-threaded. As a result, you don't need to modify settings for shared memory and inter-process communication (IPC), because there is no inter-process communication or shared memory in a multi-threaded process model (all memory is shared by the same process).

## ntp
## Set up time synchronization

If your instance does not have public Internet access, make sure the ntp package is installed:
YugabyteDB relies on clock synchronization to guarantee consistency in distributed transactions. Chrony is the prefered NTP implementation for clock synchronization.

### Install Chrony

To install chrony, run:

```sh
$ sudo yum install -y ntp
$ sudo yum install -y chrony
```

As of CentOS 8, `ntp` is no longer available and has been replaced by `chrony`. To install, run:
### Configure Precision Time Protocol

{{<tags/feature/tp>}} Precision Time Protocol (PTP) is a network protocol designed for highly accurate time synchronization across devices in a network. PTP provides microsecond-level accuracy. PTP relies on a PTP Hardware Clock (PHC), a dedicated physical clock device that enhances time synchronization accuracy.

Currently, PTP is only available for AWS. To check if your AWS instance supports PTP and PHC, see [AWS PTP Hardware Clock](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-ec2-ntp.html#connect-to-the-ptp-hardware-clock).

Configure PTP using the `configure_ptp.sh` script in the bin directory of your YugabyteDB home directory as follows:

```sh
$ sudo yum install -y chrony
sudo bash ./bin/configure_ptp.sh
```

### Configure ClockBound

{{<tags/feature/tp>}} [ClockBound](https://github.com/aws/clock-bound) is an open source daemon that allows you to compare timestamps to determine order for events and transactions, independent from the instances’ respective geographic locations. ClockBound provides a strict interval in which the reference time (true time) does exist.

Although optional, configuring ClockBound improves clock accuracy by several orders of magnitude. ClockBound requires chrony and can be used in conjunction with PTP.

Configure ClockBound using the `configure_clockbound.sh` script in the bin directory of your YugabyteDB home directory as follows:

```sh
sudo bash ./bin/configure_clockbound.sh
```

## ulimits
After configuring ClockBound, you must configure the [YB-TServer](../start-tservers/) and [YB-Master](../start-masters/) servers with the `time_source=clockbound` flag.

If the ClockBound agent is configured with PTP, use a more aggressive clock error estimate such as `clockbound_clock_error_estimate_usec=100`.

#### Verify ClockBound configuration

Verify that ClockBound is configured properly using the following command:

```sh
systemctl status clockbound
```

A correctly configured ClockBound service reports no errors. The following shows example output with PTP enabled:

```sh
● clockbound.service - ClockBound
Loaded: loaded (/usr/lib/systemd/system/clockbound.service; enabled; preset: disabled)
Active: active (running) since Wed 2024-10-16 23:49:38 UTC; 53s ago
Main PID: 92765 (clockbound)
Tasks: 3 (limit: 22143)
Memory: 4.1M
CPU: 18ms
CGroup: /system.slice/clockbound.service
└─92765 /usr/local/bin/clockbound --max-drift-rate 50 -r PHC0 -i eth0

Oct 16 23:49:38 ip-172-199-76-70.ec2.internal systemd[1]: Started ClockBound.
Oct 16 23:49:38 ip-172-199-76-70.ec2.internal clockbound[92765]: 2024-10-16T23:49:38.629593Z INFO main ThreadId(01) /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/c>
Oct 16 23:49:38 ip-172-199-76-70.ec2.internal clockbound[92765]: 2024-10-16T23:49:38.629874Z INFO ThreadId(02) /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clock->
Oct 16 23:49:38 ip-172-199-76-70.ec2.internal clockbound[92765]: 2024-10-16T23:49:38.630045Z INFO ThreadId(03) /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clock->
```

</details>

## Set ulimits

In Linux, `ulimit` is used to limit and control the usage of system resources (threads, files, and network connections) on a per-process or per-user basis.

Expand Down Expand Up @@ -195,7 +250,7 @@ LimitRTPRIO= | ulimit -r | 0 |
If a ulimit is set to `unlimited`, set it to `infinity` in the systemd configuration file.
## transparent hugepages
## Enable transparent hugepages
Transparent hugepages should be enabled for optimal performance. By default, they are enabled.
Expand Down

0 comments on commit cffc1af

Please sign in to comment.