Capacity planning and troubleshooting

[robert-s-lee]

Exalate commented:

High level steps below. More detail to be added

• run the workload at concurrency 1
  collect performance data: throughput, response time
  collect capacity data: CPU, RAM, Network bandwidth, Storage IOPS, Storage bandwidth, Storage capacity

• run the workload at least 2 other concurrency to construct the curve as below. More concurrency tests will produce more accurate prediction

• The capacity where the response time meet the SLA at various concurrent can be used to forecast

Jira Issue: DOC-154

[jseldess]

Capacity troubleshooting from #2461:

Performance, availability, and stability issues can be caused by insufficient capacity.

• "Because capacity is critical to availability ..." Google SRE https://landing.google.com/sre/book.htm
• server: collect system CPU, RAM, Network, IO metrics cockroach#21416 has examples of issues capacity management can solve.

Initial suggestions below:

Check Out of Capacity Conditions

First, check adequate capacity was available for the incident for the following components.

Type	Linux Command	what to look for
file system storage capacity	df -kh	file system at 99%
Storage IOPS / bandwidth capacity	sar -d	consistent non-zero values in avgqu-sz
RAM capacity	sar -s	any non-zero value in kbswpused and %swpused
CPU capacity	sar -q	consistent non-zero values in runq-sz
Network capacity	sar -n ALL	any non-zero value in rxerr/s, txerr/s, coll/s, rxdrop/s, txdrop/s, txcarr/s, rxfram/s, rxfifo/s, txfifo/s

Check Near Out of Capacity Conditions

Type	Linux Command	what to look for
Storage bandwidth capacity	sar -d	consistently more than 80% or unusually high values in rd_sec/s and wr_sec/s (in 512 bytes sector thus not as accurate as iostat)
Storage IOPS capacity	sar -d	consistently more than 1ms or unusually high values in await
RAM capacity	sar -r	consistently more than 80% in %memused
CPU capacity	sar -u	consistently less than 20% in %idle (ie:80% busy)
Network capacity	sar -n ALL	consistently more than 50% line capacity in rxkB/s or txkB/s

Check for abnormal conditions

Historical measurements are required determine whether a metric is too high or too low. Maintaining at least 1 month of historical data will be useful.

Linix sar in crontab

Enable sar data collected at 1-minute interval

• Ubuntu

sudo bash
apt-get install sysstat
sed -ibak -e 's/ENABLED="false"/ENABLED="true"/g' /etc/default/sysstat
sed -ibak -e 's|5-55/10 \* \* \* \*|\* \* \* \* \*|g' /etc/cron.d/sysstat

[jseldess]

This docs work should wait on this related 2.1 roadmap work.

[jseldess]

Based on feedback from @drewdeally, we don't yet have the data for this. Moving to Later.

[Amruta-Ranade]

@robert-s-lee I addressed this issue here: https://www.cockroachlabs.com/docs/dev/cluster-setup-troubleshooting.html#capacity-planning-issues
PTAL and let me know if I can close this issue.

[drewdeally]

These are to high and should be adjusted

…

On Apr 16, 2019, at 2:11 PM, Amruta Ranade ***@***.***> wrote: @robert-s-lee <https://github.com/robert-s-lee> I addressed this issue here: https://www.cockroachlabs.com/docs/dev/cluster-setup-troubleshooting.html#capacity-planning-issues <https://www.cockroachlabs.com/docs/dev/cluster-setup-troubleshooting.html#capacity-planning-issues> PTAL and let me know if I can close this issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2723 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Alqpif6sPjld3F-zmXoY2xXTwkkLcKx4ks5vhhI1gaJpZM4SsFni>.

[Amruta-Ranade]

@drewdeally Can you share the correct values?

[taroface]

Relates to #7818 and #8547.

[jseldess]

We have closed this issue because it is more than 3 years
old. If this issue is still relevant, please add a comment and re-open
the issue. Thank you for your contribution to CockroachDB docs!