Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarks that runs server and client and separate processes. #1952

Merged
merged 2 commits into from
May 21, 2018

Conversation

MakMukhi
Copy link
Contributor

@MakMukhi MakMukhi commented Mar 29, 2018

Also,

  • Record CPU utilization on both client and server.
  • Profile CPU and Memory on both client and server.
  • Do not record buckets.
  • One stop shell script to run benchmarks.

Sample output:

$ ./run_bench.sh -r 1,10 -c 1,200 -req 1,100 -resp 1 -rpc_type unary,streaming
================================================================================
r_1_c_1_req_1_resp_1_unary_1522345524
qps: 9247.9
Latency: (50/90/99 %ile): 102.367µs/122.551µs/168.352µs
Client CPU utilization: 15.369409788s
Client CPU profile: /tmp/client_r_1_c_1_req_1_resp_1_unary_1522345524.cpu
Client Mem Profile: /tmp/client_r_1_c_1_req_1_resp_1_unary_1522345524.mem
Server CPU utilization: 17.173763239s
Server CPU profile: /tmp/Server_r_1_c_1_req_1_resp_1_unary_1522345524.cpu
Server Mem Profile: /tmp/Server_r_1_c_1_req_1_resp_1_unary_1522345524.mem
================================================================================
r_1_c_1_req_1_resp_1_streaming_1522345545
qps: 14345.3
Latency: (50/90/99 %ile): 67.821µs/83.16µs/104.562µs
Client CPU utilization: 22.728285653s
Client CPU profile: /tmp/client_r_1_c_1_req_1_resp_1_streaming_1522345545.cpu
Client Mem Profile: /tmp/client_r_1_c_1_req_1_resp_1_streaming_1522345545.mem
Server CPU utilization: 22.438695158s
Server CPU profile: /tmp/Server_r_1_c_1_req_1_resp_1_streaming_1522345545.cpu
Server Mem Profile: /tmp/Server_r_1_c_1_req_1_resp_1_streaming_1522345545.mem
================================================================================
r_1_c_1_req_100_resp_1_unary_1522345565
qps: 9155.9
Latency: (50/90/99 %ile): 102.661µs/125.137µs/176.439µs
Client CPU utilization: 15.744705954s
Client CPU profile: /tmp/client_r_1_c_1_req_100_resp_1_unary_1522345565.cpu
Client Mem Profile: /tmp/client_r_1_c_1_req_100_resp_1_unary_1522345565.mem
Server CPU utilization: 17.443043385s
Server CPU profile: /tmp/Server_r_1_c_1_req_100_resp_1_unary_1522345565.cpu
Server Mem Profile: /tmp/Server_r_1_c_1_req_100_resp_1_unary_1522345565.mem
================================================================================
r_1_c_1_req_100_resp_1_streaming_1522345587
qps: 13295.9
Latency: (50/90/99 %ile): 71.699µs/92.502µs/124.671µs
Client CPU utilization: 22.743129754s
Client CPU profile: /tmp/client_r_1_c_1_req_100_resp_1_streaming_1522345587.cpu
Client Mem Profile: /tmp/client_r_1_c_1_req_100_resp_1_streaming_1522345587.mem
Server CPU utilization: 22.615264368s
Server CPU profile: /tmp/Server_r_1_c_1_req_100_resp_1_streaming_1522345587.cpu
Server Mem Profile: /tmp/Server_r_1_c_1_req_100_resp_1_streaming_1522345587.mem
================================================================================
r_1_c_200_req_1_resp_1_unary_1522345607
qps: 81443.6
Latency: (50/90/99 %ile): 2.200862ms/3.746498ms/9.13501ms
Client CPU utilization: 1m40.708680062s
Client CPU profile: /tmp/client_r_1_c_200_req_1_resp_1_unary_1522345607.cpu
Client Mem Profile: /tmp/client_r_1_c_200_req_1_resp_1_unary_1522345607.mem
Server CPU utilization: 1m58.195342807s
Server CPU profile: /tmp/Server_r_1_c_200_req_1_resp_1_unary_1522345607.cpu
Server Mem Profile: /tmp/Server_r_1_c_200_req_1_resp_1_unary_1522345607.mem
================================================================================
r_1_c_200_req_1_resp_1_streaming_1522345628
qps: 149939.6
Latency: (50/90/99 %ile): 1.269794ms/1.962588ms/3.759668ms
Client CPU utilization: 1m49.214788204s
Client CPU profile: /tmp/client_r_1_c_200_req_1_resp_1_streaming_1522345628.cpu
Client Mem Profile: /tmp/client_r_1_c_200_req_1_resp_1_streaming_1522345628.mem
Server CPU utilization: 1m45.088960997s
Server CPU profile: /tmp/Server_r_1_c_200_req_1_resp_1_streaming_1522345628.cpu
Server Mem Profile: /tmp/Server_r_1_c_200_req_1_resp_1_streaming_1522345628.mem
================================================================================
r_1_c_200_req_100_resp_1_unary_1522345649
qps: 75291.4
Latency: (50/90/99 %ile): 2.332258ms/4.208357ms/9.884555ms
Client CPU utilization: 1m38.372587579s
Client CPU profile: /tmp/client_r_1_c_200_req_100_resp_1_unary_1522345649.cpu
Client Mem Profile: /tmp/client_r_1_c_200_req_100_resp_1_unary_1522345649.mem
Server CPU utilization: 1m54.293421801s
Server CPU profile: /tmp/Server_r_1_c_200_req_100_resp_1_unary_1522345649.cpu
Server Mem Profile: /tmp/Server_r_1_c_200_req_100_resp_1_unary_1522345649.mem
================================================================================
r_1_c_200_req_100_resp_1_streaming_1522345669
qps: 146473.7
Latency: (50/90/99 %ile): 1.286938ms/2.019942ms/4.119743ms
Client CPU utilization: 1m48.188005074s
Client CPU profile: /tmp/client_r_1_c_200_req_100_resp_1_streaming_1522345669.cpu
Client Mem Profile: /tmp/client_r_1_c_200_req_100_resp_1_streaming_1522345669.mem
Server CPU utilization: 1m43.985864847s
Server CPU profile: /tmp/Server_r_1_c_200_req_100_resp_1_streaming_1522345669.cpu
Server Mem Profile: /tmp/Server_r_1_c_200_req_100_resp_1_streaming_1522345669.mem
================================================================================
r_10_c_1_req_1_resp_1_unary_1522345690
qps: 41713
Latency: (50/90/99 %ile): 224.27µs/320.835µs/551.824µs
Client CPU utilization: 48.104101429s
Client CPU profile: /tmp/client_r_10_c_1_req_1_resp_1_unary_1522345690.cpu
Client Mem Profile: /tmp/client_r_10_c_1_req_1_resp_1_unary_1522345690.mem
Server CPU utilization: 54.867285235s
Server CPU profile: /tmp/Server_r_10_c_1_req_1_resp_1_unary_1522345690.cpu
Server Mem Profile: /tmp/Server_r_10_c_1_req_1_resp_1_unary_1522345690.mem
================================================================================
r_10_c_1_req_1_resp_1_streaming_1522345711
qps: 86132.1
Latency: (50/90/99 %ile): 109.487µs/151.798µs/235.289µs
Client CPU utilization: 49.221682659s
Client CPU profile: /tmp/client_r_10_c_1_req_1_resp_1_streaming_1522345711.cpu
Client Mem Profile: /tmp/client_r_10_c_1_req_1_resp_1_streaming_1522345711.mem
Server CPU utilization: 46.796757995s
Server CPU profile: /tmp/Server_r_10_c_1_req_1_resp_1_streaming_1522345711.cpu
Server Mem Profile: /tmp/Server_r_10_c_1_req_1_resp_1_streaming_1522345711.mem
================================================================================
r_10_c_1_req_100_resp_1_unary_1522345731
qps: 40753.6
Latency: (50/90/99 %ile): 227.362µs/331.936µs/583.229µs
Client CPU utilization: 47.645275133s
Client CPU profile: /tmp/client_r_10_c_1_req_100_resp_1_unary_1522345731.cpu
Client Mem Profile: /tmp/client_r_10_c_1_req_100_resp_1_unary_1522345731.mem
Server CPU utilization: 54.444179752s
Server CPU profile: /tmp/Server_r_10_c_1_req_100_resp_1_unary_1522345731.cpu
Server Mem Profile: /tmp/Server_r_10_c_1_req_100_resp_1_unary_1522345731.mem
================================================================================
r_10_c_1_req_100_resp_1_streaming_1522345751
qps: 85281
Latency: (50/90/99 %ile): 110.556µs/152.502µs/241.895µs
Client CPU utilization: 49.895785026s
Client CPU profile: /tmp/client_r_10_c_1_req_100_resp_1_streaming_1522345751.cpu
Client Mem Profile: /tmp/client_r_10_c_1_req_100_resp_1_streaming_1522345751.mem
Server CPU utilization: 47.539538087s
Server CPU profile: /tmp/Server_r_10_c_1_req_100_resp_1_streaming_1522345751.cpu
Server Mem Profile: /tmp/Server_r_10_c_1_req_100_resp_1_streaming_1522345751.mem
================================================================================
r_10_c_200_req_1_resp_1_unary_1522345772
qps: 103747
Latency: (50/90/99 %ile): 17.245362ms/32.398259ms/56.972638ms
Client CPU utilization: 1m42.37146451s
Client CPU profile: /tmp/client_r_10_c_200_req_1_resp_1_unary_1522345772.cpu
Client Mem Profile: /tmp/client_r_10_c_200_req_1_resp_1_unary_1522345772.mem
Server CPU utilization: 1m58.881603944s
Server CPU profile: /tmp/Server_r_10_c_200_req_1_resp_1_unary_1522345772.cpu
Server Mem Profile: /tmp/Server_r_10_c_200_req_1_resp_1_unary_1522345772.mem
================================================================================
r_10_c_200_req_1_resp_1_streaming_1522345793
qps: 216072.9
Latency: (50/90/99 %ile): 8.131356ms/15.229851ms/33.159954ms
Client CPU utilization: 1m56.269958794s
Client CPU profile: /tmp/client_r_10_c_200_req_1_resp_1_streaming_1522345793.cpu
Client Mem Profile: /tmp/client_r_10_c_200_req_1_resp_1_streaming_1522345793.mem
Server CPU utilization: 1m49.687516735s
Server CPU profile: /tmp/Server_r_10_c_200_req_1_resp_1_streaming_1522345793.cpu
Server Mem Profile: /tmp/Server_r_10_c_200_req_1_resp_1_streaming_1522345793.mem
================================================================================
r_10_c_200_req_100_resp_1_unary_1522345813
qps: 100977.6
Latency: (50/90/99 %ile): 17.829843ms/33.394985ms/56.336764ms
Client CPU utilization: 1m41.961430197s
Client CPU profile: /tmp/client_r_10_c_200_req_100_resp_1_unary_1522345813.cpu
Client Mem Profile: /tmp/client_r_10_c_200_req_100_resp_1_unary_1522345813.mem
Server CPU utilization: 1m57.15810443s
Server CPU profile: /tmp/Server_r_10_c_200_req_100_resp_1_unary_1522345813.cpu
Server Mem Profile: /tmp/Server_r_10_c_200_req_100_resp_1_unary_1522345813.mem
================================================================================
r_10_c_200_req_100_resp_1_streaming_1522345834
qps: 200728.4
Latency: (50/90/99 %ile): 8.647847ms/16.854851ms/35.64926ms
Client CPU utilization: 1m51.93917114s
Client CPU profile: /tmp/client_r_10_c_200_req_100_resp_1_streaming_1522345834.cpu
Client Mem Profile: /tmp/client_r_10_c_200_req_100_resp_1_streaming_1522345834.mem
Server CPU utilization: 1m45.537292628s
Server CPU profile: /tmp/Server_r_10_c_200_req_100_resp_1_streaming_1522345834.cpu
Server Mem Profile: /tmp/Server_r_10_c_200_req_100_resp_1_streaming_1522345834.mem

@MakMukhi MakMukhi requested a review from dfawley March 29, 2018 17:57
@MakMukhi MakMukhi force-pushed the bench branch 2 times, most recently from 6e47fae to ede983e Compare March 29, 2018 19:28
done

# Build server and client
go install $GOPATH/src/google.golang.org/grpc/benchmark/server/benchserver.go && go install $GOPATH/src/google.golang.org/grpc/benchmark/client/benchclient.go
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to do this with -o benchserver and avoid renaming the files?

@MakMukhi MakMukhi requested a review from lyuxuan April 12, 2018 21:50
@MakMukhi MakMukhi closed this May 4, 2018
@MakMukhi MakMukhi deleted the bench branch May 4, 2018 02:01
@MakMukhi MakMukhi restored the bench branch May 4, 2018 02:07
@MakMukhi MakMukhi reopened this May 4, 2018
trace = flag.Bool("trace", true, "Whether tracing is on")
rpcType = flag.Int("rpc_type", 0,
port = flag.String("port", "50051", "Localhost port to connect to.")
r = flag.Int("r", 1, "The number of concurrent RPCs on each connection.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These guys are global variables, and as such, should have long names. Rationale:

  1. Used far away from where defined.
  2. Shadowed by commonly-used locals.

@@ -0,0 +1,207 @@
#!/bin/bash

rs=(1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment about names -- for readability, these should be descriptive.

resps=(1)
rpc_types=(unary)

# idx[0] = idx value for rs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get it...what is idx?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All input parameters that are arrays, need an index to go over them while keeping other parameters' index unchanged. This way we can generate all combinations without using nested for loops

fi
done
if [ ${return} == 1 ]; then
rm -Rf ${out_dir}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent please


inc()
{
for i in $(seq $((${#idx[@]}-1)) -1 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer:

for _____; do

which you use below. (Let's be consistent at the very least.)

case "$1" in
-r)
shift
if test $# -gt 0; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer:

if [ $# -eq 0 ]; then   # or [[ $# == 0 ]]
  echo "error stuff"
  exit 1
fi
rs=....
...

idx_max[4]=${#rpc_types[@]}
for val in ${rpc_types[@]}; do
case "${val}" in
"unary"|"streaming");;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to vet the vals here, or can we pass them to the script and wait for it to fail?

;;
-h|--help)
echo "Following are valid options:"
echo ""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

echo "" == echo

;;
-c)
shift
if test $# -gt 0; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider a refactor:

param() {
  argname=$1
  shift
  if [ $# -eq 0 ]; then
    echo "$argname not specified"
    exit 1
  fi
  echo $1 | sed 's/,/ /g'
}
...
  -c)
    shift
    cs=$(param "number of connections" "$@")
    shift

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cs=$(param "arg1" "arg2") doesn't call param(). Maybe, I'm doing something wrong.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe so?

$ x() { echo "hi"; }
$ y=$(x)
$ echo $y
hi

-r)
shift
if test $# -gt 0; then
rs=($(echo $1 | sed 's/,/ /g'))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why a subshell?

This is preferred: rs="$(echo "$1" | sed 's/,/ /g')"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rs is an array. rs=(1 2 3)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, I see...parens are overloaded.

while [ $# -gt 0 ]; do
case "$1" in
-r)
shift
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can (re)move these shifts:

while ... do
  case "$1" in
    -r)
      set_param "number of rpcs" 0 $2
      rpcs=...
      ;;
...
  esac
  shift; shift
done

@menghanl menghanl assigned MakMukhi and unassigned dfawley May 17, 2018
@MakMukhi MakMukhi merged commit dea4e57 into grpc:master May 21, 2018
@MakMukhi MakMukhi deleted the bench branch May 29, 2018 21:01
@menghanl menghanl added this to the 1.13 Release milestone Jun 19, 2018
@lock lock bot locked as resolved and limited conversation to collaborators Dec 16, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants