This tutorial walks through the steps to create a simple HTTP server with a load tester. If you already know how to Go's HTTP libraries, skip to the "Load Testing" section to get started with Bender and HTTP.
You will need to install and configure Go by following the instructions on the
Getting Started page. Then follow the instructions on the
How To Write Go Code page, particularly for setting up your
workspace and the GOPATH environment variable, which we will use throughout this tutorial.
You will also need the latest version of Bender, which you can get by running:
go get github.com/pinterest/bender
This section will walk through the creation of an HTTP client and server, which we will use to test Bender in the following section. If you already have an HTTP client and server, you can use them instead.
In the following, all commands should be run from the $GOROOT directory, unless otherwise noted.
Create a new Go package for your HTTP server and client. We'll refer to this as $PKG in this
document, and it can be any path you want. At Pinterest, for example, we use github.com/pinterest.
cd $GOPATH
mkdir -p src/$PKG/hellohttp
Now we will create a simple HTTP service that echoes a URL argument. Firect, create a new directory:
mkdir -p src/$PKG/hellohttp/server
Then create a file named main.go in that directory and add these lines to it:
package main
import (
"net/http"
"fmt"
"log"
)
func echoHandler(w http.ResponseWriter, r *http.Request) {
msg := r.FormValue("message")
log.Printf(msg)
fmt.Fprintf(w, msg)
}
func main() {
http.HandleFunc("/echo/", echoHandler)
http.ListenAndServe("localhost:9999", nil)
}Now we will create a simple HTTP client for our HTTP service. First, create a new directory:
mkdir -p src/$PKG/hellohttp/client
Then create a file named main.go in that directory and add these lines to it:
package main
import (
"net/http"
"fmt"
"io/ioutil"
"net/url"
"bytes"
"strconv"
)
func main() {
form := url.Values{}
form.Set("message", "hello, world")
encForm := form.Encode()
req, err := http.NewRequest("POST", "http://localhost:9999/echo/", bytes.NewBufferString(encForm))
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
req.Header.Add("Content-Length", strconv.Itoa(len(encForm)))
if err != nil {
panic(err)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}
fmt.Println(string(body))
}This is an example of a POST request, which can get very complicated using the basic Go libraries. For an easier approach to creating requests, see the GoReq library. If you use that library, make sure you write a request executor that also uses it for Bender!
Run these commands to compile and install the server and client:
go install $PKG/hellohttp/server
go install $PKG/hellohttp/client
In one terminal window, run the server as ./bin/server and in another terminal window run the
client as ./bin/client. You should see "hello, world" print in both terminals.
Now that we have an HTTP server we can use Bender to build a simple load tester for it. This section uses the same directories and packages as the previous section, but it is easy to use the same instructions for any HTTP server. The next few sections walk through the various parts of the load tester. If you are in a hurry skip to the section "Final Load Tester Program" and just follow the instructions from there.
The first thing we need is a function to generate intervals (in nanoseconds) between executing requests. The Bender library comes with some predefined intervals, including a uniform distribution (always wait the same amount of time between each request) and an exponential distribution. In this case we will use the exponential distribution, which means our server will experience load as generated by a Poisson process, which is fairly typical of server workloads on the Internet (with the usual caveats that every service is a special snowflake, etc, etc). We get the interval function with this code:
intervals := bender.ExponentialIntervalGenerator(qps)
Where qps is our desired throughput measured in queries per second. It is also the reciprocal of
the mean value of the exponential distribution used to generate the request arrival times (see the
wikipedia article above). In practice this means you will see an average QPS that fluctuates around
the target QPS (with less fluctuation as you increase the time interval over which you are
averaging).
The second thing we need is a channel of requests to send to the HTTP server. When an interval has been generated and Bender is ready to send the request, it pulls the next request from this channel and spawns a goroutine to send the request to the server. This function creates a simple synthetic request generator:
func SyntheticHttpRequests(n int) chan interface{} {
c := make(chan interface{}, 100)
go func() {
for i := 0; i < n; i++ {
form := url.Values{}
form.Set("message", fmt.Sprintf("hello: %d", i))
encForm := form.Encode()
request, err := http.NewRequest("POST", "http://localhost:9999/echo/", bytes.NewBufferString(encForm))
if err != nil {
panic(err)
}
request.Header.Add("Content-Type", "application/x-www-form-urlencoded")
request.Header.Add("Content-Length", strconv.Itoa(len(encForm))
c <- request
}
close(c)
}()
return c
}The next thing we need is a request executor, which takes the requests generated above and sends them to the service. We will use a helper function from Bender's http library to do most of the work (connection management, error handling, etc), so all we have to do is write code to send the request:
func bodyValidator(request interface{}, resp *http.Response) error {
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
text := string(bytes)
if strings.HasPrefix(text, "hello: ") {
return nil
}
return errors.New(fmt.Sprintf("Body (%s) is missing prefix 'hello: '", text))
}
exec := http.CreateExecutor(nil, nil, bodyHandler)This validates that the response body has the echoed text (minus the suffix), and returns an error otherwise.
The last thing we need is a channel that will output events as the load tester runs. This will let us listen to the load testers progress and record stats. We want this channel to be buffered so that we can run somewhat independently of the load test without slowing it down:
recorder := make(chan interface{}, 128)The LoadTestThroughput function returns a channel on which it will send events for things like
the start of the load test, how long it waits between requests, how much overage it is currently
experiencing, and when requests start and end, how long they took and whether or not they had
errors. That raw event stream makes it possible to analyze the results of a load test. Bender has
a couple simple "recorders" that provide basic functionality for result analysis and all of which
use the Record function:
NewLoggingRecordercreates a recorder that takes alog.Loggerand outputs each event to it in a well-defined format.NewHistogramRecordercreates a recorder that manages a histogram of latencies from requests and error counts.
You can combine recorders using the Record function, so you can both log events and manage a
histogram using code like this:
l := log.New(os.Stdout, "", log.LstdFlags)
h := hist.NewHistogram(60000, 10000000)
bender.Record(recorder, bender.NewLoggingRecorder(l), bender.NewHistogramRecorder(h))The histogram takes two arguments: the number of buckets and a scaling factor for times. In this case we are going to record times in milliseconds and allow 60,000 buckets for times up to one minute. The scaling factor is 1,000,000 which converts from nanoseconds (the timer values) to milliseconds.
It is relatively easy to build recorders, or to just process the events from the channel yourself, see the Bender documentation for more details on what events can be sent, and what data they contain.
Create a directory for the load tester:
mkdir -p src/$PKG/hellobender
Then create a file named main.go in that directory and add these lines to it:
package main
import (
"github.com/pinterest/bender"
"log"
"os"
"github.com/pinterest/bender/hist"
"time"
"fmt"
bhttp "github.com/pinterest/bender/http"
"net/url"
"net/http"
"bytes"
"strconv"
"io"
"io/ioutil"
"errors"
"strings"
)
func SyntheticHttpRequests(n int) chan interface{} {
c := make(chan interface{}, 100)
go func() {
for i := 0; i < n; i++ {
form := url.Values{}
form.Set("message", fmt.Sprintf("hello: %d", i))
encForm := form.Encode()
req, err := http.NewRequest("POST", "http://localhost:9999/echo/", bytes.NewBufferString(encForm))
if err != nil {
panic(err)
}
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
req.Header.Add("Content-Length", strconv.Itoa(len(encForm)))
c <- req
}
close(c)
}()
return c
}
func bodyValidator(request interface{}, resp *http.Response) error {
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
text := string(bytes)
if strings.HasPrefix(text, "hello: ") {
return nil
}
return errors.New(fmt.Sprintf("Body (%s) is missing prefix 'hello: '", text))
}
func main() {
intervals := bender.ExponentialIntervalGenerator(10)
requests := SyntheticHttpRequests(100)
exec := bhttp.CreateExecutor(nil, nil, bodyValidator)
recorder := make(chan interface{}, 100)
bender.LoadTestThroughput(intervals, requests, exec, recorder)
l := log.New(os.Stdout, "", log.LstdFlags)
h := hist.NewHistogram(60000, int(time.Millisecond))
bender.Record(recorder, bender.NewLoggingRecorder(l), bender.NewHistogramRecorder(h))
fmt.Println(h)
}Run these commands to compile and install the server and load tester:
go install $PKG/hellohttp/server
go install $PKG/hellobender
In one terminal window, run the server as ./bin/server and in another window run the load tester
as ./bin/hellobender. You should see a long sequence of outputs in the server window and a final
print out of the histogram data in the other window.