Benchmarking with HTTPerf.js and NodeUnit

I covered this in the HTTPerf.js README a bit, but wanted to take a deeper look at how I use HTTPerf.js and NodeUnit to benchmark web applications.

Note on benchmarking.

I've gone back and forth on using median and 85th percentile when benchmarking. I look at 85th percentile when running load tests and performance tests — basically when I'm trying to break my site. However, for simple benchmarks like this, median gives you a better perspective.

Setup

Before getting started you'll need to install node, httperf, HTTPerf.js and NodeUnit.

# httperf
sudo apt-get install httperf # also available via brew and yum
sudo apt-get nodejs          # also available via brew and yum
npm install nodeunit httperf

Benchmark Example One

In my README, I have this basic example. It's pretty simple for small sites, where you want to benchmark just a few endpoints.

file: benchmark.js

var HTTPerf = require('httperfjs');
var httperf = new HTTPerf({
    server:      "mervine.net",
    uri:         "/",
    "num-conns": 9
});

var run;

module.exports = {
    tearDown: function (callback) {
        run = undefined;
        callback();
    },

    'homepage should be quick': function (test) {
        test.expect(1);
        httperf.run( function (run) {
            test.ok(run.connection_time_avg < 200,
                "homepage was too slow: got " + run.connection_time_avg
                   + " but expected: < 200");
            test.done();
        });
    },

    'archive should be quick': function (test) {
        test.expect(1);
        httperf.run( function (run) {
            test.ok(run.connection_time_median < 200,
                "archive was too slow: got " + run.connection_time_avg
                    + " but expected: < 200");
            test.done();
        });
    }
};

Run with:

$ ./node_modules/.bin/nodeunit ./benchmark.js

Benchmark Example Two

Now, let's pretend we have a real site, with a lot of endpoints that we potentially want to benchmark the performance of. The above will work, but as your list grows it will gradually become harder to maintain. In this example, I build my list of urls using an external json file and use a factory function within my test to generate the complete test cases.

file: tests.json

[
    { "uri": "/"                 , "expected": 250 },
    { "uri": "/about"            , "expected": 200 },
    { "uri": "/projects"         , "expected": 200 },
    { "uri": "/archive"          , "expected": 200 },
    { "uri": "/archive/2012"     , "expected": 200 },
    { "uri": "/search?q=httperf" , "expected": 1000 }
]

file: benchmark.js

var testCases = require('./tests.json');
var HTTPerf = require('httperfjs');
var httperf = new HTTPerf({
    server:      "mervine.net",
    uri:         "/",
    "num-conns": 9
});
var run;
function testFactory(uri, expected) {
    tests['"' + uri + '" should be less then ' + expected] = function (test) {
        test.expect(1);
        httperf.update_option("uri", uri);
        httperf.run( function (run) {
            test.ok(run.connection_time_median < expected,
                "too slow: got " + run.connection_time_median
                    + " but expected: < " + expected);
            test.done();
        });
    };
}
var tests = {
    tearDown: function (callback) {
        run = undefined;
        callback();
    },
};
testCases.forEach( function (testCase) {
    testFactory(testCase.uri, testCase.expected);
});
module.exports = tests;

Run with:

$ ./node_modules/.bin/nodeunit ./benchmark.js

This should generate some simple and pretty output like so:

benchmark.js
+ "/" should be less then 250
+ "/about" should be less then 200
+ "/projects" should be less then 200
+ "/archive" should be less then 200
+ "/archive/2012" should be less then 200
+ "/search?q=httperf" should be less then 1000

OK: 6 assertions (13445ms)

Enjoy!


Articles on Benchmarking with HTTPerf.js and NodeUnit

HTTPerf.js: Release 0.1.0

Removing runSync. Refactoring run to support sending spawned process SIGINT to capture current report from httperf and exit.