Benchmarking with YSlow.js on Node.js

In my last post on this topic (Benchmarking with HTTPerf.js and NodeUnit) I covered benchmarking application render times from the server to first byte. That's a very good starting point for ensuring the stability for your application, i.e. uptime. Beyond that, it's also important to ensure that you application is performing well on a browser. At the end of the day, while first byte times are wildly important, it's because they impact the end user experience. If you're not also benchmarking client load times, you're missing a big part of the picture.

In this post, I'm going cover basic client benchmarking using YSlow and PhantomJS via YSlow.js on Node.js, which is backed by Phapper (a PhantomJS wrapper for Node). Unlike in my last post, I'm not going to use NodeUnit.

Issues with Browser based Benchmarking

Clients (browsers), even the same clients, can be unpredictable when making Ajax requests. On my site, I include both ShareThis buttons and Disqus commenting. Both are fantastic services, however given that I'm ajaxing these thrid-party includes they can throw off benchmark numbers in a very big way. A basic example is with YSlow's http requests metric (r in the results object), I have seen this range from 8 to 50 on two back to back page loads of the same page.

Additionally, network speed. Are you running your benchmarks on the same network as your application, or on a saturated wifi connection at your local Starbucks? Over VPN? Etc. These, and similar conditions, can tweak your results in very large ways. A few milliseconds of latency for a basic curl can turn in to a few hundred spread across dozens of images, a few CSS and JS files and a ajax request or two thrown in for good measure. I think you get the point.

Enough gabbing already... time for the meat!

Setup

Before getting started, you'll need to install Node.js and YSlow.js on Node.js. (You can optionally pre-install PhantomJS, but if you don't Phapper will install it for you.)

sudo apt-get nodejs # also available via brew and yum
npm install yslowjs

Basic Example

Here I will get a working example, benchmarking a single endpoint for it's YSlow grade and browser loadtime.

file: bench.js:

#!/usr/bin/env node
var YSlow = require('yslowjs');
var yslow = new YSlow('http://mervine.net/about', ['--info', 'basic']);

yslow.run( function(results) {
    var grade    = (results.o >= 74);
    var loadtime = (results.lt <= 1000);
    console.log("YSlow Benchmark (/about)");
    console.log("=> Grade %s with %s (threshold: 74)."
                , (grade ? "passed" : "failed")
                , results.o);
    console.log("=> LoadTime %s with %s (threshold: 1000ms)."
                , (loadtime ? "passed" : "failed")
                , results.lt);
    process.exit((grade && loadtime ? 0 : 1));
});

Run with:

$ node ./bench.js

Sample output:

YSlow Benchmark (/about)
=> Grade passed with 74 (threshold: 74).
=> LoadTime failed with 3113 (threshold: 1000ms).

Note: I intentionally set my load time low to fail, for examples sake.

Real Example

Well now, that's cool, but not really very flexible. Just like in my last post on the subject, let's move on to a real world example.

Next, it's add the ability to include a JSON file containing a list of URLs and their respective thresholds. Setup a configuration file to include an array of objects, each containing a threshold for the yslow grade (o) and load time (lt). Both key names match what's returned by the PhantomJS yslow.js script that YSlow.js on Node.js uses at it's core.

file: tests.json

[
  { "url": "/"                 , "o": 70, "lt": 3000 },
  { "url": "/about"            , "o": 74, "lt": 2000 },
  { "url": "/projects"         , "o": 74, "lt": 2000 },
  { "url": "/archive"          , "o": 74, "lt": 2000 },
  { "url": "/search?q=httperf" , "o": 74, "lt": 1000 }
]

In this bench.js example, I get a bit more dynamic, adding a couple of extra lines to support testing page weight (w) and http requests (r). I did not include them in tests.json due to their unpredictability. Additionally, the dynamic method below can be used cleanly with HTTPerf.js in place of YSlow.js.

file: bench.js

#!/usr/bin/env node
var tests     = require('./tests.json');
var YSlow     = require('yslowjs');
var failed    = 0;
var hostname  = "http://mervine.net";

console.log("YSlow Benchmark (%s)", hostname);
tests.forEach( function(test) {
    var yslow  = new YSlow(hostname+test.url);
    var results = yslow.runSync(); // best to not run run this async
    console.log("\n* %s", test.url);
    Object.keys(test).forEach(function(key) {
        if (key === 'url') { return; } // skip url
        var passed;
        if (key === 'o') {
            passed = results.o >= test.o;
        } else {
            passed = results[key] <= test[key];
        }
        console.log("=> '%s' %s with %s (threshold: %s).", key
                    , (passed ? "passed" : "failed")
                    , results[key], test[key]);
        if (!passed) { failed++; }
    });
});
if (failed) { // zero === false
    console.log("\nFailed: %s", failed);
}
process.exit(failed);

Run with:

$ node ./bench.js

Sample output:

YSlow Benchmark ()

* /
=> 'o' passed with 86 (threshold: 70).
=> 'lt' passed with 1841 (threshold: 3000).

* /about
=> 'o' passed with 97 (threshold: 74).
=> 'lt' passed with 708 (threshold: 2000).

* /projects
=> 'o' passed with 94 (threshold: 74).
=> 'lt' passed with 1325 (threshold: 2000).

* /archive
=> 'o' passed with 94 (threshold: 74).
=> 'lt' passed with 1270 (threshold: 2000).

* /search?q=httperf
=> 'o' passed with 93 (threshold: 74).
=> 'lt' passed with 910 (threshold: 1000).

What Next?

Well for me, it's Bench — An extremely raw and super alpha utility for benchmarking and graphing web pages.

bench screenshot

This is an example of what can be done with YSlow.js, HTTPerf.js and Phantomas (via Phapper).

More Info

You can view details on YSlow.js for Node.js, Phapper, HTTPerf.js and my other projects at mervine.net/projects or on github.

Good luck and happy benching!