Notes on Performance Testing

A couple weeks ago I did an ad hoc talk at the LAWebSpeed Meetup, hosted by MaxCDN, on general performance testing in the web world. I was asked to put together a list of the tools I spoke about and some brief notes on them, so here goes.

TL;DR

Here's a brief list of the tools I mentioned in my talk, for those that were there.

Server Side Tools

Client Side Tools

Server Side Testing

In general, I use server side load / performance testing with two goals. The first is break testing, finding your applications break point and monitoring it's behavior on the way to breaking. The second is for a solid understanding of current state capacity, both for the purpose of future planning and to avoid critical failures.

When running break tests, I run multiple tests, increasing the load in each until a break point is found, which is what Autoperf (see below) was designed for. These tests can often be done in short controlled bursts as opposed to long running break tests, although depending on your platform (i.e. Java) there are exceptions. Additionally, I tend to point these kinds of tests a a single endpoint or a few select higher priority endpoints as opposed to a more general traffic log. Again, the effectiveness depends greatly on platform and application design.

In testing for capacity, I take the sweet spot from the break tests mentioned above and I run those over long periods of time, and attempt to more closely reproduce real production traffic (the closer the better). Once I've validated the sweet spot, I'll use that to determine capacity.

Note on capacity planning.

This will greatly depend on your traffic patterns and application designed. As such, there's no real formula out there. My typical rule of thumb is "400% peak", with a minimum of "250% peak". How "peak" is defined will vary almost from site to site, however, if you're familiar with your sites traffic patterns, it shouldn't be too hard to figure out.

Tools

httperf

For both break testing and capacity testing, I use httperf - a command line tool for Linux which is designed to generate load at a controlled rate. There are rpm's and deb's for it, but I use my own fork with more verbose output, which I'll talk about more later. For more information, see Performance Testing with Httperf and httperf-0.9.1 with individual connection times.

Some other popular options for generating load are Siege, and Apache Bench (aka Ab). Siege is designed for time based break testing and is good at it, but has limited results data and I've found it to be less predictable the httperf, that said I haven't used it too much. I haven't personally played with Ab, as in my initial inspection it appeared to require the install of Apache, which I had no interest in doing and as I already have a tool to meet my needs, I didn't spend too much time digging deeper.

Autoperf

Autoperf is a ruby driver for httperf, designed to help you automate load and performance testing of any web application - for a single end point, or through log replay.

This has been refactored from the original -- github.com/igrigorik/autoperf -- to include HTTPerf.rb as a simplification and to include additional features. This has been refactored and released with permission from Ilya Grigorik, the original author.

HTTPerf.rb

HTTPerf.rb is a simple ruby interface for httperf, which is used under the hood of Autoperf. It can also be very useful on it's own for running individual tests with httperf and manipulating the results programmaticly.

I have also ported HTTPerf.rb to several other languages, although not always with all the time, care and features that the original ruby implementation had.

HTTPerf.rb ports:

For more information on HTTPerf.rb see Automating Performance Testing with HTTPerf.rb.

Client Side Testing

Although some may disagree with me, I consider client side performance testing about as far from an exact science and you can get in the world of automated testing. There are too many external variables, possible data points and decisions to make on what's important to your business model.

A great read on this topic is Improving performance on twitter.com, which while more targeted at overall performance improvements and monitoring, has some good ideas that can carry over to what to look for when client side performance testing -- namely the "Time to First Tweet".

Tools

WebKit's Navigation Timing API

W3C Navigation Timing is a collection of performance data generated by most modern browsers which contains timing for key events of the page load.

Note: I've linked to an article on it by Ilya Grigorik because reading spec's is just no damn fun.

WebPageTest.org

With WebPageTest.org run a free website speed test from multiple locations around the globe using real browsers (IE and Chrome) and at real consumer connection speeds. You can run simple tests or perform advanced testing including multi-step transactions, video capture, content blocking and much more. Your results will provide rich diagnostic information including resource loading waterfall charts, Page Speed optimization checks and suggestions for improvements.

Google PageSpeed

Google PageSpeed is a collection of client side testing tools from Google.

"Analyze and optimize your website with PageSpeed tools to implement the web performance best practices.

Fast and optimized pages lead to higher visitor engagement, retention, and conversions. The PageSpeed family of tools is designed to help you optimize the performance of your website. PageSpeed Insights products will help you identify performance best practices that can be applied to your site, and PageSpeed optimization tools can help you automate the process."

YSlow

YSlow from Yahoo! analyzes web pages and why they're slow based on Yahoo!'s rules for high performance web sites.

PhantomJS

PhantomJS is a headless WebKit scriptable with a JavaScript API. It has fast and native support for various web standards: DOM handling, CSS selector, JSON, Canvas, and SVG. With it, you can easily and headlessly automate and collect data about your sites client side performance.

A few tools that leverage PhantomJS include: