By Brian J. Noggle
The vendor’s salesman/trainer sat next to me at the conference table, his laptop open between us with the software package that my employer and his chief developer had selected before they hired me as their professional quality assurance person. The vendor was demonstrating the load testing capabilities of the package, how it could take a regression test script and run it with simultaneous virtual users to create a load test. He showed me how to establish the ramp-up time, wherein I could add five users every thirty seconds to gently increase the load on the application under test.
He slid his laptop closer to me and let me give it a try. I set the virtual user number to the licensed maximum, 200, and set the ramp-up time to 0, and then I clicked the button to launch the test, bringing a virtual thunder strike of load against the Web application.
“What are you doing?” the seller whimpered, sounding not unlike the examiner during my first driver’s license test.
What was I doing? I was certainly ignoring the white-gloved niceness of performance testing, where one uses the tool to generate a gentle upward slope or step line of user load. Performance testing that allows one to nicely project how the delicate little application will perform with 25 users on a sunny Saturday afternoon in late May, when the winds are low and the cumulous are fluffy harbingers of continued good weather.
Your performance testing strategy needs not only to account for a nice, gradual increase in Web traffic and to create a sense of well-being in knowing that, in optimal circumstances, your Web application won’t falter. Your testing strategy must also challenge your application as much as possible, up to and including launching all of your virtual users at once, even targeting all virtual users at a single function instead of using “realistic” user scenarios.
Once your application is live, any number of circumstances will mimic that load testing assault except that the user load will probably exceed your virtual user licenses. Many scenarios might arise, often without input from the IT staff developing or deploying the application that can suddenly and dramatically increase traffic to your application. In many of these circumstances, your organization’s IT staff might only know about the traffic spike when it happens.
Circumstances Outside the IT Realm Change
Business considerations outside the IT world can lead to a sudden spike in user activity. Recently a backend processing provider for health insurance prescription benefits ended its relationship with a large pharmacy chain in the United States. (1) As a result, other pharmacies began transferring many millions of prescriptions records, many of them in the first few weeks of the year. The hundred million or so records represented an addition to the normal daily transactions.
My pharmacist reported that was down for a whole day during the first weeks of January when angry customers queued at service counters and shook their fists. One might conclude that a load issue caused the failure (although an insider reported that it was the common promote-untested-code-to-production practice that caused the outage). Still, it provides an example where a business decision, a partnership ending, or other circumstance outside of IT’s control could cause a surge in load for which your gradient load tests don’t account.
Your Application Goes Viral
You probably already know the meaning of the Slashdot Effect, but I’ll explain it anyway: When the technology blog Slashdot (2) mentions a Web site or article, the resulting traffic often overwhelms underpowered Web sites. If everyone who reads this magazine now goes to access the given link for the Slashdot website we are by association increasing their traffic. In the years since the Slashdot effect was first noted, other highly trafficked Web sites and social media, such as Facebook, Twitter, and Reddit, have arisen with the ability to focus a large number of users onto a Web site or application very suddenly.
Your organization might only discover the traffic spike when it happens, or when your application breaks because the thousands of new users arrived without stepping up gradually, patiently queuing in an orderly fashion to wait for applications, pools, or database connections to become available. Viral users are more like rowdy football fans, all scrambling for the entrance at once, than theatre patrons.
A Media Blitz
If your organization or application faces consumers instead of industry professionals, at some point the company might decide on a media blitz to drive consumer traffic, and this media planning might not take into account technical considerations. In my time at an interactive agency, we maintained a web site for a consumer packaged good complete with streaming media and data collection forms. Some months after we deployed the site, the client decided to promote the product through non-Internet means, including television commercials, print media, and press releases that gained the notice of national television programs.
Since these decisions lie outside of IT, sometimes they can be opaque or unnoticed. Unless you ensure your applications can handle that user assault at the beginning, you might be called upon to pick up the pieces very suddenly when your URL appears on the telly.
Don’t be Gentle
When you have a load testing tool at hand, you should use it not only to provide some idea of how your application will perform under load (the answer, regardless of the numbers, will probably be “good enough”). Instead, you should also use that load testing tool to try to break the software through testing the application with the maximum number of users all at once to see if your application can handle that meager number of users without choking on the various sorts of resource allocation issues that you would not find with a measured increase in test load. This is important because no matter what usage level your organization anticipates for your application, once you turn it on, you might find that the actual usage can suddenly and catastrophically spike beyond even your worst sleepless nightmares.
- What is the “Slashdot Effect?” – http://slashdot.org/faq/slashmeta.shtml#sm600
Author Profile – Brian J. Noggle
Brian J. Noggle has worked in quality assurance and technical writing for over a decade, working with a variety of software types and industries. He currently works as a freelance software testing consultant through his own company, Jeracor Group LLC and has recently published a novel set in the IT world, John Donnelly’s Gold.