People often equate load testing with performance testing. Load testing is seen as a way of answering the question “How fast does the system respond?” This view then tends to mean that load testing is seen as an end of project activity. Only at the end of development will we have the final implementation for performance testing and so we can confirm only then that it performs quickly enough in the real world and smoothly transition into live service.



Wrong approach! This is extremely risky and misses out on the many benefits of starting load testing early and applying it throughout the project. With this approach does the system sail through load testing and transition smoothly into service? Occasionally yes. But more frequently the system starts to fail as load starts to be applied, even with small increases in volume.. For the first time there are concurrent demands on the system and arbitration over resources is required. Paths through code that have never been executed are triggered, situations arise that nobody really thought through. Transactions fail. Systems crash. After these problems are fixed and more load is applied in a test, we then encounter problems like resource exhaustion, buffer overflows, timeouts and inconsistent behaviour. The real work needed to turn a functional pre-production system into a robust solution has only just begun.

Examples abound of products that failed when load testing started and, after lots of effort, stress and expenditure, have been shelved. Worse still are the ones that missed load testing altogether and failed dramatically during live operation. An internet portal developer recently stopped development of a new service, one that had completed functional development, when load testing revealed fundamental structural problems and inefficient coding which led to a poorly performing and unstable system.

So what should you do to avoid these risks? We all know it is better to find faults early when they cost far less to fix yet load testing is still left until as late as possible. The types of faults it finds frequently need architectural changes and major rewrites which are by then are hugely expensive to implement. The answer is that you should start early. Different forms of load testing should be repeatedly applied throughout the project to identify problems early and to check that the system is not going off track.

This is a natural extension of the practice of test led development. Test led development, where automated tests are written first and code must pass these tests as it is developed, offers major benefits. However, in its current form, the focus of this testing is on functionality. As it evolves the functional status of the software is always known and hence manageable, functional faults are nipped in the bud avoiding high cost fixes, the functional risk is greatly reduced. Not so other risks. If a project performs early and continuous load testing it gets much wider and comprehensive risk reduction. To make this effective:

1. Study the system and perform a risk analysis to help to order the threats to the system, this will help you to prioritise load testing activities.

2. Collect data to allow comparison of the efficiency of different builds. This permits monitoring of the long term trend, “Is the system using more and more processor time to do the same work?” This data can be used to predict resource requirements at different levels of demand and so support scalability predictions.

3. Execute tests that aim to assess the behaviour of the system and to trigger faults under load. Use workloads that simulate expected patterns of demand to observe the aggregate behaviour of the system. Use specially targeted extreme workloads to probe the vulnerabilities of the system.

4. Include the full spectrum of load tests into the test suite. This means performance testing with typical and busy period work loads; stress testing to check both atypical demand spikes and resource exhaustion impacts; endurance testing that uses both operational period and cumulative operation tests; reliability testing that runs lots of transactions and then checks whether occasional transactions fail; concurrency testing of two users working on the same account at the same time.

5. Design measurement activities as scientists would design an experiment, design them to provide data that can be analysed. Sample the system under different steady state workloads to provide multiple data sets to support interpolation. Chose the workloads to permit estimation of the resource costs for each transaction type.

6. Target the middleware first with generic activities and evolve the suite as functionality is developed. Start early and then test each incremental release of the system, firstly with the previous suite and then with a modified suite that addresses new functionality.

7. Invest the time and resources to work at a representative scale. Maybe the test bed can't be full scale but it should not be two orders of magnitude smaller than the intended system. Be smart and innovative to use resources effectively to provide an appropriate scale test bed. The costs that will be incurred if this is not done will far exceed the cost of providing the test bed.

8. Don't delay; test an increment as soon as possible. Don't skip one or you'll end up skipping them all. Compare the measurements and behaviour with the previous one, is it better or worse?

9. Provide a background load for functional testing. Features that work offload may fail when the system has other things to think about.

10. Consider occasional events such as server failures and reconfiguration of the system. Do these need to be tested under load?

In conclusion, you need to incorporate load testing throughout the development process. Leaving load testing until the final run in to live service is a recipe for disaster. If this became common practice then a lot more applications and systems that work would be delivered on time and to budget.