Every November, hundreds of thousands of hopeful ticket buyers’ hearts sink when they are served this page:
People inevitably complain about the poor performance of the Glastonbury ticket sellers’ website, but that page is actually evidence of the website performing exactly as expected. Demand for tickets outweighs the website’s capacity (and actual tickets), but instead of the website crashing, buyers are placed in a queue and processed accordingly. In contrast, millions of Swifties overwhelmed Ticketmaster’s site trying to buy tickets for the Eras tour, resulting in hours of downtime for disappointed fans.
Ticketmaster failed to prepare their systems for the amount of load in this extreme case.
But performance issues don’t just stem from traffic spikes — technical incidents and misbehaving third-party integrations can also bring your application down. Instead of being reactive, you can prepare yourself for performance issues or even avoid them altogether by load testing.
Why does load testing matter?
Proactive performance testing is critical for companies whose core business is online, but even for businesses that mostly operate offline, performance degradation is not just a loss of reputation or revenue. Restoring service and tracing the root cause of a performance issue in production require engineering time, especially if you don’t have tracing and observability set up. When performance problems are discovered in production instead of during testing, it can disrupt regular work and you could end up needing to troubleshoot outside of working hours.
This is part 1 of a series introducing you to load testing. In this part, we cover what load testing is for and what it can help you do, as well as the role of a performance tester. In the parts that follow we’ll cover the different types of load tests and what they can tell you, and give an overview of the steps involved to get started with load testing and reporting results.
By the way: This series of articles is based on my nearly 20 years of experience with performance testing in gaming, e-commerce, network infrastructure and finance. I maintain the open source load testing tool Locust, and recently started locust.cloud so you don’t have to do the heavy lifting of setting up and maintaining load testing infrastructure.
Goals of load testing
Gather performance requirements and discover your limits
Some companies have known performance requirements (such as supporting a certain number of transactions per second and a maximum acceptable response time). Sometimes you can estimate your load requirements based on past data or market analysis, but it’s equally common that you won’t know exactly what your requirements are. A lot of the time you’ll want to aim for discovering your limit (sometimes called breakpoint testing), not just testing to meet a specific requirement.
When load testing, as you increase the load, at what point is the system “saturated” so that throughput goes flat or even starts turning down? In the graph below, you see this as a sharp increase in response times at a particular load:
This test has found the maximum throughput of our system, and the question then becomes: Is it good enough? And if not, what can be done to improve it?
Reproduce a performance problem
In many cases production issues emerge only when there is an unusual amount of load (such as Black Friday, a product launch, a sporting event, etc). Finding the root cause of the issue can be hard to do after the fact, and you can’t afford to wait around for the next organic load increase to be able to see what’s happening. With load testing you can reproduce the behavior in a test environment to gather more information and validate a fix.
Validate performance of new features or changes
You can also use load testing to check if an architectural change the team has been working on performs as expected, or validate that a change improves performance.
Test scalability
When load testing, it’s important not just to test absolute performance, but also scalability: you may have the ability to spin up another instance of a service when under demand, but do you know if the service is actually twice as fast with two instances? Sometimes throwing extra resources at a problem doesn’t actually solve performance problems. Also, in a lot of cases, some part of your system might not be horizontally scalable, and ends up becoming the new bottleneck as you ramp up additional instances of your service.
The role of a performance tester
Own performance testing architecture
As a performance tester, you’ll be responsible for maintaining and improving the performance testing architecture, including performance tests, test setup scripts, and maintaining load generators (unless you’re outsourcing these to a SaaS provider like locust.cloud).
Plan performance testing for new features and projects
You’ll also plan or help to schedule performance tests for projects. Ideally, these should happen early enough to have time to fix issues, and late enough that regular functional testing can detect issues that can be fixed before throwing off load tests.
Advocate for improvements
At smaller companies, you may be both writing the code for the application and performance testing it. This can make things simpler because you’re in a position to fix any issues you discover. At the other end of the spectrum, you might be a dedicated performance tester, in which case you likely don’t know all the internals of the system you’re testing and are certainly not going to be able to fix performance issues on your own. Your challenge then becomes communicating the impact of the performance issue to the development team responsible, and advocating for a fix (we’ll cover communication in more detail in an upcoming post).
As a performance tester, it will often be up to you to drive performance improvements, because most likely nobody else will. There’s usually far greater pressure from the business side for functional improvements to your application — except for when there is an actual performance issue in production (then suddenly everyone will be very interested in performance).
Start load testing without the heavy lifting
My experience with performance testing has convinced me that companies shouldn’t be running their own load testing infrastructure. If you’re here because you’re setting up load testing for your company, locust.cloud provides an easy way to get started with load testing, and supports load tests, big and small. Locust Cloud gives you access to hosted, easily scalable, and distributed load generation, as well as advanced reporting — all while preserving the flexible “it’s just Python” approach to load test scripting that Locust provides.
Rebecca Dodd contributed to this post.