Four reasons developers should write their own load tests
Around a year and a half ago my team started performance testing our own services, rather than relying on help from a dedicated performance team. There were all sorts of benefits that I hadn’t expected, and I’m now convinced that this should be standard practice. Here’s why:
Developers have the most skin in the game
Over recent years development teams have assumed more and more responsibility. They no longer just write software: they also test it, deploy it and keep it healthy in production. If developers are responsible for the health of their services then they need to be confident that they’ll perform well under real load. Load testing is a great way to build this confidence.
Carrying out performance tests and examining how my services responded gave me a much deeper understanding of my software’s performance characteristics. I learnt how to make sense of its metrics. This knowledge meant that I was in a much better position to understand what I was seeing in production and made diagnosing and fixing live problems faster.
Load tests as code
Keeping your load tests as code alongside your application is a great way to keep them up to date. If a developer adds or changes an endpoint then they can update the load tests in the same pull request. The advantage here is that your load tests are always ready to use, and you don’t need to wait for a performance testing expert to write them for you.
We used Rake to generate test data and RubyJmeter to configure JMeter. This meant that we could carry out a load test on demand with a single command. RubyJmeter provides a reasonable DSL for configuring JMeter and is much nicer that JMeter’s native XML files. However you may be able to find more modern tools better adapted to load tests as code.
Load tests let you experiment
Once running a performance test is easy developers start using them to experiment. They make tricky questions easy to answer. They give you the ability to try things out and measure the impact under load rather than making ‘educated guesses’ and hoping for the best! Here are a few examples of things we learnt using our load tests over the last 18 months:
- How to configure our app to handle database fail-overs gracefully. This would have been close to impossible without the ability to easily apply load to a testing environment.
- Whether a migration was safe to run under load in the production environment. When we found out that it wasn’t we tested alternatives until we found one that worked.
- How our app would perform if an external dependency became slow or went down.
- What the source of a threading related problem was, and how to fix it.
- What storage type our database should use.
- How best to configure our web server.
- Which version of a code change performed best.
In short, using load tests to power our experiments helped us learn a huge amount about our application and significantly improve its robustness and performance without interfering with production traffic.
Developers know their apps best
When you put together a load test you aim to simulate real traffic, often using synthetic data. It can be difficult for dedicated performance testers to know what type of variety the data needs to show, and what edge cases must be included in order to put together a representative test. They often make assumptions about how the application works when designing the test, and not all of them turn out to be correct. Developers know their applications best, and so they’re in a great position to write the load tests.
Whilst developers know their services best, they often don’t have a good understanding of how all of an organisation’s services fit together. For that reason I think it makes most sense for dedicated system test or performance teams to carry out system level performance tests. Developers are usually not experienced load testers, so performance teams also have an important role in supporting and educating them on testing their own systems. They’re usually more aware of things like GC or JVM behaviour, database optimization, connection pooling and network latency, so don’t hesitate to leverage this knowledge.
I hope you’re convinced that developers should performance test their own code. Personally I’ve found it crazy-fun and there’s nothing that has helped me understand my applications better or given me more confidence in my services. With all that said, I made (or saw) plenty of load testing mistakes over the past 18 months, so check out my follow-up post on four load testing mistakes developers love to make.