“I've finished the feature, I'm just updating tests now”. I hear this several times a week from developers I work with, usually a long while before their feature actually goes live.

Automated tests are an extremely effective practice in software quality control. They can be written once, maintained as the codebase evolves, and be effortlessly re-run an infinite number of times. Tests can automate the equivalent of hours of manual testing in minutes, and I'm a strong supporter of them, but they have their drawbacks.

Tests cost money to write, money to maintain, and can be painfully slow to run at scale. In many teams, intermittent failures also hold up deployments for hours each week while frustrated developers fix temperamental tests.

We also often forget that tests don't actually guarantee anything. When they pass, they're just telling us that our potentially buggy test code has determined our potentially buggy application code is behaving as expected under the conditions we thought to check. There's a lot of room for false positives (and negatives) there.

Unfortunately, relying on poorly maintained test suites as the only means of quality control means many teams suffer through 50% or more of their developer's time being lost to test maintenance. To improve the situation, we need to recognise our tests are just one of many tools we have available.

Making tradeoffs

In order for a test to be worth having, the ongoing cost of development and maintainance needs to be significantly lower than the ongoing cost of not having it at all. This tradeoff isn't always worth making.

Rather than defaulting to tests for everything, we should look at how our application is going to be developed and used, and think broadly about the various tools we have available to maintain quality. We can then compliment lean, valuable test suites with tools and processes that allow us to maintain quality while keeping the tradeoffs working very much in our favour.

An example could be needing to support Internet Explorer (IE) on a single-page JavaScript web application. It would be tempting to make the tests run against IE in addition to modern browsers, but in doing so we'd increase the test suite runtime, and we'd force developers to cater to the quirks involved in automating multiple browsers going forwards.

Since only a small number of our customers are likely to still use IE in 2021, a better approach might be to use an error tracking service to alert us to any IE errors raised in the live environment. This way, the occasional IE user might stumble on an error, but it can be quickly identified and fixed around other work. It's a much more sensible tradeoff between application quality and ongoing cost.

Tools and processes for code quality

Looking beyond tests, there are many tools and processes available to us that we can use to maintain software quality. Many of them are things we do already, but don't use to their full potential. Here are a few I regularly encourage teams to do more of:

• Improving developer documentation.
• Spending time planning new features.
• Spending time on architectural discussions.
• Frequent pair programming.
• Improved code linting.
• Introducing code typing.
• Better code review.
• Targeted manual testing.
• Periodic accessibility and user experience reviews.
• Improving application logging.
• Using an automated error tracking service.
• Using an application metrics monitoring service.
• Using (or understanding) a product analytics service.
• Developer experience improvements.
• Proactively removing old/unused features.
• Proactively collecting customer feedback.

None of these are a complete solution to quality, and each of these have their own tradeoffs to consider. Rather than replace tests, they can be used together with tests to provide a more complete approach to maintaining quality software. They can help to restore a healthy tradeoff between feature development and quality control, enabling developers to iterate quickly and ship with less frustration.

So, next time you find yourself spending the majority of your development time tending to a test suite, consider what other tools and processes you have at your disposal to improve software quality, and make some better tradeoffs.

I make a living by helping companies bring digital products to market with solid foundations, room to scale, and costs under control. If it sounds like I could help you, have a look over my services. If you'd like to hear more from me, you should follow me on Twitter, for more product development content in future.