Startups: When Should You Do QA?

As a startup founder, one of the most critical decisions you'll face is how much time and resources to invest in testing and quality assurance (QA). It's a question that every startup grapples with at some point: do you prioritize building new features and generating revenue, or do you focus on creating a robust testing infrastructure to ensure the long-term stability and reliability of your product?

In my experience, the answer is not always straightforward. When I founded my first company, VinSolutions, we grew the business to $35 million in annual revenue over 8 years without hiring a single QA person until the last year or two before we sold the company. 

We had a small, tight-knit team of developers who knew the codebase inside and out, and we didn't have any unit testing, automated QA, or manual QA in place.

At the time, it felt like the right decision. We were chasing revenue, trying to get our product to market, and win customers. Investing in testing and QA seemed like a luxury we couldn't afford – it felt like it would slow us down and divert resources from our core mission.

However, as the company grew and the codebase became more complex, we started to see the limitations of this approach. When new developers joined the team, they struggled to make changes without breaking things. The lack of testing made the codebase more fragile and harder to maintain over time.

So, when is the right time for a startup to invest in testing and QA? Here are a few key factors to consider:

1. The size and complexity of your team

If you only have one or two highly skilled developers, you may be able to get away with minimal testing. However, as your team grows and becomes more diverse in terms of skill levels and experience, the need for testing becomes more critical. Not every developer is equally skilled at testing every possible scenario, and it's easy for bugs and issues to slip through the cracks.

2. The nature of your product

The level of testing you need depends heavily on the type of software you're building. If you're creating a simple reporting dashboard, the consequences of a bug might be relatively minor – you can fix it, and users can log back in the next day. However, if you're building software that controls an airplane, the stakes are much higher, and you need to invest heavily in testing to ensure safety and reliability.

3. The long-term vs. short-term tradeoff

In the short term, investing in testing can feel like a drain on resources that could be better spent elsewhere. However, in the long run, a lack of testing can create significant technical debt and make it harder to innovate and adapt to changing market conditions. As a founder, you need to weigh the short-term benefits of speed and agility against the long-term benefits of stability and maintainability.

In my experience, the best approach is to find a balance that works for your specific situation. At my current company, Full Scale, we offer manual QA services that allow startups to embed a QA person into their development team on a week-to-week basis. This can be a cost-effective way to get the benefits of manual testing without the overhead of hiring a full-time QA team.

Automate All the Things

On the automation side, I'm a big believer in building a robust suite of automated tests, but I also recognize that it's not a replacement for manual testing. Some things, like complex integrations with third-party systems, are simply too difficult to automate effectively.

Instead, I recommend at a minimum starting with a set of basic "smoke tests" that cover the core functionality of your system. Every time you deploy a new version, you should run these tests to ensure that critical functions like login, navigation, and key workflows are still working as expected.

As your product and team mature, you can gradually expand your automated testing suite to cover more edge cases and scenarios. However, it's important to be strategic about what you choose to automate. Focus on tests that provide the most value in terms of catching bugs and ensuring reliability rather than trying to automate everything for the sake of it.

Another key consideration is the use of unit testing. In my experience, unit testing is essential for certain types of code, like complex algorithms or rule engines. In these cases, even small changes to the logic can have far-reaching consequences, and unit tests are the only way to catch potential issues before they make it into production.

I would caution against building unit testing for everything. Code that does a simple SQL query and serializes it to JSON to the front end likely has no actual “logic” to test multiple scenarios on. Why would you mock something that doesn’t even have any logic to test?

When QA Testing Can Help You Go Faster

While it may seem counterintuitive, there are situations where investing in QA testing can actually help a company move faster and be more agile. Let's consider a real-world example to illustrate this point.

Imagine you're a startup that has built a successful product in the marketing automation space. You've grown rapidly, and your product has become quite complex, with numerous integrations and features. Suddenly, a new opportunity arises in the market – a change in consumer behavior or a new platform emerges that you need to adapt to quickly.

In this scenario, having a robust suite of automated tests can actually help you pivot and adapt much faster than if you had no testing at all. Here's why:

1. Confidence in making changes: With a comprehensive set of tests in place, your developers can make changes to the codebase with confidence, knowing that if they break something, the tests will catch it. This reduces the fear and hesitation that can often slow down development velocity.

2. Faster iteration: Automated tests allow you to verify the stability of your product much faster than manual testing. You can run thousands of tests in a matter of minutes, whereas manual testing would take days or weeks. This faster feedback loop means you can iterate and improve your product more quickly.

3. Reduced regression risk: As you pivot and adapt your product, there's always the risk of introducing regressions – breaking existing functionality in the process of adding new features. With a robust test suite, you can catch these regressions early, before they make it into production and impact your customers.

4. Easier onboarding of new team members: If you need to scale your team quickly to meet the demands of your new opportunity, having a well-tested codebase makes it much easier to onboard new developers. They can rely on the tests to understand how the system works and to catch any mistakes they might make as they learn.

In this type of scenario, the upfront investment in testing pays off in the form of faster adaptation and a more agile development process. The tests act as a safety net, allowing the team to move quickly and confidently without fear of breaking things.

Of course, this doesn't mean that you should aim for 100% test coverage or that testing is a silver bullet. There are still tradeoffs to consider, and it's important to focus your testing efforts on the areas that provide the most value. But in situations where speed and agility are paramount, a solid testing infrastructure can be a key enabler rather than a hindrance.

Real World Examples

At my previous company Stackify, we had to build what most people would consider integration tests that were absolutely mission-critical. Our software collected performance data for different programming languages and tech stacks, and we had to create complex rules to process that information and generate the desired output. Without those integration tests, there was no way we could confidently make changes to the logic without breaking things. The integration tests were essential to maintaining the integrity of the system.

Similarly, at my current company, At Capacity, we have a rules engine that controls Google Ads based on factors like time of day, day of the week, and various performance thresholds. The logic is so complex that it would be impossible to build and maintain without extensive unit testing. Even small changes to the rules could have unintended consequences across a wide range of scenarios. In cases like these, unit testing is not just nice to have – it's a hard requirement.

However, for simpler code, like basic CRUD operations or JSON APIs, unit testing may not provide much value. Writing tests for the sake of writing tests is not a good use of developer time and can actually slow down development velocity. As an example, if you have a simple API that retrieves a customer record from a database and returns it as serialized JSON, there's little point in writing a unit test that mocks the database query. The test doesn't add much value, and it's unlikely to catch any meaningful bugs.

Like Everything… “It Depends”

Ultimately, the decision to invest in testing and QA comes down to a careful consideration of your specific circumstances as a startup. You need to weigh the short-term benefits of speed and agility against the long-term benefits of stability and maintainability, and find a balance that works for your team and product.

One thing is certain, though: no matter how much testing you do, it's impossible to catch every possible issue before it happens. That's why it's critical to invest in monitoring and error tracking alongside your testing efforts. Tools like Stackify, Datadog, and New Relic can help you proactively identify and fix issues in real-time, even if they weren't caught by your tests.

In the end, building a successful startup is all about finding the right balance between short-term and long-term thinking. Testing and QA are no exception. By being strategic about when and how you invest in these areas, you can set your team up for success and build a product that stands the test of time.

As a founder and CTO, I've learned the hard way that there are no easy answers when it comes to testing and QA. What works for one startup may not work for another. But by carefully considering your specific circumstances, looking at real-world examples like Stackify and At Capacity, and being deliberate about your approach, you can find a testing strategy that works for you and helps you build a stronger, more resilient product over time.

By the way, if you need help hiring QA developers, we can help you at Full Scale!