Designing for Testability: Breaking the Next Barrier

In our ongoing DevOps for the Win series, we’ve been documenting our DevOps transformation journey. In Chapter 1, we introduced the DevOps Triangle: Tools, Architecture, and People in Product Development, where we discussed the foundational elements that drive DevOps success. In Chapter 2, we explored The First Steps in Our DevOps Transformation, sharing how we laid the groundwork for meaningful changes.

Now, in Chapter 3, we tackle the next major challenge we encountered: Designing for Testability. After improving our team structure, refining our Definition of Done, and implementing automated code analysis, we realized that the primary bottleneck in our pipeline had shifted to software design-specifically, testability. Work was piling up in the testing phase, preventing us from achieving faster release cycles. This chapter delves into how we overcame this challenge and improved our ability to deliver high-quality software efficiently.

The Testing Bottleneck

The main reason for this delay was that testing was predominantly manual, focusing on UI testing and user workflows. Since the UI was usually one of the last items to be completed, there was little to no automation in place, and manual testing became the only viable approach. Automated UI test cases (e.g., Selenium tests) were typically written after manual testing was completed, primarily for regression purposes.

This reliance on manual testing resulted in-

• Significant delays in software releases.
• The need for multiple deployments to allow parallel testing by more than one QA team member.

This highlighted the importance of making the code more testable to optimize the flow of value. However, achieving this turned out to be more difficult than we had anticipated.

Prioritizing API Tests and Unit Tests

To address this challenge, we focused on two main areas:

1. Unit Tests - Ensuring individual components could be tested in isolation.
2. API Tests - Reducing reliance on UI-based testing and database dependencies.

API Testing: Shifting Left for Faster Feedback

We redefined our API testing strategy with these key improvements:

1. Well-Defined APIs- APIs were designed to be simple, well-documented, and available early in development.
2. Avoiding Database as an Integration Point- Relying on databases for integration created dependencies that slowed down testing. Instead, we moved towards API-based integration using REST over HTTP and GraphQL. This minimized database setup time and improved test automation.

This shift significantly reduced delays, allowing faster automation and early-stage testing, showing that focusing on API-first designs improved both testability and efficiency.

Unit Testing: A Mindset Shift

Initially, our unit testing efforts led to rapid test coverage increases, but we quickly identified issues:

• Some tests were superficial, written just to meet code coverage targets.
• They lacked meaningful validation of functionality, edge cases, and failure scenarios.

To counter this, we emphasized:

• Educating developers on writing valuable tests.
• Refactoring code to improve testability.

Challenges in Writing Testable Code

The biggest roadblock to effective unit testing was the lack of testability in the codebase itself. Key issues included:

• Large, monolithic functions.
• Tightly coupled components.
• Poor separation of concerns.
• Insufficient abstraction.

These issues made it difficult to isolate and test individual units effectively. To address this, we:

1. Provided Guidelines: We shared best practices on:
   • Selecting functions for unit testing.
   • Refactoring code to improve testability.
   • Designing and using mocks to facilitate testing.
2. Focused on Incremental Improvements: Developers were encouraged to make small, meaningful changes to improve testability over time.

Slow but Steady Progress

Despite these efforts, progress was slow, especially for legacy products. The existing architecture limited the number of unit tests we could write. However, these actions weren’t just aimed at improving the current codebase—they were an investment in the future:

• Improving developer skills in designing testable code ensured that future projects wouldn’t suffer the same issues.
• Incremental improvements prevented disruption while steadily increasing test automation coverage.
• A shift in mindset helped teams see testability not as a burden but as a necessity for sustainable DevOps success.

For teams and organizations embarking on a DevOps transformation, testability is a critical area of focus. It helps alleviate bottlenecks in the development phase. However, it’s important to manage expectations—immediate results may not be possible, especially when dealing with large legacy codebases. Refactoring such code without sufficient unit and regression tests is a major challenge.

Lessons from The Pragmatic Programmer

For scenarios where starting with a new codebase isn’t feasible, The Pragmatic Programmer: Your Journey to Mastery, by Andy Hunt and David Thomas offers actionable guidance:

• Target Incremental Changes: Focus on refactoring small, manageable parts of the codebase.
• Decouple Components: Reduce dependencies to make individual units easier to test.
• Break Down Monolithic Functions: Split large functions into smaller, more focused units.
• Adopt Modular Design: Make the code more testable and maintainable by improving modularity.

The message here is clear: once the initial bottlenecks have been addressed, testability should become a key focus area. This will ease constraints in the testing phase and enable faster delivery of high-quality software.

Looking Ahead

Our journey to improve testability taught us valuable lessons about the role of software design in enabling smooth DevOps transformations. By focusing on testability, we addressed bottlenecks, improved developer skills, and laid the foundation for future success. While immediate results may be slow, the long-term benefits in terms of quality, efficiency, and team growth make this investment worthwhile.

As we continue our DevOps journey, measuring success becomes the next key challenge. In Chapter 4, we’ll explore how we established KPIs and Dashboards to track our progress and identify further areas for improvement.

Stay tuned as we dive into how data-driven decision-making helped us refine our DevOps processes and optimize performance!