Click to learn more about author Drew Horn.
This is part one of a three-part series on introducing and building test automation into your application development and deployment pipeline.
In order to move more quickly and efficiently – while simultaneously saving on costs – many companies are starting to embrace test automation. In fact, in 2017 KPMG found an 85 percent increase in test automation in a two-year period across all industry domains.
When operating in an Agile development environment, automation is a tool that all companies should prioritize. That said, although one of the top priorities of test automation is to move faster, maturing test automation within a deployment pipeline should be a progressive, iterative process. In other words, if you try to automate too much, or automate the wrong things, you can end up slowing down – and even hurting – development and testing efforts.
Your success will depend on how you set up your automation practice, so the more attention you pay to the early stages and phases, the more benefits you will realize in the long run.
On the road to mature test automation within your deployment pipeline, you will move through three separate stages: beginner, intermediate, and expert. We will explore the beginner stage in part one of this series.
Making the Leap
At its core, automation is about doing things better, faster, and cheaper. It improves coverage by augmenting the number of tests you can run and advances test quality through repeatability. However, you need to lay the foundation before you can build the house, so it’s best to start small with your test automation efforts.
The beginner stage of test automation starts with assessing the maturity of your testing organization and setting goals of where you want to go. You should also develop a (or integrate an existing) framework on which to run your unit, integration, and functional tests. You can then determine your testing maturity by self-evaluating across five key criteria:
- Team. Who is on the team? Where are the gaps in expertise? Are QA and Dev teams working in silos?
- Technology. What’s on the deployment pipeline? How are automated tests triaged? How is automation integrated with a VSC/TCM/BTS?
- Process. When are tests written? How are bugs addressed and tests updated? How fast are sprint cycles? How does feedback impact test strategy?
- Reporting. What is the test coverage? What are the everyday devices used? How are test results analyzed? How is a “go/no-go” decision made? What is the automation ROI?
- Pain. What are the existing pain points? What bugs have been missed?
Once you’ve determined your maturity and developed a framework, you will want to focus on one key thing: Creating quick wins for your team. This will help to build confidence in your automation practice and get the entire team involved.
Your development team can begin by checking out unit tests to receive pass/fail feedback right away. After seeing these unit tests pass on a consistent basis, the team should then begin to develop a core set of functional smoke tests.
How often a set of smoke tests should be run initially will depend on your team’s experience and availability. A good rule is to start with a nightly smoke test that you can review each morning. This allows you to focus on code quality, test reliability, and process as opposed to performance and optimization. Soon enough, you’ll want to get these automated tests up and running as part of your overarching build efforts so they are automatically triggered upon a successful build.
Steady iteration, where success is built on success, will help you gain valuable experience with your automation. Once you see your early unit and smoke tests passing consistently, you can begin adding more smoke tests to your automation suite. As a rule of thumb, you’ll want to keep the runtime of your tests to 15 minutes or less. However, as you advance your automation practice, you will find additional ways to run more smoke tests in a 15-minute period, such as test parallelization. We’ll cover these strategies in the next article.
Although it’s not exactly the most scintillating part of test automation, setting up a documentation process is an important step in creating a successful automation practice. Everyone on the team should agree on what will be your definition of “done” for automation tasks that need to be written out during sprint planning. While developing and documenting your automation processes, you should also look to answer the following questions: When should initial tests be run? How is the team alerted when tests fail? Who is responsible for fixing failed tests? What are the procedures for triaging failures and logging bugs? Document the answers to these questions so everyone understands the protocol. Without that, you’ll run into problems in the future as you scale up the speed and breadth of your practice.
The amount of time you spend in the beginner stage of your automation journey will depend on several factors. Overall, the most difficult part of this stage is just getting the ball rolling. Try to specify your goals about eight weeks prior to your initial test automation practice. This is a good pace that will all you to keep momentum. When time constraints, lack of prior experience, and competing priorities are large factors, consider pulling in turn-key solutions to get up and running quickly. Keep in mind that your strategy can always be tweaked later. It’s much easier to make adjustments within an existing practice in place that can provide value to the team almost immediately.
In article two of this series I explore the intermediate stage of maturing test automation.