5 Simple Strategies to Reduce Maintenance for Automation Tests

Automated Testing for Continuous Delivery

Continuous delivery can be incredibly lucrative for companies looking to release new features quickly and stay in front of their customers’ expectations. However, without an effective testing strategy to keep up with rapid development, releases could be buggy or delayed.

To combat this, many companies are leveraging automation as part of a continuous testing strategy to keep the CI/CD pipeline running smoothly.

Need help automating your QA strategy?


What Maintenance is Involved in Automating Tests?

While continuous testing is a great solution, there is a certain amount of maintenance involved in automating your tests. In fact, of the 135 industry professionals we surveyed, over 80% agreed that maintenance efforts for automated testing are increasing.

As your company scales to meet market demand, or as features are added and your platform evolves, your automated test suite will need attention to address any problems that arise and ensure it continues to run smoothly.


5 Strategies to Reduce Maintenance in Automation Tests

While maintenance is a necessary part of automated testing, we have come up with 5 simple strategies you can put in place to reduce both the workload and the headache of testing maintenance.


1. Empower your UI/UX developers to keep testing in mind

Work with your UI/UX (user interface/user experience) engineers to develop a static naming convention for objects in the user interface.

UI regularly changes to give users a fresh or more user-friendly experience. Often times, UI/UX engineers do not have testing at the forefront of their mind when writing code. By working with them to identify a naming convention, a static quality assurance (QA) id can be added to each object’s code.

As other aspects of UI/UX design and code change, your tests scripts will continue to recognize objects based on this stable QA id, minimizing the chance that changes in your design cause problems for your test suite.


2. Be strategic when creating your automation design: keep things simple.

While this is sometimes easier said than done, use simple test scripts as often as you can. We recommend using low-level tests that are easy to maintain and execute quickly,  such as unit tests or automated API tests. By keeping your scripts as simple as possible, your maintenance efforts will be low.

For long or complicated scenarios, create reusable automated test cases and try to limit overcomplicating your test suite with complex test cases.

Complicated test suites create more opportunity for breakdown, so keeping things as simple as you can will reduce maintenance in the long run.


3. Develop a smart strategy for test execution

Ever hear the phrase “work smarter not harder”? Well, that applies to testing. You do not need to run a full test cycle every time you deploy new code.

We recommend implementing nightly automated testing that cover all test cases. These are not the mission-critical tests that determine whether a release is ready for deployment but rather regular maintenance tests.

Based on the results of this automated testing cycle, you can appropriately delegate efforts to either address or further explore issues identified in the nightly tests or perform maintenance on the automated test suite.


4. Establish a process to keep automation up to date

When designing and implementing your testing process, you want to consider the longevity of your process. Have you put a system in place to regularly check and update your test automation scripts? If not, do it!

If your team is delivering features and continually improving your application (and we sure hope they are!) then the reality is, your tests scripts will need to follow suit. Having a strategy in place to know when and how to update your tests will prevent things from slipping through the cracks and save work in the long run.


5. Identify and reduce flaky tests

So, what is a flaky test? These are tests that produce inconsistent results, even when run under the same parameters. Particularly prevalent in UI testing, flakiness is an error in the test setup, not in the development code.

To determine if a test is flaky or has simply failed, automatically re-run failed tests. If a test continuously fails, it is likely an issue with the development code and is not a flaky test.

If the failed test passes when you run it again or produces inconsistent results, then you have found a flaky test.

Flaky tests are an unfortunate reality of continuous testing, but identifying and responding to them quickly is an important step in maintaining your test suite.


Don’t let maintenance deter you from realizing the advantages of automation. Trust us, the benefits far outweigh the maintenance efforts…especially if you minimize those efforts with our simple tips!


Interested in adopting continuous testing? Find out how to implement a culture of continuous testing at your organization!

Read the blog on how to build a culture of continuous testing in your organization

Introduction to CI/CD Performance Testing: Common Problems and Key Solutions

Continuous integration/continuous delivery (CI/CD) is being rapidly adopted by many organizations to improve their time to market. According to the World Quality Report 2017-2018, 88 percent of companies are using or experimenting with DevOps principles. Since the initial stage for DevOps is the CI/CD pipeline, it is critical to have testing woven into every build cycle. The concept of shifting left while integrating performance testing in the continuous delivery model is an important goal for testers.


Here, we’ll identify some typical challenges development and QA teams encounter when setting up performance testing within a CI/CD pipeline, as well as the solutions that can be used to resolve these issues.

Today’s Testing Environment

“88% of companies are using or experimenting with DevOps principles.”

Applications are becoming increasingly complex, with a lot of dependencies and integrations with third-party platforms. This makes building a production-like environment very involved – and expensive.

A recommended goal for testing in CI/CD is to verify the performance trends of the application following changes to a simpler configuration. This allows testers to evaluate different versions compared to a baseline result on an identical configuration. This approach also means testers can use simplified containers as opposed to building a full, realistic environment.

Time Required for Testing

Executing performance testing is a time-consuming task due to the high volume of data prepared and long execution for resource decay. With this in mind, data should be prepared in advance and reused wherever possible.

Shorter duration test cases for new functional changes can also be helpful in reducing testing time, and should be executed in CI/CD pipelines first. Longer duration end-to-end performance tests and load/stress tests should be done in a concurrent, yet separate cycle of testing.

In addition, testing teams should also look to adopt practices including:

  • Early involvement of testers: Performance/load testers should be involved in the project as early as possible, allowing them to work with other team members to apply inputs of their testing environments and strategy. Those tasks should be defined according to stories planned within each sprint.
  • Expanded acceptance criteria for user stories: There should also be performance acceptance criteria for each story, so testers can create and execute performance tests based on those specifications.
  • Short feedback loops: The goal of CI/CD pipelines is to encourage short feedback loops, so this approach should be taken with performance tests for new components as well. Small load testing to measure performance or to monitor a trend compared with historical runs is necessary. In addition, it can be helpful to use parallel test suites for even faster feedback.
  • Release-ready testing: End-to-end scenarios with high volumes of data and transaction loads are still needed during the production phase to ensure a quality product. These types of time-consuming tests should be executed nightly or after sprints are completed, independent of CI/CD pipelines.
Man in suit using touch screen featuring software development abstract image. The current application testing environment is complex, but using CI/CD can help streamline testing processes.

Recommended Tools for Performance Testing

It’s important that testing teams have the right tools in place to support integrated performance testing. Some tools we’ve found most successful include:

  • Apache Jmeter
  • Smartmeter.io
  • Webload

Check out this resource from Software Testing Help for more reviews of available tools.

In addition, there are also certain plugins that can be used with Jenkins to enable performance result analysis, such as:

  • Plot
  • Performance Publisher
  • Blazemeter
  • Taurus

Stackify also recommends these DevOps tools.

In the current competitive world of complex, integrated applications and websites, performance must be evaluated and monitored. With the basic fundamental approach to testing earlier and testing often, it’s possible to avoid the pitfalls many users face when adopting CI/CD for evaluation application performance.