KMS Technology Survey: Continuous Testing Driving Faster Release Cycles, But Issues Remain

ATLANTA, Nov. 29, 2018 — KMS Technology has completed a survey to better understand the challenges and achievements of companies that adopted the process of continuous testing as part of their software delivery. The survey was completed by 135 software developers and testers. A key takeaway from the survey is that companies are achieving many of the goals they set for continuous testing, such as faster release cycles and more rapid feedback on observed defects; however, the process of implementing and performing continuous testing still has some challenges.

Software release cycles are shorter than ever, and yet, too often, quality is sacrificed for speed. Continuous testing, with its “test as soon as possible, test often” mantra, can help companies avoid the pains inherent to a “minimally test” software development mindset, such as poor customer experience. In addition, continuous testing helps achieve numerous benefits, including higher confidence in the delivered code quality and increased application reliability. Introducing a culture of continuous testing to an organization’s software delivery model will encourage teams to focus on testing from the inception of their development lifecycle.

The KMS Technology survey confirms the benefits of continuous testing. According to the results, 92% of respondents agree or strongly agree that continuous testing has enabled faster feedback on observed defects within their organizations, and 58% said faster feedback on observed defects was the primary reason for adopting continuous testing. Meanwhile, 84% of respondents agree or strongly agree that continuous testing has helped them achieve shorter release cycles; 84% also agree or strongly agree that continuous testing has helped them achieve better test coverage with the same test team size. An additional 80% of respondents agree or strongly agree that continuous testing has led to fewer reported software defects.

Nevertheless, continuous testing challenges remain. For example, 61% of respondents agree or strongly agree that it was difficult to recruit automation testers with the necessary skill sets, and 56% agree or strongly agree that manual testers were hesitant to acquire the coding skills needed to perform new duties. Even after continuous testing was adopted for teams, some challenges were encountered: 84% of respondents agree or strongly agree that the maintenance effort of automated tests increased, and 67% agree or strongly agree that the total execution time of tests expanded.

The KMS Technology survey also looked at what companies recommend as part of their retrospective, what could have improved the adoption of continuous testing. Ninety-one percent of respondents agree or strongly agree that better training of their teams would have helped, while 88% agree or strongly agree that a directive from executive leadership would have accelerated the adoption or increased the likelihood of success.

A few other noteworthy data points from the KMS Technology survey:

• The majority of respondents (47%) said that continuous testing was adopted by a few teams at a time. Ninety percent said the process had taken six months or less, and 50% could do it in less than three months.
• Fifty-six percent said that integration testing was the primary area of test coverage; 48% said security testing, while 44% said their primary focus was UI testing.
• Performance/load tests (57%) and security tests (53%) were most likely to be always automated, followed by unit and component tests (47%), and integration tests (46%).

“The biggest surprise from the survey was the difficulty companies are facing to find and train testers in continuous testing,” says Mush Honda, Vice President of Testing at KMS Technology. “We have learned from experience how important continuous testing is to meet the demands of delivering applications faster with high confidence in quality. As an industry, however, we will not achieve both speed and quality in software production until we can commit to training on the continuous delivery mindset for a new generation of testers. Once this happens, all companies will be able to realize the benefits of rapid release cycles and high software quality.”

Resources

A blog post about “How to Build a Culture of Continuous Testing in Your Organization”