Truly Becoming Agile by Piping in Automation Testing

If you have worked in the technology industry during the last few decades, you have heard the term ‘agile’ more times than you can count.

Agile, DevOps–the industry tosses these buzzwords around like free candy. But how can companies become agile in today’s technology sector?

The answer might surprise you: Testing.

Ok ok, we have to admit…that statement is a bit misleading. Not all testing is going to kick CI/CD (Continuous Integration/Continuous Delivery) into overdrive.

But automation testing will.

Find out how KMS can help you become fully agile with our testing services!


Challenges without Automation Testing

Without automation, testing bottlenecks at the end of the SDLC (software development lifecycle)

But automation addresses many testing challenges. For example, automation can mitigate low team bandwidth, small team size, long regression cycles, limited test coverage, narrow automation focus, and more.

By spreading testing throughout the delivery cycle, companies increase the efficiency of their teams AND keep continuous delivery in motion

Better convinced by the numbers? Check out these stats from our latest ebook, 3, 2, 1…Liftoff! How Continuous Testing Launches Continuous Delivery into Orbit.

Results from survey about continuous testing and automation testing. How did Continuous Testing benefit your team? Achieve shorter release cycles (83.7% agree or strongly agree), faster feedback on observed defects (91.8%), smaller test team size (67.4%), better test coverage with the same test team size (84.4%), developers doing more testing (80.7%), no manual testing in pre-prod environments (53.3%), exploratory testing in prod (74%), fewer reported defects in prod (80%).

Want to know what else we uncovered in our survey? Check out the full Continuous Testing infographic

What Tests Should We Automate? 

A lot of teams tend to focus on automating UI (user interface) tests. It makes sense, your customers directly interact with your UI.

But, automating UI tests is not enough to enable agile software development. These tests are time-consuming, flakier, and harder to maintain. Bottom line, only automating UI tests won’t improve time to market.

That said, let’s look at both N-Tier and Microservices architectures. Check out these images.

This slide shows visuals of where manual and automation testing falls in two architecture types, N-Tier on the left, and microservices on the right. For N-Tier, there is a triangle, divided into sections. The bottom, largest section is blue and says “Automated Unit Tests.” The next 3 sections are green and from largest to smallest or bottom to top, read “Automated Component Tests, Automated Integration Tests,” and Automated API Tests. The tip of the triangle is white and reads “Automated GUI Tests.” At the very top of the triangle, there is a cloud shape and inside it is labeled, “Manuel Session Based Testing.” On the right, the Microservices image is a hexagon. It is divided into 3 sections and reads from top to bottom, “Integrated,” “Integration,” and “Implementation Detail.”

Segment size matters–and you want to distribute automation focus accordingly.

For N-Tier platforms, focus on Unit Tests, followed by component tests, integration, and so on.

If you are working in a microservices application, check out our 2 part blog series, Crash Course to Testing in Microservices.

Another way to develop your automation strategy is with Liza Crispin’s Agile Test Quadrants. In the image below, you can see where different tests fall within these quadrants.

By understanding the effects of automating different tests, you can weave automation throughout the software development lifecycle (SDLC).

Test Automation - Agile Test Quadrants graph shows where manual and automation testing types fall within the agile quadrants. The graph is divided down the middle with a vertical axis. At the top, the axis is labeled “Business Facing,” and “Technology Facing” at the bottom. A horizontal axis also runs through the middle of the graph. This axis is labeled “Supporting the team” on the left and “Critique Product on the Right.” In the bottom left corner (in the technology facing, supporting the team quadrant), Q1 is labeled “Automated” and includes Unit tests, API tests, Web Services testing, component testing. Moving clockwise, Q2 ( the business facing, supporting the team quadrant) is labeled “Automated & Manuel” and included functional testing, story tests, prototypes, simulations. Q3 (business facing, critique product) is called “Manual” and includes exploratory testing, scenerio based testing, usability testing, user acceptance testing, and alpha/beta. Q4 (technology facing, critique product) is labeled “Tools Automated” and includes performance and load testing, security testing, and *ility testing.

Setting Up Automated Testing 

Once you know which tests to automate, you can begin automation set up.

  1. Create logical test suites.
  2. Select and choose what test suites are needed.
  3. Configure automation tools with CI/CD server
  4. Trigger and run tests
  5. Report and analyze results

Tools to Automate Testing Throughout the CI/CD Pipeline

With automation strategy in hand–let’s look at the tools that keep this automation machine moving. 

An image showing where automation testing tools fall within the CI/CP pipeline. Following the pipeline from left to right, the first circle is “code,” followed by “Commit” and “Related Code.” The next large circle is the “CI Pipeline” and includes build, unit tests, integration tests. Automation testing tools include Jenkins, Katalon, Selenium, and Bamboo. The final large circle on the pipeline is the “CD Pipeline.” It includes review, staging, and production. Tools for this pipeline are Docker and Kubernetes.

By weaving automation throughout the CI/CD Pipeline, you can truly become agile. Better still, you can reap the benefits: 

  • Faster iteration
  • Wider coverage
  • More confidence
  • More bandwidth for testers to do actual testing! 

Sounds pretty sweet right? Find out how KMS can help you level up your test strategy. 


This information was originally presented as part of Agile Day Atlanta 2019. To dive a little deeper, we wanted to share some of the top questions and answers from the event. 

Agile Day Atlanta – Questions and Answers

How do I make my less technical testers use automation? 

In agile organizations, testers are going to need to learn a little bit of coding. There is no way around it. But there is a way for your testers to easily learn necessary code. By using tools like Katalon, testers can manually perform tests and use the outputted script to begin to understand the code.

While these tools are great, your testers will need to be diligent and openminded to learn automation.

But, its a win-win for your testers. You will always need manual testing, but automation can tackle the repetitive (*cough* boring)  stuff.

Think of it this way, I hate mowing the lawn. If I can find a way to automate that process so I can sit on my deck and drink a beer instead…of course, I’m going to do it.

What makes for effective reporting? 

Reporting has to be insightful. Having reports that are just pass/fail is not going to cut it.

When building your strategy, look for tools that tell you exactly where there is a bug and give you actionable results.

Should we automate all of our tests?

No, you should not automate everything in your CI/CD pipeline.

You can’t run a full regression every time a build kicks off. That would take forever…and defeats the point of agile development and faster time to market.

Instead, sit down when you are sprint planning to understand what user stories are being used. Based on those user stories, you can pick and choose tests suites that pertain to those areas.

When is the best time to automate?

Ideally, automate your tests as soon as possible. Don’t wait until something is fully functional to test.

Start with the APIs. If you test your APIs and they fail…what is the point to move on to the next phase of development? 

With automation tools like SoapUI making API testing fast and easy…it’s silly to not tackle these tests ASAP. 

If your API doesn’t work, you UI won’t either. So don’t waste your time waiting to test.

So, is your organization truly agile? If not, KMS is happy to help. Check out our testing services or lets us know how we can work together to help your organization leverage automation and become truly agile. 


This information and images were originally shared as part of Agile Day Atlanta 2019.

Schedule a Free Consultation

Quickly ramp-up teams and accelerate the delivery of your new software product.