The Secret to Agile Delivery During Unprecedented Digital Demand

This blog is an updated view on our previous post: Truly Becoming Agile by Piping In Automation Testing. In this update, we explore automation and agile delivery through the lens of this year’s increased digital demands, and with updated data from Katalon’s Test Automation Landscape Report. Download the report here.

2020 Marks a New Era in Digital Solutions

This year has brought an unprecedented change in global business. While digital solutions have been on the rise for many years, COVID-19 unexpectedly catapulted digital adoption forward, leaving the market scrambling to navigate this important inflection point.  

According to a report by McKinsey,

“Recent data shows that we have vaulted five years forward in consumer and business digital adoption in a matter of around eight weeks.” 

This is because consumer behaviors and preferred interaction channels have drastically shifted in response to the virus.

However, even as a vaccine becomes available, the need for digital channels is not expected to change significantly. In fact, 75% of those using digital channels for the first time, intend to continue doing so long term. 

Scale your development efforts now with KMS engineers 

SEND US A NOTE
 

The Secret to Agile Delivery

To meet this growing demand and remain competitive, pressure is on for rapid software development. Agile platforms and teams are more important than ever. 

But, what is the secret to agile delivery? 

The answer might surprise you. It’s Test Automation. 

 

Why is Test Automation So Important Right Now

Let’s break things down. Stakeholders and customers have high expectations and low tolerance for defects. This makes quality assurance (QA) a key focus for any company hoping to compete. 

However, testing has traditionally been a bottleneck in the Software Development Lifecycle. This leaves teams searching for a solution that balances full-throttle delivery speeds but maintains product quality and security. 

Test Automation strikes that balance and enables teams to maintain or accelerate pace without sacrificing product integrity. This is why automation is the secret weapon for truly agile teams. 

Benefits of Test Automation for Agile Delivery

But, it’s not so secret anymore. According to Katalon’s report, Test Automation Landscape in 2020, 87% of companies have started adopting automation, with 70% having leveraged automation for over a year. 

“Automation is seen as an essential part of a successful Agile and DevOps transformation…Automation is also an enabler for organizations to focus on delivering Quality at Speed, which means shortening time to market while achieving the desired quality of delivery and user satisfaction”  

Automation addresses many testing challenges such as low team bandwidth, long regression cycles, limited test coverage, narrow automation focus, and more. 

By spreading testing throughout the delivery cycle, companies increase the efficiency of their teams AND keep continuous delivery in motion.

Benefits include: 

  • Faster iteration
  • Wider coverage
  • More confidence
  • More bandwidth for testers to do actual testing! 

Tools to Automate Testing Throughout the CI/CD Pipeline

For organizations looking to adopt automation or improve automation efficiency, we recommend combining a few different tools. According to Katalon’s report, 89% of organizations are using at least 2 tools in their automation projects. 

This image shows the top 20 automation tools used for agile delivery. Top 3 tools are Selenium, Katalon Studio, and TestNG.

Why? Different tools support different parts of the Continuous Integration / Continuous Delivery (CI/CD) Pipeline, so it’s important to find the right blend of tools for your platform. 

This shows where certain tools best fit in the CICD pipeline to enable Agile delivery. Tools for continuous integration include Jenkins, Katalon, and Selenium. Tools for continuous delivery include Docker and Kubernetes.

Need help building your automation strategy? Learn more about our Automation Consulting service.

LEARN ABOUT AUTOMATION CONSULTING
 

What Tests Should We Automate? 

Now that we have looked at tool options, where does automation start? 

A lot of teams tend to focus on automating UI (user interface) tests. It makes sense, your customers directly interact with your UI.

But, automating UI tests is not enough to enable agile software development. These tests are time-consuming, flakier, and harder to maintain.

In times where speed and agility is key, this might not prove the best use of time. That said, let’s look at both N-Tier and Microservices architectures. 

This slide shows visuals of where manual and automation testing falls in two architecture types, N-Tier on the left, and microservices on the right to enable agile delivery. For N-Tier, there is a triangle, divided into sections. The bottom, largest section is blue and says “Automated Unit Tests.” The next 3 sections are green and from largest to smallest or bottom to top, read “Automated Component Tests, Automated Integration Tests,” and Automated API Tests. The tip of the triangle is white and reads “Automated GUI Tests.” At the very top of the triangle, there is a cloud shape and inside it is labeled, “Manuel Session Based Testing.” On the right, the Microservices image is a hexagon. It is divided into 3 sections and reads from top to bottom, “Integrated,” “Integration,” and “Implementation Detail.”

Segment size matters–and you want to distribute automation focus accordingly.

For N-Tier platforms, focus on Unit Tests, followed by component tests, integration, and so on.

Another way to develop your automation strategy is with Liza Crispin’s Agile Test Quadrants. In the image below, you can see where different tests fall within these quadrants.

Agile Test Quadrants graph shows where manual and automation testing types fall within the agile quadrants to enable agile delivery. The graph is divided down the middle with a vertical axis. At the top, the axis is labeled “Business Facing,” and “Technology Facing” at the bottom. A horizontal axis also runs through the middle of the graph. This axis is labeled “Supporting the team” on the left and “Critique Product on the Right.” In the bottom left corner (in the technology facing, supporting the team quadrant), Q1 is labeled “Automated” and includes Unit tests, API tests, Web Services testing, component testing. Moving clockwise, Q2 ( the business facing, supporting the team quadrant) is labeled “Automated & Manuel” and included functional testing, story tests, prototypes, simulations. Q3 (business facing, critique product) is called “Manual” and includes exploratory testing, scenerio based testing, usability testing, user acceptance testing, and alpha/beta. Q4 (technology facing, critique product) is labeled “Tools Automated” and includes performance and load testing, security testing, and *ility testing.

By understanding the effects of automating different tests, you can weave automation throughout the software development lifecycle (SDLC).

Setting Up Automated Testing 

Once you know which tests to automate, you can begin automation set up: 

  1. Create logical test suites.
  2. Select and choose what tests suites are needed.
  3. Configure automation tools with CI/CD server
  4. Trigger and run tests
  5. Report and analyze results

Or, if you want to make things even easier, accelerate automation with KMS. 

GET IN TOUCH WITH KMS
 

 

Truly Becoming Agile by Piping in Automation Testing

If you have worked in the technology industry during the last few decades, you have heard the term ‘agile’ more times than you can count.

Agile, DevOps–the industry tosses these buzzwords around like free candy. But how can companies become agile in today’s technology sector?

The answer might surprise you: Testing.

Ok ok, we have to admit…that statement is a bit misleading. Not all testing is going to kick CI/CD (Continuous Integration/Continuous Delivery) into overdrive.

But automation testing will.

Find out how KMS can help you become fully agile with our testing services!

SEND US A NOTE

Challenges without Automation Testing

Without automation, testing bottlenecks at the end of the SDLC (software development lifecycle)

But automation addresses many testing challenges. For example, automation can mitigate low team bandwidth, small team size, long regression cycles, limited test coverage, narrow automation focus, and more.

By spreading testing throughout the delivery cycle, companies increase the efficiency of their teams AND keep continuous delivery in motion

Better convinced by the numbers? Check out these stats from our latest ebook, 3, 2, 1…Liftoff! How Continuous Testing Launches Continuous Delivery into Orbit.

Results from survey about continuous testing and automation testing. How did Continuous Testing benefit your team? Achieve shorter release cycles (83.7% agree or strongly agree), faster feedback on observed defects (91.8%), smaller test team size (67.4%), better test coverage with the same test team size (84.4%), developers doing more testing (80.7%), no manual testing in pre-prod environments (53.3%), exploratory testing in prod (74%), fewer reported defects in prod (80%).

Want to know what else we uncovered in our survey? Check out the full Continuous Testing infographic

What Tests Should We Automate? 

A lot of teams tend to focus on automating UI (user interface) tests. It makes sense, your customers directly interact with your UI.

But, automating UI tests is not enough to enable agile software development. These tests are time-consuming, flakier, and harder to maintain. Bottom line, only automating UI tests won’t improve time to market.

That said, let’s look at both N-Tier and Microservices architectures. Check out these images.

This slide shows visuals of where manual and automation testing falls in two architecture types, N-Tier on the left, and microservices on the right. For N-Tier, there is a triangle, divided into sections. The bottom, largest section is blue and says “Automated Unit Tests.” The next 3 sections are green and from largest to smallest or bottom to top, read “Automated Component Tests, Automated Integration Tests,” and Automated API Tests. The tip of the triangle is white and reads “Automated GUI Tests.” At the very top of the triangle, there is a cloud shape and inside it is labeled, “Manuel Session Based Testing.” On the right, the Microservices image is a hexagon. It is divided into 3 sections and reads from top to bottom, “Integrated,” “Integration,” and “Implementation Detail.”

Segment size matters–and you want to distribute automation focus accordingly.

For N-Tier platforms, focus on Unit Tests, followed by component tests, integration, and so on.

If you are working in a microservices application, check out our 2 part blog series, Crash Course to Testing in Microservices.

Another way to develop your automation strategy is with Liza Crispin’s Agile Test Quadrants. In the image below, you can see where different tests fall within these quadrants.

By understanding the effects of automating different tests, you can weave automation throughout the software development lifecycle (SDLC).

Test Automation - Agile Test Quadrants graph shows where manual and automation testing types fall within the agile quadrants. The graph is divided down the middle with a vertical axis. At the top, the axis is labeled “Business Facing,” and “Technology Facing” at the bottom. A horizontal axis also runs through the middle of the graph. This axis is labeled “Supporting the team” on the left and “Critique Product on the Right.” In the bottom left corner (in the technology facing, supporting the team quadrant), Q1 is labeled “Automated” and includes Unit tests, API tests, Web Services testing, component testing. Moving clockwise, Q2 ( the business facing, supporting the team quadrant) is labeled “Automated & Manuel” and included functional testing, story tests, prototypes, simulations. Q3 (business facing, critique product) is called “Manual” and includes exploratory testing, scenerio based testing, usability testing, user acceptance testing, and alpha/beta. Q4 (technology facing, critique product) is labeled “Tools Automated” and includes performance and load testing, security testing, and *ility testing.

Setting Up Automated Testing 

Once you know which tests to automate, you can begin automation set up.

  1. Create logical test suites.
  2. Select and choose what test suites are needed.
  3. Configure automation tools with CI/CD server
  4. Trigger and run tests
  5. Report and analyze results

Tools to Automate Testing Throughout the CI/CD Pipeline

With automation strategy in hand–let’s look at the tools that keep this automation machine moving. 

An image showing where automation testing tools fall within the CI/CP pipeline. Following the pipeline from left to right, the first circle is “code,” followed by “Commit” and “Related Code.” The next large circle is the “CI Pipeline” and includes build, unit tests, integration tests. Automation testing tools include Jenkins, Katalon, Selenium, and Bamboo. The final large circle on the pipeline is the “CD Pipeline.” It includes review, staging, and production. Tools for this pipeline are Docker and Kubernetes.

By weaving automation throughout the CI/CD Pipeline, you can truly become agile. Better still, you can reap the benefits: 

  • Faster iteration
  • Wider coverage
  • More confidence
  • More bandwidth for testers to do actual testing! 

Sounds pretty sweet right? Find out how KMS can help you level up your test strategy. 

GET IN TOUCH WITH KMS

This information was originally presented as part of Agile Day Atlanta 2019. To dive a little deeper, we wanted to share some of the top questions and answers from the event. 

Agile Day Atlanta – Questions and Answers

How do I make my less technical testers use automation? 

In agile organizations, testers are going to need to learn a little bit of coding. There is no way around it. But there is a way for your testers to easily learn necessary code. By using tools like Katalon, testers can manually perform tests and use the outputted script to begin to understand the code.

While these tools are great, your testers will need to be diligent and openminded to learn automation.

But, its a win-win for your testers. You will always need manual testing, but automation can tackle the repetitive (*cough* boring)  stuff.

Think of it this way, I hate mowing the lawn. If I can find a way to automate that process so I can sit on my deck and drink a beer instead…of course, I’m going to do it.

What makes for effective reporting? 

Reporting has to be insightful. Having reports that are just pass/fail is not going to cut it.

When building your strategy, look for tools that tell you exactly where there is a bug and give you actionable results.

Should we automate all of our tests?

No, you should not automate everything in your CI/CD pipeline.

You can’t run a full regression every time a build kicks off. That would take forever…and defeats the point of agile development and faster time to market.

Instead, sit down when you are sprint planning to understand what user stories are being used. Based on those user stories, you can pick and choose tests suites that pertain to those areas.

When is the best time to automate?

Ideally, automate your tests as soon as possible. Don’t wait until something is fully functional to test.

Start with the APIs. If you test your APIs and they fail…what is the point to move on to the next phase of development? 

With automation tools like SoapUI making API testing fast and easy…it’s silly to not tackle these tests ASAP. 

If your API doesn’t work, you UI won’t either. So don’t waste your time waiting to test.

So, is your organization truly agile? If not, KMS is happy to help. Check out our testing services or lets us know how we can work together to help your organization leverage automation and become truly agile. 

GET IN TOUCH WITH KMS

This information and images were originally shared as part of Agile Day Atlanta 2019.

Which APM Tool is the Heavyweight Champion?

Let’s talk about application performance. Any consumer who has experienced the infamous spinning beachball knows that slow or glitchy performance can mean K.O. for a company’s bottom line and reputation. 

To minimize the impact of this blow, smart companies are using Application Performance Management (APM) to monitor performance in real-time. This live performance feedback can help teams quickly identify and address system bottlenecks and poor performance. 

Looking for a partner to help you get started with APM? KMS is here to help! 

 

GET IN TOUCH WITH KMS

 

Choosing the right APM tool enables companies to monitor the pulse of the application performance and effectively roll with the punches.

However, there are a number of tools on the market and knowing which one to choose can be overwhelming. Of course, we have you covered and have pitted some of the top APM tools against one another to help you choose the right one. 

With that said…

*Ding, Ding* 5 of the top APM tools step into the ring. Which will come out on top? 

 

How to Choose the Right APM Tool

While each of these tools throws a good punch, in reality–choosing the right tool really comes down to your application and organization’s specific needs. 

While this does not make for the most entertaining boxing match, it does make it easier for your organization to look critically and find a tool that works for you. 

Of course, there are a few important questions you should consider when choosing an APM tool: 

  • What languages does the program support?
  • How well does it integrate with your other tools?
  • What application analytics does the tool offer?
  • Do the monitoring features meet your needs and does the tool provide code-level diagnostics?
  • How much does it cost?

 

Comparing the Top APM Tools

With those 5 questions in mind how do the contenders compare? 

APM Tool: New Relic

Supported Languages: Java, .NET, Node.js, PHP, Python, Ruby, Go

Integrations: Over 200 plug-in integrations 

Application Analytics: New Relic analytics uses applied intelligence to give insight into the source and recommend solutions for application errors. 

Monitoring Features / Code-level diagnostics: APM, Infrastructure, Browser, Synthetics, Mobile. Gives you code-level visibility

Cost: Offers 30-day free trial. 

$9.37 to $200 per host per month (billed annually). Pricing varies depending on the number of hours each host/instance is running, the size of the host/instance, and the total number of hosts/instances.

Offers a price calculator. 

APM Tool: AppDynamics(Typically supports big enterprises)

Supported Languages: Java, .NET, PHP, Node.js, C++, Python, Go

Integrations: 190+ supported technologies

Application Analytics: AppDynamics offers real-time analytics and data visualization. Provides  transaction, log, and browser-mobile analytics.  

Monitoring Features / Code-level diagnostics: APM, End User Monitoring, Business Performance Monitoring, Infrastructure Visibility. Offers immediate and automated code-level diagnostics. 

Cost: Contact Vendor for Pricing.

APM Pro: contact vendor for pricing (from internet: $300 per unit / mo, $3,300.00/year )APM Advanced: contact vendor for pricing.

APM Peak: contact vendor for pricing.

APM Tool: Stackify Retrace

Supported Languages: Java, .NET, PHP, Node.js, Ruby, Python

Integrations: Integrates with AppVeyor, Atlassian Bamboo, AWS CodePipeline, CircleCI, Jenkins, OCtopus Deploy, TeamCity, Vidual Studio Team Services

Application Analytics: Automatically monitors and tracks key metrics for Java, PHP, and .NET applications. Also allows for customer application metrics. 

Monitoring Features / Code-level diagnostics: APM, Code Profiling, Error Tracking, Centralized Logging, App & Server Metrics. Get deep, code-level insights.

Cost: $50 per month/server 

Retrace APM for small servers – $25/month per server.

Retrace monitoring without code profiling (APM) – $15/month per server.

Retrace APM for pre-production servers – $10/month per server.

14-day free trial 

APM Tool: Scouter (Open Source Tool) 

Supported Languages: Java

Integrations: Plugins include: email alerts, InfluxDB, and transfer alerts to Telegram, Slack, Line, and DinTalk, etc. 

Application Analytics: Provides metrics about users (Active user, Recently used user, Today visitor), services (Active service, TPS, Response time, Application profiles), and resources (CPU, Memory, Network and Heap usage, Connection pools etc.)

Monitoring Features / Code-level diagnostics: Scouter has the functionality to monitor targets like Java agent for Web applications, Zipkin, Redis, NginX, MongoDB, RabbitMQ and Elasticsearch. 

The tool monitors the service transactions as an individual.(XLOG)

Cost: Free

APM Tool: Amazon CloudWatch (Cloud Provider Tool) 

Supported Languages: Java, Go, PowerShell, Node.js, C#, Python, and Ruby

Integrations: CloudWatch is natively integrated with more than 70 AWS services such as Amazon EC2, Amazon DynamoDB, Amazon S3, Amazon ECS, AWS Lambda, Amazon API Gateway, etc.

Application Analytics: Built-in metrics: collect default metrics from more than 70 AWS services. 

Granular data and extended retention: allowing to monitor trends and seasonality with 15 months of metric data (storage and retention). This data allows the user to perform historical analysis to fine-tune resource utilization. 

Monitoring Features / Code-level diagnostics:Unified operational view with dashboards: allow user to create re-usable graphs and visualize cloud resources and applications in a unified view. They can graph metrics and logs data side by side in a single dashboard to quickly get the context and go from diagnosing the problem to understanding the root cause. 

Cost: Offers a free and paid tier. Paid includes a calculator to estimate price. 

So, which APM tool came out on top for your organization? 

Curious to know more about how APM fits into a comprehensive performance engineering strategy? Check out our last blog, Continuous Performance Testing and Application Performance Management, A Performance Engineering Love Story.

Need a partner to handle APM in your organization? 

 

SEND US A NOTE

 

Workshop Recap: Katalon Studio for Web and API Continuous Testing

On March 28th, the KMS team was joined by other Atlanta-based testers for the Ministry of Testing Atlanta Meetup.

Ministry of Testing (MOT) is a global software testing community consisting of local groups of professional testers. The organization offers online content, conferences, and networking events for its members. The Atlanta community of MOT brings professionals together to learn, ask questions, share resources, and have a good time!

GET IN TOUCH WITH KMS
 

MOT Workshop: Using Katalon Studio for Web and API Continuous Testing

KMS hosted the most recent MOT meetup, a workshop focused on using Katalon Studio for Web and API Continuous Testing.

The evening opened the way any good event should…with pasta and wings. After grabbing some grub and chatting with other testers, KMS Director of Testing Services, Nilesh Patel began the workshop

It was not long into the presentation that the conversation got moving. The group covered everything from how to not sacrifice quality for speed, understanding non-tester’s perspectives on shift-left testing, and how testers navigate unrealistic business goals.

But, as fun as the free-form conversation was, it was time to dive into Katalon.

 

Katalon Overview

For any readers unfamiliar with Katalon Studio, it is a free test automation tool supporting web, API, and mobile automation.

Patel demoed the program. He highlighted various features including different ways to build test cases, the outcomes of running parallel tests, the Katalon user forum, analytics features, different plugin options, and more.  

 

Q&A

Some attendees already use Katalon at their companies, while others were unfamiliar with the program. Naturally, the group posed a lot of questions–and answers were provided by both Patel and other attendees who had experience working with Katalon.

 

Which programs integrate well with Katalon?

Katalon is built on top of Selenium and Appium and integrates with multiple programs such as Bamboo, Jenkins, JIRA, GIT, etc.

Patel then went on to show the group Katalon’s newly launched plug-in store. The store already offers both free and paid plug-ins that support Slack integration, custom keywords, and more. Katalon users are encouraged to upload customizations they have built as plug-ins. Other members of the Katalon community can then take advantage of these plug-ins to enhance their experience using the platform.

 

How is the user experience?

For this question another attendee chimed in, explaining how his manual testing team was initially intimidated by automation. However, Katalon is designed as a tool for all levels and intended to be user-friendly. That said, he went on to explain that within about a week of using Katalon, his team had easily picked it up and have been loving the new skillset.

Another user shared a slightly different experience. His team was comprised of test automation engineers who were developers by trade. They too were hesitant to move away from their programs and adopt Katalon. However once again the transition was successful and his team found that with a few exceptions, they could do most of the same things in Katalon that they could do in IntelliJ or Eclipse.

 

How useful are Katalon’s support programs and product documentation?

Patel began answering this question, explaining that while Katalon is free, they do offer paid support packages.

However, there is a Katalon support forum that is free for everyone to use. As Patel showed the group the forum, other attendees weighed in on problems they have had in the past and various solutions provided on the forum.

As for tutorials and product documentation, Katalon offers a wiki-style platform which includes tutorials made in-house and those uploaded by independent users.

After the demo, the event ended with the group breaking off into smaller conversations about Katalon, testing challenges, and more. Overall, the meetup was a great success and many asked for a second demo to dive deeper into the program…stay tuned!

In the meantime, check out our post on selecting automation candidates for Katalon.

Get Involved with Ministry of Testing

Missed out on all the fun and interested in joining us next time? Check out the Ministry of Testing Atlanta Meetup Group. We will see you at the next event!  

Looking for a software testing provider to help with your QA strategy?

GET IN TOUCH WITH KMS
 

Continuous Performance Testing and Application Performance Management: A Performance Engineering Love Story

In the spirit of Valentine’s Day, we would like to tell you an unlikely love story.

Now we know what you’re thinking…what? This is a technology blog! I come here for cutting edge tech info, not sappy love stories. We hear you, but bear with us…we promise this info is too good not to share.

GET IN TOUCH WITH KMS
 

At first glance, this pair doesn’t seem to have much in common. But we all know opposites attract and, ultimately, it was their differences that make them a perfect complement to one another.

Ready for that technology spin? Of course, we are not talking about a famous celeb duo or dramatic romance lost to history. No, we are talking about the relationship between Continuous Performance Testing (CPT) and Application Performance Management (APM).

Before we get ahead of ourselves and tell you why they work so well together as part of a holistic performance engineering strategy, let’s explain CPT and APM independently.

 

So, what is Continuous Performance Testing (CPT)?

Typically, the continuous delivery cycle has largely focused on automated functional tests. These tests are set up to continuously run and support your release schedule with the goal of finding issues as early as possible. The earlier in the Software Development Lifecycle (SDLC) that a coding issue is found, the faster it can be resolved and released to production. Sounds pretty good, right?

But something is missing. You want to ensure your code is solid every step of the way but… can your application hold up during heavy usage? Does adding new features slow performance?

See where we’re going with this? What has commonly been ignored in the Continuous Delivery Model is performance testing. Oftentimes this step falls at the end of the SDLC, but at KMS we advocate for a Continuous Performance Testing model. Why? Because with the user demand for high quality and quick feature availability, teams can’t wait until the end of the cycle to test application performance. The shift-left mentality of CPT allows you to identify and resolve performance issues early in your development cycle–meaning less rework and major cost savings.  

Luckily, with the rise of DevOps and use of tools such as Jenkins, Bamboo (and, of course, with the help of the experts at KMS *wink*), and the availability of powerful tools such as Neoload, Locust.io, and JMeter, it is now easier than ever to pipe performance tests into your CI/CD pipeline.

Ok we get it, CPT is a total catch but what’s the deal with APM?

 

What is Application Performance Management (APM)?

While CPT is great for measuring performance during the development cycle, it only mimics traffic and does not monitor and evaluate live user interactions.

That’s where APM comes in. Once a product is launched and in use in the wild, APM uses a variety of tools to measure and monitor application performance in real-time.

Typical performance testing is ‘synthetic’, using virtual users to mimic loads and stress on application performance. On the other hand, APM continuously monitors your application and assesses performance in real-time. This live feedback loop offers insights on performance issues not available when only using standard performance tests. Identifying bottlenecks in your system as they arise allows you to respond quickly and ensures your application runs as smoothly as possible for your end users.

Starting to see why we love APM too?

APM can be leveraged to support a shift-right approach that allows you to continuously evaluate and monitor application performance in production.

Keep in mind that APM tools primarily focus on troubleshooting issues just-in-time, as they arise. While this is typically how APM tools are used, we challenge you not to leverage it as a way to proactively evolve scenarios used in performance testing. Use highly skilled resources and train your team to not only troubleshoot but to recommend solutions that help prevent future performance issues.

When it comes to APM tools, there are plenty to choose from. To help you decide, our usual go-to’s are AppDynamics, New Relic, Stackify (licensed) as well as Pinpoint, Scouter and SkyWalking (open-sourced) .

Alright. We’ve built up the backstory, but now it’s time for this romcom to get started. *Cue the romantic dinner, candlelight, and cheesy love song*

 

Why do CPT and APM work so well together?

CPT and APM are strong when used independently but, when used together, they comprise a dynamic Performance Engineering Strategy. Talk about a power couple!

CPT and APM work seamlessly together. CPT helps avoid performance issues from leaking into production, while APM detects issues after release. Uniting these methods gives you performance feedback during each stage of your SDLC, including production. This type of comprehensive feedback allows you to continually improve application speed and robustness without rebuilding your platform.

A great strategy to effectively leverage both methods is to involve performance test engineers in every phase of the process. This synergy breaks down organizational silos and keeps performance at the forefront of product engineering and maintenance.

Ultimately, this is a win-win for your customers and your team. Your customers are presented with a better performing product and your team has less rework in their bandwidth. Your attrition rate is going to skyrocket with this well-matched duo.

Now you know why we’re excited about this love story and we hope you consider implementing both APM and CPT in your performance engineering strategy. Because let’s be honest, a strategy that includes both is destined for a happily ever after.

GET IN TOUCH WITH KMS

How to Build a Culture of Continuous Testing in Your Organization

The trends in software development are showing that more and more companies are adopting CI/CD methodologies to deliver their software applications.  We all know that the market demands quicker releases. The days of waiting for months for new releases are gone. Software is now being released at record speeds!  Adopting CI/CD does just that. It helps get your application out the door to the market as often as possible. However, one key aspect that seems to be overlooked is Continuous Testing. It’s great that CI/CD is getting software out quicker but quality should not be sacrificed.  To solve that you have to test early, test often! Adding a culture of Continuous Testing to your model will provide the following benefits because now you’re focused on testing from the beginning of your SDLC

  • Faster Release Cycles
  • Better Code Quality
  • Better Test Coverage
  • Better Reliability

 

Let KMS help you leverage test automation and adopt a culture of continuous testing at your organization.

SEND US A NOTE
 

Now that we know to truly have a CI/CD methodology you need continuous testing as well. But just like adopting CI/CD, continuous testing requires an organizational culture shift.  So how do you build that culture? It’s a different mindset that relies heavily on automation: Automated Unit Tests, Automated Functional and Non-Functional Tests, Automated Regression Tests, and Automated Deployments.  Basically, anything that can be automated should be automated! That is the key principle of Continuous Testing, test from the beginning and automate as much as possible to ensure faster release cycles.

Test from the beginning and automate as much as possible to ensure faster release cycles.

With that being said, the mindshift has to change from just testers being responsible for quality to the entire team. The following two culture changes will help transform your organization into continuous testing:

Shared Responsibility:

With continuous testing, there are no silos and quality is built in from development to deployment. The whole team has to take on testing initiatives. Developers are responsible for automating their unit tests and integrating them with CI/CD pipeline. Testers of the team are focusing on functional and non-functional testing and having those integrated to the pipeline as well. From every step of the release cycle, quality is built in through the pipeline. This continues with deployments into productions, as operations can utilize all of the tests that have been automated. They can also focus on production monitoring tests, that use the automated test that developers and testers have created. As you can see, every part of the team is doing some form or fashion of testing, AND it’s occurring from the beginning and continues throughout the pipeline.

Collaboration:

Collaboration is key to continuous testing. Developers, Testers and operations all have to work together. For example, developers are going to normally create APIs for their applications first.  While they are developing the APIs, testers can start automating those APIs by working with Developers. The Operation team now knows that APIs are going to be automated and will then work with devs and testers to have those tests automatically integrated for build deployments. Operations now have a set of tests they can run at every build release and testing is again occurring at every stage of the pipeline. That is just one example of how collaboration is pivotal. Continuous Testing will not succeed if collaboration between team members is not present.

With Continuous Testing you are performing multi-layer tests such as API, front-end, back-end, performance, load and security testing. The results of these tests are helpful to all team members, which in turn results in better quality of the product. If these tests are automated, then your product will be released much faster with higher confidence in quality. In order to do that you have to make sure that your teams know testing is a shared responsibility and the collaboration between team members are rock solid.

GET IN TOUCH WITH KMS
 

We can help your organization with continuous testing - find out more

Strategies for Migrating from Selenium IDE

GET IN TOUCH WITH KMS
 

Unless you are living under a rock, you probably already know that Selenium IDE has been discontinued and no longer supported. This is largely due to trends in our industry that show more and more organizations are utilizing test automation not just for the UI tier but also middle and database tiers. Selenium IDE was primarily used in UI based testing only. As automation transforms the industry, it forces testers to look for more comprehensive tools such as Katalon Studio, that can automate all their needs. Selenium IDE just could not keep up with the rapid evolution of testing automation, hence the reason Firefox decided to discontinue the IDE. Katalon Studio is the perfect alternative in migrating from Selenium IDE as it can automate all your needs.

Discover our testing services

 

OK, now I know what you’re thinking and panic mode has started to set in:

“I have tons of scripts written with Selenium IDE!”

“ What am I supposed to do with those scripts?”

“ I don’t have the bandwidth to re-write all those scripts in the new tool”

“ How do I migrate my old scripts?”

 

Now breathe….Migration of scripts does not have to be a daunting task. This is the perfect chance to come up with a sound migration strategy that will help your organization transform to the next level. Migrating Automated scripts is nothing more than any other data migrations you may have done in past. The same strategy and practices will apply here as well. When deciding to move to a new platform such as Katalon, the following tips will help ease the process.

 

  • Do some housekeeping: Now is the perfect time to remove scripts or tests that are no longer valid due to application changes. This also provides an extra benefit of reducing execution time.
  • Leverage Tools: This is the most important part of your strategy. Migrating scripts can take a lot less time if you can just simply take your old scripts from Selenium IDE and port them over to your new tool of choice. For example, Katalon Studio has a built-in feature that allows you to import your old Selenium IDE scripts. If your automation tool of choice does not have a migration tool or import feature you should really consider a more robust tool that can handle that for you. (Hint: You can visit this forum post for a step by step guide on Katalon migration.)
  • Create a Checklist: You will need to know exactly what you will be migrating, so it’s important to create a detailed list of all scripts, prerequisites and data files to be migrated. This is basically a finalized list of artifacts from your housekeeping!
  • Execute and verify the migration: This can be as simple as the import feature we mentioned above with Katalon. You want to make sure that everything you need has been brought over and is working as designed.
  • Take advantage of new features: Selenium IDE had very limited functionality. The new tool you have chosen probably has a lot of new features that will accelerate your test automation and help deliver quality applications.  For example, Katalon integrates with your CI/CD tool and can help kick off your automation suites right when builds are deployed. This is very important if your organization is transforming to a DevOps model.

 

As you can see, Selenium IDE being discontinued threw a wrench in a lot of organizations, but it doesn’t have to be as troublesome as you might think. It actually can be a good thing as it will now get organizations to think of automation in a broader spectrum which will ultimately lead to quicker releases with higher confidence.

Our testing services can help your team

Automation in Shift Left Testing

What is Shift Left Testing?

The global movement in testing has become centered around shift left models as more and more companies adopt agile and DevOps practices, which further support microservices architecture. One of the main focuses of shift left strategy is proper utilization of automation, but before we get into that, what is a shift left model?  Shift left testing is basically testing ‘early and often’, using tools that can be adopted by developers of the team and provide validation ‘as needed.’ It becomes the focal point in adopting Continuous Integration and Continuous Deployment models.  The testing is moved left on the project timeline to reduce project costs of finding and resolving bugs. You start testing early in SDLC so that:

  • Defects are found and fixed earlier
  • You will have better test coverage
  • You’ll have quicker releases with increased confidence
  • Quality is the responsibility of the whole team

At KMS, we apply a shift-left, automation-first mindset to everything we build. Contact us to see how we can help automate your QA strategy.

SEND US A NOTE
 

How does Automation fit into Shift Left?

Now that you know what shift left testing is, how do we get the biggest bang for our buck?? Automation! Traditional automation was mainly focused on GUI and End-to-End tests. The problem with that approach is that it takes way too long to complete execution (opposite goal of fast feedback to dev), and you can’t start automating until you have a “finished” product. The old way is very costly due to the effort it takes to create automated  UI scripts, maintenances of flaky tests, and limited test coverage.

With companies adopting agile and DevOps, you cannot wait until the UI is built to automate GUI tests.  When you start testing early, you more than likely will not have fully functional UI, but you will have components (unit tests) and an API layer to support integration points with other systems/areas of the enterprise solution. If you focus your automation on those layers first, you will have the majority of critical test coverage automated. In addition, unit and API tests run the quickest, with faster feedback on quality.  Reducing UI tests will definitely increase velocity in your sprint, and guess what happens when your velocity is increased? You release more often, with better confidence in product quality. That is the main objective of shift left testing!

Now you’re probably wondering, “I am a QA resource. I can’t write or automate unit tests.” In the shift left model, everyone on the team, including dev and QA, is responsible for testing. The developer can write automated unit tests and leverage tests/criteria from the testers as part of the CI process. The tester can include their automated tests into the test suite and add additional value by applying context-driven manual tests. This alone already shows better test coverage.  You also don’t have to wait for a stable UI to automate. It can start left of the project timeline! By executing a combination of automated and manual tests earlier, you will more than likely find defects early on. Uncovering and fixing major bugs early in the cycle is a lot cheaper than finding the bug towards the release date.

Better quality is proven with shift left testing.

Below are some better practices that will help you get started with your shift left strategy:

  • Testers define testing scenarios/coverage expectations with checkpoints
  • Testers/Developers write unit test, automation test based on test scenarios
  • Team Reviews code quality and quality of tests created
  • Code is written with testability in mind

GET IN TOUCH WITH KMS
 

Selecting Automation Candidates in Katalon

Every executive management team wants their testing teams to apply automation to their software application, and they want it started now!  The second part is easy, you can get up and started in minutes using Katalon. How to apply and select tests for automation is a completely different story.

We are pros at test automation — contact KMS to see how we can automate your test strategy.  

SEND US A NOTE
 

Before you even start thinking about automation you need to know what kind of benefits you are getting by automating those tests. There are 3 main benefits of applying test automation:

  1. Reducing Testing Cycle times
  2. Reducing Man effort on testing
  3. Wider Test Coverage

So now that you know what the key benefits are, how do we determine which tests should be automated?? Below are five key tests that I would suggest everyone automate:

  • Real World Business Scenarios – These are critical user workflows for your application. For example, if you had a banking application, the critical scenario would be to make a deposit. This needs to be automated because if this fails, the application fails! With Katalon this can be done simply by using record/playback features. This will create the scenario while also adding the object into repository for future use. Now Record/Playback is for beginners, so more experienced automation testers can create test scripts using the keyword driven editor or code directly in groovy language.

 

  • Integration/Component Tests – API tests are prime candidates for automation. They verify the component layer of your product. API tests are very easy to automate, and on top of that, they are quick to execute. APIs and Integration tests should be run every time a build is kicked off because they are good candidates for sanity tests. If your APIs and Integration Tests fail, your application will fail, hence the build can be rejected at that instance. These types of test should be run on every build/release. The best way to set this up is to utilize your CI tool such as Bamboo, Jenkins, or TeamCity so that your smoke, sanity, and regression tests automatically kickoff when the build is kicked off. This can easily be done with Katalon as it supports console mode execution which integrates with Bamboo, Jenkins, and TeamCity.

 

  • Tests that create data – Most times we don’t really consider business workflows that you use to create data as tests. For example, if you’re in a Patient Record application, you might need to create 100 patients in order to verify a certain test. Creating 100 patients could take a long time, but if you automated that test, it would save considerable time and effort. With Katalon, you can create via application, xml, or directly through DB.

 

  • Tests that use the same workflows but different data inputs – A good example of this would be role based workflows. For example, if you login as an admin user you get more options in dropdown menus. Using Katalon, you can specify your data input values in excel, xml, or directly in GUI. Katalon will then run your tests using all iterations of your data values within minutes. Data Driven tests provide you with wider test coverage.

 

  • Prerequisites/Data Setup – These types of tests are functions that can be reused across other tests.  For example, an online shopping application would require you to login and then add an item to the shopping cart. These steps happen all the time, and with automation, you can create functions for login and add items. You can then reuse these functions in part of a larger test case. In Katalon, there is a feature that allows you to call test cases. If you create your common test cases as functions, for example ‘login’, you can the call that test case in other scripts. A good automation practice is to first list out all common functions and then automate them. This allows you to speed up test case creation so you don’t have to repeat those steps over and over.

Getting started with Automation can be intimidating, but if you already know what you would like to automate, half the battle is complete. Katalon Studio allows you to automate all of the above, keep them organized in a nice, clean view, and be able to reuse objects across multiple types of tests. Due to the very intuitive UI, you can group all of your different types of tests/test suites for execution. For example, you have test suites for all of your Real Word Scenarios, Integration Tests and Data Driven Tests. These tests can be run across multiple browsers, as well as integrated with your CI/CD process. If you’re starting automation, Katalon is your best bet as a one-stop shop!

GET IN TOUCH WITH KMS
 

Passing an audit with maximum testing coverage and high confidence

Software testing for companies in highly regulated industries, such as pharma or clinical research, is a daunting task. Regulatory bodies follow a strict audit process in these sectors because you are dealing with software products that could potentially have life-threatening consequences if not tested properly. The compliance challenges force organizations to provide a deeper level of task/test traceability than in traditional software testing.

Luckily, KMS has the experience and expertise to implement strong, compliant QA strategies in these industries. Learn more about what we can do for your company. 

SEND US A NOTE
 

With all that said, the underlying goal is still the same: You need to maximize testing coverage to create the highest level of confidence in the product and release to market as soon as possible. The caveat here is that you have to ALSO pass a strict audit. Here are the four most common questions that every executive team will expect to answer when trying to deal with regulatory pressure:

  • How can we show direct evidence of testing?
  • How can we streamline the testing process?
  • How can we ensure high confidence in the quality of the application/product?
  • How can we become more effective?

Providing proof across testing operations
In the regulatory world, proof of test activity is everything. Visual proof is even more critical. You can’t say, “It passed my tests, so it’s good,” results need to be seen to be believed. Demonstrable proof has to be there in the form of data. To show proof you have to store test results, time-stamped per test execution, in a central repository – preferably in electronic form. Test cases and execution results can be stored in tools such as qTest, JAMA, HP ALM and similar solution. In addition to documenting that testing was completed, you need to show proof of the specific evaluation processes you followed. This can be accomplished by:

  • Offering screenshots of test results.
  • Tracking a time-stamp of test executions.
  • Providing records of who executed various tests.
  • Recording what version of the software was verified in each assessment.
  • Detailing defect verification and closure.

These are just a few of the essential details you should gather for each project. Maintaining these records allows an auditor to validate ALL tests you have run because you are providing clear proof in the form of a concise audit trail.

Streamlining processes to improve time to release
Now that you have shown proof and taken screenshots of testing activities, you have to be prepared to address how your organization can streamline its testing process. In many cases, creating a thorough audit trail using traditional methodologies can lead to a situation where it is “taking too long” to get the software ready for release.

In order to properly streamline any testing process you need to first list all the tasks and roles involved into those operations. You should also take time to evaluate the purpose and goal of each of those tasks. A good way to visualize this is through the use of mind maps. Some of these tasks might not be needed as they are covered (combined or repeated) with other tasks. A mind map can help you identify such redundancies, making it a good way to eliminate redundant tasks.

Once you believe the process is streamlined, you need communicate and get buy-in from auditors, as they are ultimately responsible for auditing not only the testing tasks, but applied processes as well. If the process fails an audit, your testing will fail as well.

“Anything that is repeatable should be automated.”

Performing the above assessment will also help drive further efficiencies and reduce effort. It will point out where failures occur in your testing process. Knowing wait times (idle time) will allow you to recommend alternate test approaches that will eliminate those failures.

For example, you notice that deploying a build is a manual process that takes hours to finish. This is causing the testing team to be idle and wait for build to test. You can speed this up with  automated build deployments, so that new instances are ready whenever code is checked in. Automating thus will allow the team to continue to test in CI/CD model.

When putting such a plan in place, keep in mind that you are streamlining the process, not the testing. Performing the assessment does not have a negative effect on the test coverage, as we are still completing the tests needed to pass an audit.

Most regulated industries have repeatable process and products – and anything that is repeatable should be automated. Once automation is verified you don’t have to do those steps again and don’t have to worry about audits either, as the process is scripted to pass an audit. Automation will help drive down cycle times which in turn will make you more efficient.

automationIdeally, automation works in tandem with testing, streamlining the process.

Using testing to create confidence in product quality
Ultimately, confidence in quality will be the number one driver of your testing in the regulatory world, as defects that escape your evaluations can have potentially life-threatening impacts. If there is an industry that expects (or has zero tolerance for) minimum defect escapes, it would be the regulatory sector. That’s is why effective testing is critical.

Of course, you cannot to do exhaustive testing, so the first thing a tester needs to have is solid domain knowledge. This will help the testers know exactly where to look for defects. An effective test plan will be key too, as that is driving your testing. Test plans should give assessors enough details on what to evaluate and how to perform those tests to ensure they are maxing out test coverage. Testers then should take that plan and write the most detailed test cases possible. In addition, testers need to preemptively minimize gaps in coverage. To make this possible, you need to have solid a communication plan or channel between devs and testers to ensure code or spec changes are properly communicated to the teams. A good way to do that is through code repositories, such as GitHub.

Creating a culture of sustained test effectiveness
The reason for extra procedural tracking and documentation in regulated sectors is, again; you need to let auditors know exactly what you are testing and what the outcomes are. Another important way to demonstrate confidence in product quality will be through metrics. Metrics are a good way to validate that the testing process and product quality are meeting expectations. If the metrics show that confidence in product quality and effort applied is not optimum, then that is key indicator that things need to change.

Testing in highly regulated industries may seem like it’s the same type of testing you have done elsewhere, but it’s definitely not. You will face much more scrutiny across the software assessment process. This may seem daunting, but the cost of have a defect escape into the actual application cloud be a life-threatening risk, making the tester’s job much more critical.

GET IN TOUCH WITH KMS