Regression – What Is It and When Is Enough Enough?
Looking for a partner to scale your testing strategy?
I started off my QA career as an automated regression tester, transitioning from a project manager into this role because of a defect that cost the company money. The company had a newly formed QA team of a year or so and they were rocking and rolling with their functional testing preventing production bugs, building friendships with PMs and love hate relationships with developers, in instant things changed. One functional change that passed all testing caused another part of the code to have a small costly defect, the solution regression testing became part of the process however with the size of the team and the enormity of the applications these dual tasks became too much and a position for automated QA regression Analyst as opened. I became that Analyst.
There was a little bit of training for QA back then but not much on regression so there I was starting as a testers with limited industry direction but great co-worker input. I began writing automated regression test for this large company on a home grown automation tool developed by my manager with no clue what I should or should not be covering. Now the neat thing about where I was at the time was with a home grown tool all I had to do was ask for updates to make testing easier and within days my manger delivered the updated tools, not dissimilar to the situation I am in now with a tool that can be updated upon request to perform many testing duties unique to the company or environment. Today I am focusing thoughts and opinions on “What is regression suites and when is enough enough?”
To scope today’s subject we are discussing what should be regression tested not what should be automated from that regression. Automation can affect the coverage amount for regression for and perhaps in a future dialogue I will express my opinion on that subject but for today I am focusing on what should be included in regression.
What should be in a regression suite?
Schools of thought: Functionality used by 75% of the users; Everything; only the Happy Paths. What type of regression do you run: Thoughts include functional based testing and other scenario based testing?
Let’s start with the Happy Path in what should be in a regression suite. Happy paths should be the starting point for all regression testing, what is the most likely path most users will follow when they initially run this application or access this URL? If this is your bread and butter you need to make sure it does not become toast and cover this as a user story or scenario that follows the path of the user. When you start to build your regression suite start there, followed by the money items. Anything that may not be in a scenario but is high on the revenue stream regression test this functionality. For instance if you get browsing at 200 a day but buying at only 2, you will still need to have the pages for purchasing fully regressioned testing because those two transaction funds the bread and butter.
What and how do you build from there, do you simply take the functional test cases and move them to regression suite after each release? My opinion is no, functional test are just that test of functionality, regression is the test of the user experience they will not follow the requirements written, they will follow their fingers and in a regression suite you need to be testing where the fingers will lead the user not where the requirements intended them to go. Regression test cases can start from functional test cases but need to be modified to a user scenario based case that may require modification to the existing test case to make the scenario realistic.
So where do you stop at 50%, 75% or 100%? As a realistic dreamer I like to suggest no less than 75% but never 100% of the functionality that is just unachievable unless your application is consistent and never changes I would be OK with 99%. In the real world everything changes from new devices to upgraded operating systems so to not frustrate the QA Analyst I would steer clear of the 100%. This leads to the question, “If we do not have 100% coverage would we miss something in testing”? The answer is maybe, however if the regression suite covers 75% or higher the user experience and the money sections the likelihood of disruption to the user or revenue stream from a production defect/bug.
I believe very strongly in scenario based testing as opposed to functional, in implementation there should be a distinction between the functional and the regression testing. Functional should include testing the change but also exploratory testing which does cross into the regression realm. Exploratory testing is A good transition to regression, it is an important step and expands coverage but does not eliminate the need for formal, consistent regression testing. Keeping in mind the functional test approach will ensure more functionality coverage it may miss the flow and usability defects.
In conclusion, my thoughts on regression testing are follow the leader and the money through most of their steps to build and maintain your regression suite.
