Secrets to Setting Expectations for Exploratory Testing

If you adopt exploratory testing the right way, you’ll find the benefits are compelling. It can enable your test team to find defects earlier in production, boost their satisfaction levels and skillset as they’re afforded more responsibility, and result in a better quality product. But, it’s easier said than done. There may be resistance, or a lack of understanding – that’s why you have to set the right expectations and get the whole team to buy into a new mindset.

We are experts at exploratory testing. Let us know how we can help support your testing team.


Earlier results and less documentation

Exploratory testing is not akin to recklessly throwing the map out of the window and going off-road, but it does produce less documentation and gives testers more freedom. It’s important to show everyone on the team that its success still depends on proper planning.

Workflow charters and mind-maps allow testers to examine the flow of a product. This holistic approach tends to get results much more quickly than the traditional approach. It’s also more visual, as testers use recording tools to capture test sessions. Make sure there’s plenty of communication between the test department and development, so the new processes are clear.

Taking responsibility for results

Testers are expected to really push themselves with exploratory testing and go the extra mile. They have to think critically, explore new ideas, and track down problems wherever they may lie. It means shifting from following a rigid set of steps in a test case, to achieving the same goal by beating their own path.

Group evaluations are vital. Testers must discuss and analyze their findings to help each other set priorities, severities, and to evaluate the next plan of attack. That additional responsibility requires a proactive attitude. Self-directed testing will ultimately lead to a much deeper understanding of the product and greater job satisfaction, but there’s a learning curve there as testers adjust.

More coverage with fewer metrics

That traditional focus on documentation and the generation of statistics about the number of test cases passed and failed can be hard to let go of. Exploratory testing defines broad areas to be covered and produces fewer numbers. If there’s doubt, then the defect count is a good baseline and proof of its efficacy.

Serious defects are generally uncovered earlier in the process with an exploratory approach, and that gives developers more time to fix them. The earlier a defect is discovered, the cheaper and easier it is to fix. It also helps pave the way for a natural progression from bug fixing to polishing, and that results in a great end product.

In the right circumstances, exploratory testing really delivers results, but it’s important to set expectations at the outset and really explain the process. Educate and train your entire team about how this technique works, and encourage them to talk directly to each other. If your team can approach exploratory testing with the right attitude and a spirit of collaboration, it will lead to a better quality product every time.


Regression – What Is It and When Is Enough Enough?

Looking for a partner to scale your testing strategy?


I started off my QA career as an automated regression tester, transitioning from a project manager into this role because of a defect that cost the company money. The company had a newly formed QA team of a year or so and they were rocking and rolling with their functional testing preventing production bugs, building friendships with PMs and love hate relationships with developers, in instant things changed. One functional change that passed all testing caused another part of the code to have a small costly defect, the solution regression testing became part of the process however with the size of the team and the enormity of the applications these dual tasks became too much and a position for automated QA regression Analyst as opened. I became that Analyst.

There was a little bit of training for QA back then but not much on regression so there I was starting as a testers with limited industry direction but great co-worker input. I began writing automated regression test for this large company on a home grown automation tool developed by my manager with no clue what I should or should not be covering. Now the neat thing about where I was at the time was with a home grown tool all I had to do was ask for updates to make testing easier and within days my manger delivered the updated tools, not dissimilar to the situation I am in now with a tool that can be updated upon request to perform many testing duties unique to the company or environment. Today I am focusing thoughts and opinions on “What is regression suites and when is enough enough?”

To scope today’s subject we are discussing what should be regression tested not what should be automated from that regression. Automation can affect the coverage amount for regression for and perhaps in a future dialogue I will express my opinion on that subject but for today I am focusing on what should be included in regression.

What should be in a regression suite?

Schools of thought: Functionality used by 75% of the users; Everything; only the Happy Paths. What type of regression do you run: Thoughts include functional based testing and other scenario based testing?

Let’s start with the Happy Path in what should be in a regression suite. Happy paths should be the starting point for all regression testing, what is the most likely path most users will follow when they initially run this application or access this URL? If this is your bread and butter you need to make sure it does not become toast and cover this as a user story or scenario that follows the path of the user. When you start to build your regression suite start there, followed by the money items. Anything that may not be in a scenario but is high on the revenue stream regression test this functionality. For instance if you get browsing at 200 a day but buying at only 2, you will still need to have the pages for purchasing fully regressioned testing because those two transaction funds the bread and butter.

What and how do you build from there, do you simply take the functional test cases and move them to regression suite after each release? My opinion is no, functional test are just that test of functionality, regression is the test of the user experience they will not follow the requirements written, they will follow their fingers and in a regression suite you need to be testing where the fingers will lead the user not where the requirements intended them to go. Regression test cases can start from functional test cases but need to be modified to a user scenario based case that may require modification to the existing test case to make the scenario realistic.

So where do you stop at 50%, 75% or 100%? As a realistic dreamer I like to suggest no less than 75% but never 100% of the functionality that is just unachievable unless your application is consistent and never changes I would be OK with 99%. In the real world everything changes from new devices to upgraded operating systems so to not frustrate the QA Analyst I would steer clear of the 100%. This leads to the question, “If we do not have 100% coverage would we miss something in testing”? The answer is maybe, however if the regression suite covers 75% or higher the user experience and the money sections the likelihood of disruption to the user or revenue stream from a production defect/bug.

I believe very strongly in scenario based testing as opposed to functional, in implementation there should be a distinction between the functional and the regression testing. Functional should include testing the change but also exploratory testing which does cross into the regression realm. Exploratory testing is A good transition to regression, it is an important step and expands coverage but does not eliminate the need for formal, consistent regression testing. Keeping in mind the functional test approach will ensure more functionality coverage it may miss the flow and usability defects.

In conclusion, my thoughts on regression testing are follow the leader and the money through most of their steps to build and maintain your regression suite.