Exploratory Testing for Product Configuration


When testing product configurations, testers don’t usually rely on exploratory testing. They argue that they are not testing traditional software products but various different configurations for the product. In reality, exploratory testing can be a very useful technique that can reduce testing efforts and improve speed to market. But before we dive into that let me first provide you with some key definitions:

Exploratory testing is methodology where a tester designs and tests at the same time. The focus here is to spend less time on formal documentation like test conditions, and detailed test steps. The process is really about using the tester’s skill and domain knowledge to explore/test the product for issues instead of just ‘checking’.

Product Configuration or product customization, is an activity of customizing or configuring a software product to meet the needs of a particular customer. For example, a pharmaceutical company has asked a clinical trial company to develop a drug study. These drug studies will be different for each pharmaceutical company even though they could be using the same base. In addition, the set of questions that appear to the end-patient will be different depending on multiple factors.

Now that you have some of the key definitions, let’s discuss how exploratory testing can be used in configuration testing.

Stakeholders will tell you that they want effort and speed to market reduced, while still maintaining a high level of quality. No process/approach can provide 100% coverage but testers do have to utilize testing approaches that provide maximum test coverage while providing a high level of confidence. In configuration based testing, the scenarios can be very specific or at times can be very vague. Testing teams will not have the bandwidth to test all scenarios. For example, in our past experience working with a pharmaceutical based ISV that develops customized software for clinical trials, we have noticed many different configurations. Trial users will have different sets of question workflows based on the answers provided. There are many flows that can occur here and not all them could be documented. More importantly, if there are defects due to settings/configurations, it can potentially be life threatening. Applying exploratory testing, you can create sessions and charters to expand the scope, especially if not everything is documented. This provides better coverage.

Exploratory testing can also be used to speed up your release as well. Majority of testers will spend a lot of time on formal documentation, especially if it’s configuration based. They will write tests for every single scenario/configuration. That could take quite a bit of time. Sometimes they spend more time writing then testing. If you create charters and sessions, you are already documenting what you are testing. You save time here as now you’re spending more time testing and less time on documentation. In addition, you can use tools that can automatically capture/record your test steps while you’re testing. This is very important in regulated industries as proof of testing is key. Once such tool that does this is qTest Explorer by QASymphony. It tracks user’s interactions from the testing session and uses that information to automatically create defect documentation. It supports the recording of a wide range of applications and technologies on the browser as well as desktop apps. Using exploratory testing and tools can drastically reduce your effort, hence speed to market is improved.

There are also other areas in configuration based testing that exploratory testing can help. Its proven to find major defects early in testing, which everyone knows is a lot cheaper to fix early in a cycle. It is also good to apply for regression after all defects have been resolved. This ensures maximum coverage.

In conclusion, you can see that exploratory testing can and should be applied to configuration based testing due to the fact that it will provide better coverage and help reduce effort.

Find out more about our software testing services


Passing an audit with maximum testing coverage and high confidence

Software testing for companies in highly regulated industries, such as pharma or clinical research, is a daunting task. Regulatory bodies follow a strict audit process in these sectors because you are dealing with software products that could potentially have life-threatening consequences if not tested properly. The compliance challenges force organizations to provide a deeper level of task/test traceability than in traditional software testing.

Luckily, KMS has the experience and expertise to implement strong, compliant QA strategies in these industries. Learn more about what we can do for your company. 


With all that said, the underlying goal is still the same: You need to maximize testing coverage to create the highest level of confidence in the product and release to market as soon as possible. The caveat here is that you have to ALSO pass a strict audit. Here are the four most common questions that every executive team will expect to answer when trying to deal with regulatory pressure:

  • How can we show direct evidence of testing?
  • How can we streamline the testing process?
  • How can we ensure high confidence in the quality of the application/product?
  • How can we become more effective?

Providing proof across testing operations
In the regulatory world, proof of test activity is everything. Visual proof is even more critical. You can’t say, “It passed my tests, so it’s good,” results need to be seen to be believed. Demonstrable proof has to be there in the form of data. To show proof you have to store test results, time-stamped per test execution, in a central repository – preferably in electronic form. Test cases and execution results can be stored in tools such as qTest, JAMA, HP ALM and similar solution. In addition to documenting that testing was completed, you need to show proof of the specific evaluation processes you followed. This can be accomplished by:

  • Offering screenshots of test results.
  • Tracking a time-stamp of test executions.
  • Providing records of who executed various tests.
  • Recording what version of the software was verified in each assessment.
  • Detailing defect verification and closure.

These are just a few of the essential details you should gather for each project. Maintaining these records allows an auditor to validate ALL tests you have run because you are providing clear proof in the form of a concise audit trail.

Streamlining processes to improve time to release
Now that you have shown proof and taken screenshots of testing activities, you have to be prepared to address how your organization can streamline its testing process. In many cases, creating a thorough audit trail using traditional methodologies can lead to a situation where it is “taking too long” to get the software ready for release.

In order to properly streamline any testing process you need to first list all the tasks and roles involved into those operations. You should also take time to evaluate the purpose and goal of each of those tasks. A good way to visualize this is through the use of mind maps. Some of these tasks might not be needed as they are covered (combined or repeated) with other tasks. A mind map can help you identify such redundancies, making it a good way to eliminate redundant tasks.

Once you believe the process is streamlined, you need communicate and get buy-in from auditors, as they are ultimately responsible for auditing not only the testing tasks, but applied processes as well. If the process fails an audit, your testing will fail as well.

“Anything that is repeatable should be automated.”

Performing the above assessment will also help drive further efficiencies and reduce effort. It will point out where failures occur in your testing process. Knowing wait times (idle time) will allow you to recommend alternate test approaches that will eliminate those failures.

For example, you notice that deploying a build is a manual process that takes hours to finish. This is causing the testing team to be idle and wait for build to test. You can speed this up with  automated build deployments, so that new instances are ready whenever code is checked in. Automating thus will allow the team to continue to test in CI/CD model.

When putting such a plan in place, keep in mind that you are streamlining the process, not the testing. Performing the assessment does not have a negative effect on the test coverage, as we are still completing the tests needed to pass an audit.

Most regulated industries have repeatable process and products – and anything that is repeatable should be automated. Once automation is verified you don’t have to do those steps again and don’t have to worry about audits either, as the process is scripted to pass an audit. Automation will help drive down cycle times which in turn will make you more efficient.

automationIdeally, automation works in tandem with testing, streamlining the process.

Using testing to create confidence in product quality
Ultimately, confidence in quality will be the number one driver of your testing in the regulatory world, as defects that escape your evaluations can have potentially life-threatening impacts. If there is an industry that expects (or has zero tolerance for) minimum defect escapes, it would be the regulatory sector. That’s is why effective testing is critical.

Of course, you cannot to do exhaustive testing, so the first thing a tester needs to have is solid domain knowledge. This will help the testers know exactly where to look for defects. An effective test plan will be key too, as that is driving your testing. Test plans should give assessors enough details on what to evaluate and how to perform those tests to ensure they are maxing out test coverage. Testers then should take that plan and write the most detailed test cases possible. In addition, testers need to preemptively minimize gaps in coverage. To make this possible, you need to have solid a communication plan or channel between devs and testers to ensure code or spec changes are properly communicated to the teams. A good way to do that is through code repositories, such as GitHub.

Creating a culture of sustained test effectiveness
The reason for extra procedural tracking and documentation in regulated sectors is, again; you need to let auditors know exactly what you are testing and what the outcomes are. Another important way to demonstrate confidence in product quality will be through metrics. Metrics are a good way to validate that the testing process and product quality are meeting expectations. If the metrics show that confidence in product quality and effort applied is not optimum, then that is key indicator that things need to change.

Testing in highly regulated industries may seem like it’s the same type of testing you have done elsewhere, but it’s definitely not. You will face much more scrutiny across the software assessment process. This may seem daunting, but the cost of have a defect escape into the actual application cloud be a life-threatening risk, making the tester’s job much more critical.