Passing an audit with maximum testing coverage and high confidence
Software testing for companies in highly regulated industries, such as pharma or clinical research, is a daunting task. Regulatory bodies follow a strict audit process in these sectors because you are dealing with software products that could potentially have life-threatening consequences if not tested properly. The compliance challenges force organizations to provide a deeper level of task/test traceability than in traditional software testing.
Luckily, KMS has the experience and expertise to implement strong, compliant QA strategies in these industries. Learn more about what we can do for your company.
With all that said, the underlying goal is still the same: You need to maximize testing coverage to create the highest level of confidence in the product and release to market as soon as possible. The caveat here is that you have to ALSO pass a strict audit. Here are the four most common questions that every executive team will expect to answer when trying to deal with regulatory pressure:
- How can we show direct evidence of testing?
- How can we streamline the testing process?
- How can we ensure high confidence in the quality of the application/product?
- How can we become more effective?
Providing proof across testing operations
In the regulatory world, proof of test activity is everything. Visual proof is even more critical. You can’t say, “It passed my tests, so it’s good,” results need to be seen to be believed. Demonstrable proof has to be there in the form of data. To show proof you have to store test results, time-stamped per test execution, in a central repository – preferably in electronic form. Test cases and execution results can be stored in tools such as qTest, JAMA, HP ALM and similar solution. In addition to documenting that testing was completed, you need to show proof of the specific evaluation processes you followed. This can be accomplished by:
- Offering screenshots of test results.
- Tracking a time-stamp of test executions.
- Providing records of who executed various tests.
- Recording what version of the software was verified in each assessment.
- Detailing defect verification and closure.
These are just a few of the essential details you should gather for each project. Maintaining these records allows an auditor to validate ALL tests you have run because you are providing clear proof in the form of a concise audit trail.
Streamlining processes to improve time to release
Now that you have shown proof and taken screenshots of testing activities, you have to be prepared to address how your organization can streamline its testing process. In many cases, creating a thorough audit trail using traditional methodologies can lead to a situation where it is “taking too long” to get the software ready for release.
In order to properly streamline any testing process you need to first list all the tasks and roles involved into those operations. You should also take time to evaluate the purpose and goal of each of those tasks. A good way to visualize this is through the use of mind maps. Some of these tasks might not be needed as they are covered (combined or repeated) with other tasks. A mind map can help you identify such redundancies, making it a good way to eliminate redundant tasks.
Once you believe the process is streamlined, you need communicate and get buy-in from auditors, as they are ultimately responsible for auditing not only the testing tasks, but applied processes as well. If the process fails an audit, your testing will fail as well.
“Anything that is repeatable should be automated.”
Performing the above assessment will also help drive further efficiencies and reduce effort. It will point out where failures occur in your testing process. Knowing wait times (idle time) will allow you to recommend alternate test approaches that will eliminate those failures.
For example, you notice that deploying a build is a manual process that takes hours to finish. This is causing the testing team to be idle and wait for build to test. You can speed this up with automated build deployments, so that new instances are ready whenever code is checked in. Automating thus will allow the team to continue to test in CI/CD model.
When putting such a plan in place, keep in mind that you are streamlining the process, not the testing. Performing the assessment does not have a negative effect on the test coverage, as we are still completing the tests needed to pass an audit.
Most regulated industries have repeatable process and products – and anything that is repeatable should be automated. Once automation is verified you don’t have to do those steps again and don’t have to worry about audits either, as the process is scripted to pass an audit. Automation will help drive down cycle times which in turn will make you more efficient.
Ideally, automation works in tandem with testing, streamlining the process.Using testing to create confidence in product quality
Ultimately, confidence in quality will be the number one driver of your testing in the regulatory world, as defects that escape your evaluations can have potentially life-threatening impacts. If there is an industry that expects (or has zero tolerance for) minimum defect escapes, it would be the regulatory sector. That’s is why effective testing is critical.
Of course, you cannot to do exhaustive testing, so the first thing a tester needs to have is solid domain knowledge. This will help the testers know exactly where to look for defects. An effective test plan will be key too, as that is driving your testing. Test plans should give assessors enough details on what to evaluate and how to perform those tests to ensure they are maxing out test coverage. Testers then should take that plan and write the most detailed test cases possible. In addition, testers need to preemptively minimize gaps in coverage. To make this possible, you need to have solid a communication plan or channel between devs and testers to ensure code or spec changes are properly communicated to the teams. A good way to do that is through code repositories, such as GitHub.
Creating a culture of sustained test effectiveness
The reason for extra procedural tracking and documentation in regulated sectors is, again; you need to let auditors know exactly what you are testing and what the outcomes are. Another important way to demonstrate confidence in product quality will be through metrics. Metrics are a good way to validate that the testing process and product quality are meeting expectations. If the metrics show that confidence in product quality and effort applied is not optimum, then that is key indicator that things need to change.
Testing in highly regulated industries may seem like it’s the same type of testing you have done elsewhere, but it’s definitely not. You will face much more scrutiny across the software assessment process. This may seem daunting, but the cost of have a defect escape into the actual application cloud be a life-threatening risk, making the tester’s job much more critical.
