How I got started in testing medical software
I am Yulia (Yudit) Sharabi, and I‘ve worked in the testing industry for over a decade.
I started in the networking domain as a student, doing system testing in a waterfall environment. Then I moved to component testing and Agile practice. Following that, I transitioned into test automation.
My experience in the medical domain came during my mid-career role as QA lead at a small, fast-paced medical company. My role was multifunctional for a while, but as the company scaled, I recognized that a dedicated QA function was needed. After nine months of studying regulatory and compliance requirements on the job, I knew what kind of testing team to recruit to fit the unique challenges of the medical tech field.
Unique challenges in testing medical tech
Medical tech poses a number of challenges for product teams. Testing and quality are especially critical in medical software because they have direct effects on life-and-death health decisions. For example, think about the role of quality with respect to heart-monitoring machines, lab result portals, and blood-sugar monitors.
Some of the software used in the medical field is "active," while other software is "preventive." Active medical software acts directly on the body, including such technology as robots that perform heart operations and wearable devices like insulin pumps. Preventive software serves to monitor and report without making active interventions itself. Examples include applications that monitor vital signs.
Software bugs can cause vital signs and other critically important information to be misreported. Those erroneous reports can lead to bad health decisions, like the administration of too much or the wrong medication. That's why testing is so important.
Strict regulation of medical software
The medical domain is heavily regulated to safeguard quality and safety. A company that wants to bring a medical product to market must first comply with regulatory standards and achieve certification and approval. Without it, the product cannot be legally sold.
In the United States, the FDA regulates medical technology, while in Europe, medical products must show their compliance with regulations by bearing the CE mark. A voluntary standard that is commonly followed is ISO 13485. The regulations that must be followed depend on the market the company is targeting. Those standards pave the way for our processes, and evidence is essential.
Medical software is audited by regulatory agencies from time to time, which necessitates a few processes that are unique to the medical device arena.
User experience matters much more in the medical field
The second challenge is to make sure the user experience is clear and easy to follow.
This is crucial. Consider a senior man who goes to the nurse. When he needs clarification, the nurse is there to instruct him. However, if he is using an app that isn't intuitive, especially for elderly people who aren't techies, it may result in incorrect readings and treatments, which is far more problematic.
Documentation of the software development process is critically important
Documentation is the third challenge. It can be challenging when the company is a start-up with frequent releases. While using test management tools for documentation and traceability is often sufficient in other domains, in the medical device field it doesn't work.
As my manager used to tell me, “what is not documented, DID NOT happen.” Each release needs to be indexed, each change request written, and each test case run documented, reviewed, and signed by a person in authority at the organisation. You cannot sign your own report.
Document review is an essential part of the document production process. The testing team works on it with their manager and the VP of research and development. As always, we can’t afford bottlenecks. So we use a document control system to help us navigate it. Just like code review tools, a good document control system allows us to see quickly the differences between previous versions and the current one under review. This makes the review phase much faster and the signing easier.
Complete and accurate documentation is crucial for an audit. An auditor can choose any single risk from a risk analysis file, then ask you to show him the corresponding requirement, the specific test cases correlated to it, and execution reports for those tests. Being able to present the full chain of traceability can make the difference between being in compliance and getting a major finding of fault from the auditor.
What sets medical software testing apart
Usability testing
In medical software testing, we emphasize usability testing. While planning the requirements, we have to consider both the end user and other stakeholders such as nurses and doctors. This continues with a more detail-oriented approach to understanding the target user. Designing the user experience might be different depending on the age, sex, and general health of the patient.
To test the user flow for a urinalysis product, for instance, we conduct usability tests in a lab setting, where we wear gloves and work with real, legally purchased urine samples. Of course we work with guidance from lab experts. This ensures the device works correctly under actual conditions faced by the medical professional.
Similarly, for software used to treat chronic wounds, we use silicon forms of a leg or arm part, complete with a realistic wound model, to train the app's algorithm and verify its accuracy under various physical conditions.
Risk-based testing
A second key difference is the risk-based approach that we must adopt at every stage. Flows that have the greatest health impact or potential for harm are tested first, in greater detail, and with a wider scope than other parts of the application.
This commitment to accuracy often requires us to build and test unique scenarios. For the urinalysis dipstick software, we don't use standard controls on their own. We often use printed dipsticks to simulate known positive and negative results, or we dip actual sticks in water to create a perfect negative control.
Real-world edge case testing
Finally, we must simulate real-life scenarios. Testing should be done in a way that mimics a real-world situation.
Part of my onboarding for the urinalysis dipstick project was to use the app exactly as a real user would. Without further instructions, I was given a kit and told to use my phone to conduct a urinalysis. We cannot assume that users read instructions or that those instructions will even be available to them.
Users in the real world often operate at boundary conditions, like mobile devices with low battery levels or use in a poorly-lit or dark room like a restroom. Also, we must always confirm the app's timeout behavior, since the validity of urinalysis stick results is brief and putting the phone aside even for a few minutes can invalidate a dipped urinalysis kit.
Strategies for testing medical software effectively
Multidisciplinary teams
In traditional software companies, Agile methodology is common. In medical software, though, the teams frequently include not only product owners, developers, and testers, but also doctors, nurses, lab experts, customer support specialists and regulatory affairs representatives.
Multidisciplinary teams help give a wider context to the medical use cases presented by customers and stakeholders.
Risk-based approach
Another method used in the medical field is risk-based approach tooling. The SRS (software requirements specification) is a common input for software testing teams which helps us ensure we sufficiently cover all product risks in our testing.
We also use a traceability matrix that connects test cases back to test requirements. The test requirements are produced by the project manager in response to the risk and mitigation analysis done by the regulatory affairs team.
- For example, for a wearable blood test monitor, the risk of improper device placement leads to a software mitigation: the requirement that the software itself must instruct the user on the correct way to wear the device, complete with illustrations. This specific software requirement generates a direct test case to verify that the instruction exists as specified.
- In another example, a device might be stored in an environment above or below room temperature. This risk is mitigated solely by a warning in the user manual. Since the software is not responsible for environmental monitoring, this risk has no corresponding software requirement and thus no test case.
In most cases, it is best practice to mitigate risks using software requirements, since that helps us control the conditions and setup for the use of the medical device. However, as the second example above shows, it is not always possible.
Risks are reviewed and mitigated, and the traceability matrix becomes part of the testing documentation, all of which is signed off. As a QA lead, it was my duty to make sure all documentation was in place and that traceability from risk analysis to the test case reports was done properly.
Continuing education and monitoring of regulatory changes
As important as learning and training are in any company, they are critical to medical devices. Employee training is mandatory and occurs yearly for all employees in the company. The healthcare industry is subject to a variety of standards and licenses, which are updated from time to time.
The regulatory affairs team regularly performs a “gap analysis” to identify and analyze the differences between current and previous standards. Specialists then mentor the various teams on the changes relevant to their work.
To wrap up
Being a software tester in medical device companies carries a lot of responsibility. Your work affects people's lives and the way they interact with the medical world. You can ease access to treatment and prescription medications. There are challenges, and you need some training and knowledge to do proper work that matters. But having a mission to make the medical world more accessible, make lab results available quicker to patients and their doctors, and save time and money for patients is worth it all.
What do YOU think?
Got comments or thoughts? Share them in the comments box below. If you like, use the ideas below as starting points for reflection and discussion.
- What makes software testing in medical technology more complex than testing in other domains, from your perspective?
- How would you test for “real-world usability” in a device meant for doctors or patients under pressure?
- What strategies have worked best for you when collaborating with cross-functional teams (engineers, clinicians, regulators)?
- Do you think risk-based testing is the future of software testing in high-stakes industries? Why or why not?
For more information
- Testing Ask Me Anything - Testing in Compliance and Regulation, Milan Kuveljic
- The Surprising Benefits of Exploring Other Disciplines and Industries, Conor Fitzgerald
- From Radiotherapy To Quality Assurance: How A Healthcare Background Bolsters QA Excellence, Cristina Carrageis
