usability

Medical Devices and Exploratory Testing

Not long ago I sat down and had a really good talk with Ilari Henrik Aegerter from House Of Test. Ilari has an extensive background in high-quality testing and is one of the leading opinion makers in the area of context-driven testing. Ilari is also a strong advocate of Exploratory Testing and I challenged him to explain how Exploratory Testing could fit into the world of strict and regulated medical device development.

"One way of thinking about testing is value creation. Value is created by increasing the knowledge about the system.” starts Ilari, “Through testing, you gather information about your system’s desirable and undesirable behavior. Now, some testing methodologies accomplish this more efficiently than others.”

"In traditional, scripted testing, the task is to verify that the system behaves according to a formally specified input, a.k.a "confirmatory checking". I see this approach a lot in the medical device industry, often accompanied by extensive documentation, meticulously describing the input and output.” 

“The drawback of this approach is that it only traverses a very narrow path of the system. The generated results is equally meager when it comes to unveiling previously unknown knowledge about the system. It also abstains from tapping into the creativity and system expertise of the tester. 

Let me make this clearer with an illustration.”

 Ilari

“This diagram describes a medical device. As you can see, we have three ways of looking at the product:

  1. The User Needs: this area represents the features set the customer actually wants.
  2. The formal Specifications: this area is made up of the Design Controls, where the manufacturer attempts to formally define the system he is planning to build.
  3. The actual Implementation: this final area represents the feature set the instrument actually supports 

In the ideal case, a medical device is characterized by perfect intersection between all three areas.

Now, let's evaluate what happens when this is not the case.

"Specified but not implemented" - This area translates to turn up as failed tests in the medical device documentation, something that becomes accentuated in companies that focus on verifying formal specifications. Not uncommon in the medical device industry.

"Implemented and specified but not covered by User Needs" - Sometimes called "Gold Plating". From a scripted test perspective, this area is going to transform into passed test cases. One might question the value of implementing things that the Users don't need, but there might exist business reasons that explains this area.

"Implementation and User Needs intersect but are not covered by Specifications" - This is the result of an experienced developer who knows what the User Needs even though it is not specified. Unfortunately, this part is not going to be covered by the formal verification so we can only hope that the implementation actually works.

Finally, the most important area is represented by the "unmet User needs". This area is going to come back as bugs and change requests in the post-market phase. Despite having done the things right, we have apparently failed to do the right things. Some of these user needs might have been explicitly left out for cost reasons or with the intention to be implemented later. However, the critical part consists of the "things we didn't think about".

And voila, here is where exploratory testing can make a big difference. By applying a broader testing mindset, not being constrained by the narrow path of formal specifications, a wider test coverage is reached and more knowledge is obtained about the system. More knowledge is more value. The best part is that studies have shown that exploratory testing is much more efficient in finding bugs per time unit compared to traditional testing."

At this point, I asked Ilari if these unmet User Needs would not be uncovered during the Design Validation? Wasn't this exactly the objective of validation, to check that we have done the right things? 

"Partly", says Ilari, "the notions are related but not the same."

“In recent years, we have seen an increased focus on usability and human factor engineering in medical device development. We have also seen an increased regulatory focus on performing proper validation for each new release. The whole point of these disciplines is to engage the user perspective, since the user probably knows more about the final usage than the manufacturer. Usability and human factor engineering are valuable tools to emphasize, charter and corroborate user needs in the design process.

Exploratory testing is focusing on similar issues. It leverages the creative and critical thinking of the tester, mimicking the thought process of a user. It challenges the tester as a domain expert and explicitly attempts to uncover tacit and unspecified needs."

I was still skeptical. "But, Ilari, we still need to be compliant with the FDA. We still have to show that the specifications are met and we still need to have documented proof. Exploratory testing makes a big point how traditional, scripted testing is obsessing about repeatability and documentation, indicating that these are not important steps." 

"That is not true.", Ilari responds, "I think this is a misconception about exploratory testing, i.e. that it does not test specifications and does not produce documented evidence. Of course exploratory tests are documented. It's just that it might not use the same level of detail and that it better highlights undesired behavior, instead of only focusing on meticulously writing down everything that works as specified.” 

“My opinion is that exploratory testing generates more knowledge about the system. This is particularly valuable during the early development stages. If you have worked in the industry, you know exactly what I am talking about. At this stage, the specs are constantly changing, the device is continuously modified, a lot of factors have not yet stabilized and it is simply not efficient to start writing formalized test scripts at this point. During these phase, exploratory testing is the superior testing method for generating knowledge about the current system state.” 

“This idea is partly reflected in the FDA document Design Considerations for Pivotal Clinical Investigations for Medical Devices - Guidance for Industry, Clinical Investigators, Institutional Review Boards and Food and Drug Administration Staff”. The document states that an exploratory studies in the early, and even pivotal stage of a clinical investigations, is advantegous, simply because it generates more knowledge about the system. In this line of thought, exploratory testing does not replace scripted testing, but certainly precedes and complements it.”

I must say that Ilari is building a case here. We leave the discussion at this point, both excited about the prospects and challenges of exploratory testing in a medical device context.

Take Usability Engineering seriously!

"Dexcom has announced a massive and serious continuous glucose monitors recall, involving more than 260,000 devices with alarms that may fail to sound."

Not hearing the alarm of a medical device is not a good thing. When should the manufacturer have picked up on this? During the usability validation at the latest.

Underestimating usability engineering and proper usability validation can have serious consequences (obviously). Still, usability engineering is still in many places regarded as an outsider practice by many "regular" engineers, considering it being a discipline about "taste". I assure you that this is not the case.

usability

There is a tremendous potential in applying usabaility engineering during product development. Standards covering usability and human factors design such as the IEC 62366-1:2015 address the potential risks of poor usability design. However, being compliant to theses standards will not only make the medical device safer. They also makes the product easier to use and thereby creates more value to the users and patients. 

To make the documentation of the usability engineering tasks faster and easier, we have added a preconfigured IEC 62366-1:2015 setup to Aligned Elements.

The Aligned Elements IEC 62366 Usability Configuration

With the 2015 update of the IEC 62366-1:2015 and the issuing of the FDA  Human Factor Guidance, usability engineering in medical device development has received increased attention recently.

Human factor analysis gets more important with the rising trend of offering patient-centric solutions via new mobile health applications and wearables.
When patients, rather than specialists become the medical device user, increased focus needs to be placed on the patients capabilities and the environment in which the device is used. IEC 62366-1 is applied in an effort to increase patient and user safety by identifying, assessing and mitigating Use Errors, by paying attention to the usability of the device design and harness existing usability verification and validation methods to make sure that usability requirements are met and use errors are avoided.

usability

For medical device manufacturers with limited previous experience in usability engineering, the task of implementing IEC 62366-1 might seem intimidating. However, the updated 2015 version of the standard has simplified and clarified the required process steps and tasks and Aligned Element now features a preconfigured setup that integrates the inputs, outputs and risk relevant elements of the usability process into the oveall Design Control traceability.

The configuration includes:

  • 8 Design Control templates for capturing the input and output elements of the usability process
  • Pre-configured content validation checks assessing the consistency of the project in real-time
  • ISO 14971 compliant risk management to identify, assess and mitigate Use Error-driven risks
  • Interactive Checklist for reviewing the Use Specifications Document against IEC 62366-1:2015 
  • 20 usability example risks for Use Errors during the Transport, Storage, Installation and Decommissioning
  • 2 preconfigured Traceability Tables
  • Integrated test protocols for established test methods such as:
  • Cognitive Walkthrough
  • Heuristic Evaluation (based on the Nielsen-Schneidermann heuristics)
  • Simulated-Use Testing for Usability Validation

On top of all that, the Aligned Elements IEC 62366 includes all the standard features of Aligned Elements, including:

  • Completed change control of all Design Control Items
  • Complete audit trail for all changes
  • Keep your company specific look and feel of all report outputs
  • End-to-end tracability with real-time impact analysis
  • Easy and fast Word reporting using drag and drop
  • FDA QSR 21 CFR Part 11 compliant User Management
  • Efficient decision-making process using workflows and E-signatures


By applying selected parts of the Aligned Elements IEC 62366-1 configuration to your Aligned Elements setup, you can efficiently leverage Aligned Elements in your IEC 62366-1 compliance effort.

The Aligned Elements IEC 62366 configuration is available for download in our extension library.

For a live demonstration of the Aligned Elements IEC 62366 configuration, please This email address is being protected from spambots. You need JavaScript enabled to view it. to set up an appointment.