By Scott Danhof
When designing a medical device, nothing beats direct observation of and feedback from the people who will be using it. Ideally, this research is performed as a close partnership between the human factors researchers and the engineers who will be working on the device.
Directly participating in the user research sessions and observing users first hand provides a deeper understanding of user needs and user interface requirements than simply reading the final user research report. Building effective partnerships during the user research process will also ensure that engineers will get answers to the questions that are most critical to them for making design decisions.
To get the most out of user research, engineers need to have an understanding of how the process works and how they can participate productively. Here are five things engineers should know about user research before starting on their next medical device development project.
There is a Structure to All User Research
The earliest user research is often conducted in the homes of patients in the target population. Gathering information in a home visit allows researchers to make observations about the home environment and the ways in which patients behave in their environment that go above and beyond what users actually say.
A home interview may look like an informal conversation. Human factors researchers work hard to put subjects at ease and establish rapport. But, no matter how unstructured the discussion may look to an untrained eye, there is actually a great deal of structure and skill involved in conducting a successful home interview.
All user research interviews are conducted according to a strict protocol with specific research questions established well in advance. This protocol is designed to ensure that the research questions that the study seeks to answer are addressed during the interview and that participants are treated according to guidelines established by the Institutional Review Board (IRB) overseeing the study.
During the interview, the human factors researcher will have a set of predetermined questions to ask. However, he or she must also be able to follow the threads of conversation as they evolve and know how to elicit additional information when clarification is needed and how to gently steer the conversation back to the subject of interest if it should start to veer off track.
When human factors researchers and engineers partner during user research sessions, the engineer must understand the protocol that is being followed and the research questions that will be explored during the interview. Engineers should also prepare their own research questions before the interviews begin. During the interview itself, they should follow the researcher’s lead and allow the researcher to guide the conversation with participants.
You Will Have a Chance to Ask Your Questions
There will be multiple opportunities over the course of the interview for engineers to ask additional questions. It is best to limit these to one or two good questions at each opportunity to ensure that all topics of interest are covered during the interview.
When asking these questions, it is important to avoid the introduction of bias in the wording of the question. Don’t ask leading questions such as, “Did you notice the flashing green light that told you that the therapy was completed?” Instead, ask open-ended questions such as, “How did you know when therapy was completed?”
Observation Yields the Most Reliable Data
Some of the most valuable and reliable information comes not from listening to what participants say, but from observing what they actually do. The rule of thumb here is to “watch first, ask next.” This is true at all stages of research, from early user studies using illustrations or 3D models to summative human factors validation studies.
Participants may tell researchers one thing but do something quite different when they actually complete the task. This is not because they are intentionally misrepresenting facts or omitting information from researchers, but simply because humans are unreliable narrators—to themselves as well as to others. They may forget to describe a step that comes automatically to them when they conduct the task or omit details about their environment or how they interact with a device because they think they are not important.
For example, in one study, researchers were asking participants to describe when and how they took their medication in a typical day. It was observed that most participants took their medication in the bathroom, even when the participants named a different location. This was an important distinction because the device had to be charged, creating a risk for patients who might accidentally drop the device into a sink or toilet while it was plugged in. This observation led to design revisions to better protect patients from this risk.
In another study, participants were asked to try several prototypes of a drug delivery device during a formative research study. After demonstrating how they would take a dose using each prototype, they were asked which model they preferred. Researchers observed that the model they selected did not always match the model that they appeared to use most easily and accurately. Reconciling these differences through further questions provided valuable insights that were used to guide the next stage of development.
Look for Themes
A single observation may offer a clue into a potential usability issue or user need. But to have confidence in the results of a user study, researchers must look for trends across multiple participant sessions. While one observation may be an anomaly, seeing the same pattern emerge across four or five sessions strongly suggests that the observation has validity that device designers should take into account.
The number of participant sessions needed to establish a trend with confidence will depend on how strong the pattern is. Eight sessions are usually sufficient to establish a pattern. Sometimes, a strong pattern can be determined with fewer sessions. When the data is inconsistent, you may need 10 or 12 sessions to determine a pattern.
In one formative research study, the first participant was observed holding a prototype drug delivery device the wrong way. When asked how she knew how to hold it, she said it was similar in appearance to another device she used. After seeing five participants in a row make the same use error, it was clear that changes would need to be made to the form factor of the device in order to break this established mental model and ensure safe and effective use.
Not every pattern is equally important. Sometimes, patterns emerge that have no impact on safe and successful use of the device; for example, patterns that emerge in how patients plan to store a device may not be meaningful if storage methods will not effect drug efficacy or device safety. But when trends are observed that have a potential impact on patient safety or the efficacy of the device, designers should examine the trends carefully and make sure they have enough data collected to have confidence in the results.
Refine the Design and Head Back to the Field
User research is not a “one and done” proposition. Ideally, medical device manufacturers will have several chances to collect user feedback and observations over the course of device development.
The number of rounds of user research can vary widely from project to project. In general, the more complex and novel a device is, the more rounds of user research manufacturers should plan to incorporate into their timelines and budgets.
In an Agile development model, the design team may go back to users at frequent intervals as they refine the device design. More commonly, the team plans user research at a few key points in the development cycle. These may include early formative research and evaluation of design concepts, early prototypes and final prototypes. A new set of participants should be recruited for each stage of user research to avoid the emergence of biases that can result when the same participants give input during multiple user research studies.
If time and research budgets are tight, manufacturers should put more emphasis on early research. Identifying potential problems and validating the fundamentals early in the design process can help the team reduce the likelihood that they will need to make extensive, and expensive, changes later in the design process.
However, whenever significant changes are made in the design, additional research should be completed to gain confidence in the new design. If too many design iterations are allowed to go untested, there is a greater likelihood of unwelcome surprises when it comes time to conduct final human factors validation testing prior to the regulatory body submission. An iterative approach to user research will help ensure that the final design fully meets the needs of users.
Following these guidelines will help engineers establish effective partnerships with human factors researchers. When engineers and human factors researchers work together, the result is better user research—and, ultimately, better medical devices.
About the Author
Scott Danhof is a Mechanical Engineering Research Leader in Battelle’s Consumer, Industrial and Medical Technologies group. He provides leadership for project teams for a wide range of products for government and commercial clients. For the last 20 years, much of this work has focused on medical product development, including participant/user safety, regulatory compliance, design controls and usability.