Show me the

Intermountain Healthcare Conducts Large Scale Usability Testing in the Field

Background

Intermountain HealthcareIntermountain Healthcare is a nonprofit health system based in Salt Lake City with over 26,000 employees. Serving the healthcare needs of Utah and southeastern Idaho residents, Intermountain's system comprises 21 hospitals, plus clinics, health plans and affiliated physicians. The company has more than 30 years experience in developing longitudinal medical records and software that supports outcome-driven best practices. Six of the last eight years, Intermountain has been named one of the nation's top 100 most technology savvy health systems in a study of 1,300 hospitals by the American Hospital Association's Hospitals & Health Networks magazine.

Part of Intermountain’s commitment to deploying the latest and most effective technology revolves around its expanding user experience program. The core of this program has been Intermountain’s classic two-room usability lab plus a classroom-sized “observation room” at the renowned Primary Children’s Hospital. The lab was used to examine the usability of Web sites, online content, and healthcare applications. Like many traditional labs, it once had state-of-the-art hardware but is now beginning to show its age. All of the metrics the usability team needed when it conducted a study had to be counted manually from low-resolution recordings of test sessions. The process was time-consuming, costly, and presented opportunities for varying estimations and interpretations.

Challenge

Beyond having to deal with the complexity and inconvenience of a hardware-based lab, a major hurdle the team faced was getting clinicians to the lab to conduct testing. Intermountain’s network stretches hundreds of miles across Utah and Idaho. The team needed a modern usability solution that would allow them to go to the clinicians to conduct testing. The solution would have to run on laptops, but still be able to provide all the benefits of their hardware-based lab. The team was also looking for a system that would synchronize all the usability data such as the user’s audio and video, their interaction with the applications, and all the system data.

At the same time, Intermountain and GE Healthcare had teamed up on a 5-year, multi-million dollar collaboration to test and deploy new medical software. That partnership began with the development of desktop prototypes that needed to be tested with clinicians in the Intermountain network as well as with GE’s customer base. Intermountain’s fixed usability lab simply wouldn't be practical for a usability project of this scope.

Intermountain led a trial of TechSmith’s Morae. Unlike the traditional fixed, hardware-based lab equipment, Morae has broken the mold of what user experience testing should be while at the same time knocking down many of the technical barriers that had impeded the adoption of user experience testing by many organizations.

Morae is the industry’s first all-digital user experience testing software. Consisting of three integrated components – Recorder, Observer, Manager – Morae delivers key benefits to any user experience program. Not only does it cost a fraction of a typical hardware laboratory, but it also provides superior user and system data capture, indexing, searching, and synchronization – all things that equate to more thorough, more accurate and much faster analysis.

Results

One immediate benefit the teams realized after deploying Morae was its portability, which allowed usability specialists to visit clinicians on-site where they could conduct more sessions and test more clinicians. Not only did Morae give the usability team greater capabilities, but they also realized tremendous cost savings by not having to pay for clinicians' travel expenses to Intermountain’s fixed lab. They also saved tens of thousands of dollars just on the initial cost of Morae versus trying to update or replace their old hardware equipment.

With Morae loaded on laptops, the usability teams traveled to the clinics and hospital sites where the team was able to conduct usability tests in physician’s offices and conference rooms. They were also able to test in the open clinic environments, giving developers the ability to see firsthand how the medical software supports doctors and nurses during peak workloads and through multiple interruptions.

Morae’s Observer component proved to be a real bonus because developers back in the main office in Salt Lake were able to watch usability specialists testing new applications in a far-away clinic. For example, what started with one developer, who happened to be walking by and saw the remote sessions of a clinician testing new messaging software, quickly turned into a full team of observers. Developers could immediately see obvious problems with the messaging interface and could implement changes without waiting for further analysis.

It was at this time that Intermountain and GE teamed up to begin their massive new software development partnership with GE’s usability team. After watching some of Intermountain’s tests, GE Healthcare chose to deploy Morae for its own use. In the following weeks, GE and Intermountain usability specialists and analysts traveled to clinics throughout Utah, Indiana, and Oklahoma, gathering data from about 35 clinicians representing a variety of healthcare facilities.

Morae gave Intermountain and GE Healthcare usability and design professionals the ability to capture the 360o user experience. Morae recorded the clinician’s voices, their faces, and all the interaction that happened on the computer screen as they used the medical software. And since the recording is digital, the quality is perfectly clear. Morae also recorded where a clinician clicked on a button, each page change, when and where they typed in information, and all other data ranging from the contents of dialogue boxes to error messages. Hundreds of hours of digital video were recorded using Morae.

Once all the initial testing sessions were completed, both teams convened back in Utah to compile and analyze the data. With Morae’s built-in Rich-Recording Technology (RRT), all the user and system data is immediately synchronized during the testing sessions, making it possible for the teams to later conduct extremely detailed and thorough analysis. They were able to look at time-on-task, mouse clicks, key strokes, total mistakes, and hundreds of captured comments. The initial analysis took only two days, something that would have taken weeks to process without Morae.

The tests provided important user information, much of which is still being studied as the partnership designers continue developing a new desktop based on emerging technology. The teams are also carefully studying usability of existing GE, Intermountain and IDX products to provide benchmarks for emerging new products. Morae allows the teams to measure the legacy products’ work models and compare results. In that way, much of what’s best of the older products can be implemented in the newer products.

Intermountain Finds Creative Uses for Morae

Since then, Intermountain has found other inventive ways to use Morae. With the GE purchase of IDX Systems Corporation, a leading healthcare information technology (IT) provider, there was a need to demonstrate software and architecture strategies across the worldwide GE Healthcare network. Former IDX specialists went to Salt Lake where Intermountain’s user experience team recorded their training sessions with Morae. Those sessions immediately became training videos to be distributed over the company’s intranet so everyone from Barrington, Illinois, to Bangalore, India could watch them.

Another creative use revolved around Intermountain’s developers who were having a problem with a new pilot program. Data seemed to be “disappearing” from computer records in a rare, intermittent, and unrepeatable fashion. Developers had been working for months trying to find the problem, but because it was so intermittent and unpredictable, they were stymied. The only apparent pattern was that most of the errors were coming from one clinic. So the Intermountain user experience team wired a computer in that clinic with Morae Recorder. The mysterious error occurred during the first day’s recording, and careful analysis of that recording led programmers to the exact sequence that triggered the problem.

Flexibility and portability are big values for us and so is the ability to automatically capture mouse and keyboard gestures. Morae lets us easily do metrics that just weren't practical before. It makes the quantifiable part of our work extremely reliable.

- Wendy Sudar,
Sr. Human Factors Engineer
More customer stories