Show me the

Optimal Usability Delivers Hard Data to Back Up Recommendations

User Experience Challenge

optimal usability logoIn July 2004, Wellington, New Zealand-based Optimal Usability, Ltd. was asked to conduct an expert review and analysis of the University of Canterbury's newly redesigned Web site, and provide recommendations for improving its usability. The school, located in Christchurch, has 14,000 students and 1,400 faculty and staff members. The Web site is the school's primary point of contact with prospective students. The goal of the redesign was to help the school attract new students and, ultimately, increase enrollment.

The Scope

  • 15 participants
  •  2 testing locations
  • A cross-section of 150,000 Web pages and 120 sub-sites
  • 18 hours of usability testing

Background

An American study conducted in 2001 by Poock, M. & Lefond, D., suggested that high school students are becoming much more reliant on the Web in their decision making about colleges. The study found that 40 percent of university-bound students accessed a university's site before applying, and 83 percent before enrolling in classes.

In late August and early September 2004, Optimal Usability's experienced team of consultants performed a usability study on the University of Canterbury's Web site (www.canterbury.ac.nz). The main goal of the study was to provide an independent and objective analysis of the site to confirm or dispel assumptions that were made as the site was undergoing a redesign. The school also wanted to identify the site's strengths and weaknesses, and asked Optimal Usability to provide recommendations for ways to improve the site during the next phase of the redesign.

The study was conducted at two locations; one in Auckland, the other in Christchurch. Five participants were staff and 10 participants were high schoolers who planned to attend university. The task given to the high schoolers was to find the typical information that students would want if they were considering attending the university. Examples include a list of compulsory courses for specific degrees, information about orientation programs, information about financial aid and scholarships, and details about student accommodations.

The staff participants were asked to use the Web site for tasks that they do on a daily basis. This includes accessing the online library catalogue, looking up colleague contact information, searching for general information, and reading staff news.

Testing with Morae

At both the Auckland and Christchurch locations Morae was installed on a desktop computer to record all application and system data in real time. In addition, the consultants placed a mini camcorder and Web cam on top of the monitor to record the user and the audio.

Morae's Observer was setup on a laptop in another room so the consultants could observe what was happening on the user's desktop without impacting the test. Using two cameras to hear and see the user is no longer necessary. As of October 2004, Morae automatically synchronizes video and voice (as well as application and system data), and supports multiple observers regardless of location. Each observer can also add their own markers and notes, which will be saved with the recorded file, and each observer can save a copy of the session to their own desktop's hard drive.

Before the availability of Morae, which utilizes software and a Webcam, Optimal Usability used to point a video camera at a computer screen to conduct usability testing. At times, the firm even attached a mirror to the side of the monitor so consultants could record facial expressions, audio, and the computer screen at the same time. Beyond the obvious physical limitations and poor quality, this approach resulted in a highly unnatural environment for participants. The team also was forced to carry around their own LCD monitors because when they fixed the video camera on a CRT monitor, it gave the "strobe effect" where lines run across the screen.

With Morae, this arcane approach is no longer necessary. Instead, Optimal Usability analysts have Morae recording the session before the participant even sits down at the computer. It starts with the analysts creating an authentic and natural environment by adjusting the screen settings to fit what a user has on her or his home computer. Because Morae use TechSmith's lossless codec, the quality of the screen recordings are of the highest quality. This guarantees that usability professionals have the clearest view of all the information (especially hard-to-read text fonts) during analysis, and makes delivering professional digital video highlights for clients effortless.

From the Observer, consultants set markers during the live sessions that enabled them to quickly find key points later on during the analysis phase. By clicking on the marker flag analysts are taken to the exact point in time where an event occurred. It was decided in advance the team would use three primary alpha makers (F: Frustration, S: Satisfaction and I: Interesting/other) to identify points of interest so they could be found easily later on during analysis. In the past, this exercise could only be accomplished by sitting in front of a VCR for hours fast-forwarding and rewinding the tape to find critical points.

After each participant finished, their session was saved to a portable hard drive. The average file size for each participant session was 500 MB and transferring the Morae file to the hard drive only takes a few minutes, which gave analysts time to conduct exit interviews before the next participant was ready to start. In addition to portable hard drives, sessions can be saved to Flash thumb drives, CD-ROMs or DVDs, so analysis can be conducted on any computer. If this had been an in-house or corporate usability setting, the recording file would be saved to a network drive so it would be accessible to multiple analysts at the same time.

Gaining the Greatest Insights With Morae

After completing four days of testing, the team returned to Optimal Usabilty's headquarters in Wellington and began analyzing each session. Morae's Manager component automatically synchronizes all the audio, video and screen recordings, in addition to the Observer markers.

With all the information synchronized and indexed, the Optimal Usability team can see and hear the user and the application just like they did during the live testing. However, they can now perform searches based on events such as mouse clicks, key strokes and Web page changes, or even search by text. For example, if an analyst wanted to know how many times Participant 2 chose to "right click", or type "chemistry" in the search field, or click on the Scholarship page, the data is available instantly.

One perception that was quickly dispelled was the notion most people use the Web site's search function. In fact, very few did. Because Morae indexes and makes all the data searchable, the team was able to prove that people who visit the university's site don't use the search. For example, of the 15 participants and 15 hours of testing, only 52 searches were conducted. Only a third of those were successful and they were usually conducted by staff.

Upon further analysis, the team was also able to detect what exactly the participants were searching for because the participant either said it aloud or Morae recorded every letter and word they typed in a search field. It was also easy to point out that the advanced search option was inconvenient to use because it presented participants with a list of 92 items, but only displayed five at a time.

Another observation the team suspected from its primary review of the site but needed to prove was that users were floating around the site with no real direction. By being able to quantify page transitions with Morae (such as Participant 1 who changed pages every seven seconds, and even the average participant changed pages every 17 seconds), the team was able to prove the behavior was prevalent. More importantly, this allowed the team to make recommendations to correct this problem.

Morae Delivers Hard Data and Video Highlights for Emphasis

Trent Mankelow, director of Optimal Usability, has used Morae on his team's last five projects, which have included usability testing on Web sites, intranets and extranets. After analysis is completed, he and his team review and watch the participants again to create a highlight video for the client so clients can see for themselves what users are saying and how they are interacting with Web sites and applications.

Morae's Presenter feature allows Trent and his team create the highlight video. With drag-n-drop ease, he and other analysts can import video segments they create during analysis into the highlight video. The video is complete with Picture-in-Picture so the clients can see, hear and experience how users experienced their Web sites. Optimal Usability delivered a 50-minute DVD highlight video along with the final report to the university. Morae's all-digital approach made this formerly time-consuming and expensive process a snap.

"Creating a highlight video for the client is an important and highly effective tool to communicate the results to decision makers who are interested in the big picture," said Mankelow. "Developers and designers may be more interested in the highly-detailed report and recommendations. With Morae, we're able to meet both these requirements efficiently, accurately and with great persuasiveness."

In one instance, the team wanted to emphasize that participants were having a difficult time completing simple tasks. For example, it took one participant nine minutes to complete the task, "When does enrollment start?" Typically, the team organizes the highlight videos to flow along with the report so if a particular quote or issue is mentioned in the report there is accompanying video. Additionally, this highlight video can also be presented to a group during the final presentation.

Results

The most obvious benefit of Morae is its all-digital approach to recording, organizing and making the copious amounts of data easily accessible and searchable. The end result is more in-depth analysis, better communication with clients, and an immediate return on investment for consultants. For Optimal Usability, the benefits of using Morae are mostly internal because it allows them to reach depths of analysis not possible in the past, and it increases productivity by an order of magnitude, which contributes to greater profitability per project, a more scalable organization, and happier clients.

Over the next year, Optimal Usability plans on using Morae in more than 15 usability studies and expects to save thousands of dollars in hardware and traditional, cumbersome hardware-based solutions. The firm also expects to increase its competitiveness significantly when bidding for new projects. The firm saves on average four hours per project in editing time, and they have already realized the benefit, when given permission, of being able to give prospective clients highlight DVDs of other client projects. This has helped Optimal Usability win numerous new clients, while increasing credibility and obtaining a high level of professionalism.

In addition to education clients, the firm has used Morae to conduct usability tests on Web sites for government clients such as the Department of Conservation (www.doc.govt.nz) and private companies, such as the Accidental Compensation Corporation (www.acc.co.nz). Morae is now the team's go-to tool for almost every project, especially larger projects. The next project for which Optimal Usability will use Morae is a Web site analysis for the country's largest non-profit organization, the Institute of Chartered Accountants.

About Optimal Usability

Optimal Usability is a user focused Internet consulting and services firm. The company evolved from conversations between a software engineer and an industrial designer. The directors' mutual passion for easy to use technology and their dissatisfaction with the usability of the web drove them to action. Optimal Usability was incorporated in early 2003 to provide web usability services to medium to large sized businesses in New Zealand and Australia.

Before Morae, it was impossible to collect, analyze and share with clients this enormous amount of information. Having this capability has a profound impact on our clients and they are thrilled with what we can do for them and show them.

- Trent Mankelow,
Director, Optimal Usability

It’s fascinating to see behaviors that you just can't unlock any other way. That is definitely a huge advantage of using Morae. Our recommendations have always been exact, but with the hard data from Morae to back it up, it makes the recommendations rock solid!

- Trent Mankelow,
Director, Optimal Usability