SAP Provides On-Site Usability Testing and Product Design Evaluation

Background

SAPSince 1990, the America’s SAP Users’ Group (ASUG) has worked to continuously educate its members, facilitate networking among colleagues and SAP representatives, and influence future SAP product releases and direction. Today, ASUG ranks as the largest independent, not-for-profit organization of SAP customer companies and eligible partner vendors in the world with more than 1,700 corporate and 45,000 individual members.

Every year, thousands of ASUG members converge at its annual conference to meet with friends, attend educational sessions, provide feedback to SAP representatives about products and programs, and learn about new products SAP will be releasing.

How to Conduct Usability Testing at the ASUG Conference

The annual ASUG conference is an enormously user-rich environment for SAP’s User Experience (Ux) team to conduct research. Since 1999, the Ux team has held usability activities to give ASUG members the opportunity to test new and existing solutions and provide on-the-spot feedback to SAP developers, product-management specialists, and usability experts. The objective of these activities is to make SAP solutions easy to use, easy to learn, and more efficient.

A daunting challenge the SAP Ux team faces is efficiently organizing and managing the large number of usability sessions that take place at the ASUG conference. They must capture and organize the vast amount of usability assets and user feedback collected during each session.

In 2005, the SAP Ux team began exploring new ways to combat this situation so they could conduct more thorough and more accurate analysis during the conference and in post-analysis. The team needed a solution that could capture the total user experience – the user’s interaction with the application, their verbal comments, and all the notes the SAP moderators take during each session. Moreover, the solution would have to be highly reliable given the three-day window for testing.

The usability solution would also have to be easy to implement, and support the varying types of technical expertise on the Ux team, which range from usability engineering, cognitive psychology, graphic design, interaction design, and computer science.

SAP Runs Morae

SAP chose to run Morae by TechSmith on test stations at the ASUG conference to conduct usability testing.

Morae provides usability lab functionality in an easy-to-use software solution that eliminates the cost and complexity of lab-based research, and also makes it possible to do in-the-field research. Morae is innovative, practical, and affordable -- a combination that has quickly made it the gold standard in the industry, making usability testing accessible for the first time to organizations of all sizes.

What makes Morae unique is its Rich Recording Technology™ (RRT). With this new technology, SAP Ux team members are able to focus on conducting and moderating the usability sessions without having to worry about missing any of the interaction. Morae’s RRT records exactly what is seen on the screen in addition to a synchronized chronicle of system events occurring on each of the SAP test computers behind the scenes.

For example, if a test participant was tasked with navigating a new SAP interface, not only would Morae record exactly where the participant moved the mouse, it would also record the user and their words. It would record where the participant clicked, when a page changed, if a dialogue box appeared, every keystroke, and much more. All this information is automatically synchronized via RRT, indexed, and made searchable in digital format, saving countless hours during post-analysis.

The Testing Environment

One day prior to the beginning of the conference, TechSmith representatives gave a training session to the SAP Ux team where they covered Morae’s components, setup, and basic features that would be used during the usability sessions. For many members of the Ux team, this would be their first time using Morae. The typical usability session at the ASUG conference lasts about one hour. Usually, the participant is asked to complete a set of tasks to help identify the particular SAP application’s strengths and weaknesses. Typically, a member of the SAP application development team and a moderator sit with the participant as they go through the application being tested in order to record their feedback.

The Morae Recorder component is installed and pre-configured on the test participant’s machine so that when it’s time to begin the tasks, the moderator only needs to click the record button. There is no noticeable system or application degradation so the test participant isn’t distracted. At the same time, Morae’s Observer component is installed on the SAP observer’s work station where they see exactly what the participant is doing. Through Morae Observer, the SAP observer also has the option of adding markers and notes at key points that are synchronized with the rest of the usability data.

Results

Over the course of three days, the Ux team conducted usability testing on 22 separate applications with 219 ASUG participants at 15 stations. In the process, Morae captured more than a thousand hours of usability data, all crystal clear, indexed, and searchable. Not only does this give the specific Ux team members assigned to that particular application the ability to conduct fast analysis later, but it means that digitally based content can be put online so other Ux team members worldwide can access it.

Another aspect that greatly increases post-analysis efficiency is the ability for the SAP observer to add markers during the tests using Morae Observer. In advance, the Ux team members identified alpha markers so whenever something deemed “markable” happens, the moderator can add the appropriate flag.

During post-analysis, the Ux team was able to go directly to those markers of interest. So if “E” meant “Error,” then Morae makes it possible to search just for the “E” marker. Other times, usability professionals will mark certain aspects they know the development team would be especially interested in seeing. This makes it easy to clip just that segment and create a highlight movie using Morae’s Presenter feature. This way, the SAP decision makers actually see, hear, and experience for themselves exactly what the participant experienced during the live test.

By deploying Morae, SAP was able to dramatically reduce the overall burden of conducting usability tests at the ASUG conference, not only at setup, but also during the testing and data logging stages. With Morae, the testing sessions ran smoothly and with no downtime. Moving forward, Morae will continue to be a key solution and part of SAP’s everyday user experience program.

Products Featured

Morae

Conduct on-site usability testing and product design evaluation with Morae.

As a first time user of Morae, I was surprised how easy it was to work with a product that does so much. I didn’t have to think about what Morae was doing and was able to focus entirely on the participant using the application. As a developer being able to conduct these usability tests with customers on-site is an enormous benefit in helping SAP create better products.

- Chiao Lu,
User Experience Designer, SAP
More customer stories