Morae Usability Testing for Software and Web Sites

Home / Morae /

CUSTOMER:

Fog Creek Software

FOCUS:

Software Development

TechSmith Lends Morae to Project Aardvark to Show the Product’s Flexibility and Effectiveness When Used by Smaller Development Teams

BACKGROUND

In the summer of 2005, Fog Creek Software launched Project Aardvark with the goal of creating a new software product in just 10 weeks. The project was run entirely by summer software development interns. They were responsible for designing, programming, testing, and rolling out the final application. The product they created is Fog Creek Copilot https://www.copilot.com -- a consumer program that allows people to help friends, relatives, and customers fix computer problems by connecting to their computers over the Internet.

TechSmith was the official usability testing sponsor of Project Aardvark and provided Morae to the team at no cost.

THE CHALLENGE FACING SMALL DEVELOPMENT COMPANIES

Project Aardvark exposed the interns to many of the problems small software development companies face, including extremely short development cycles, limited resources, and budget constraints. Probably the biggest hurdle the team faced was coming up with a way to conduct usability testing on the application. Joel Spolsky, Fog Creek Software’s founder, has extensive experience with usability testing, having led efforts at organizations such as Microsoft and Juno. Fog Creek Copilot was designed to be secure, easy to use, work through any home or office firewall, and require no installation or configuration. Only having end users interact with the application would allow the Aardvark team to gain objective validation on their design and uncover any problem areas.

The Aardvark team didn’t have tens of thousands of dollars to outfit their own testing labs or the time or money to outsource the project to usability consultants. Mr. Spolsky also knew the most effective way for the Aardvark development team to discover, learn, and be convinced the application could have usability problems was to see it for themselves.

“What’s useful is getting the developers to watch the usability tests,” said Spolsky. “The recommendation of a usability consultant frequently isn’t convincing to a developer. The developer mindset is of constantly fixing bugs -- see the problem, hypothesize it, fix it. It’s not to get a business person showing documents full of facts, taking it home, and reading it, and then fixing it.”

TESTING FOG CREEK COPILOT VERSION 1 WITH MORAE

The interns and volunteers helping with the testing paired up 14 people -- one person played the “victim” and the other the “helper.” The everyday scenario the subjects were tasked with performing was to help the victim with a computer problem. The victim called the helper, who would direct the victim to www.copilot.com. The helper would then take control of the victim's computer and make the necessary changes -- tasks such as changing the victim’s Internet homepage or changing the default printer. The helper would talk on the phone at the same time, explaining his onscreen steps. Only one of the 14 subjects had ever used Fog Creek Copilot before the usability session.

With the usability testing being powered by Morae, all audio, video and users’ interactions -- such as mouse movements, typing, where they click, page changes, and much more was recorded so the developers could see and hear exactly where the users had problems. Morae’s patent-pending Rich-Recording Technology (RRT) makes this possible giving user experience professionals a unique view into the way that desktop software, websites, intranets and e-business applications are seen and experienced.

Morae also made it easy to quickly create highlight clips, like those below, to share with decision makers. If the team would have used traditional usability testing equipment it would have taken several days to synchronize all the elements and the quality wouldn’t be crystal clear.

Captured Comments video:
http://video.techsmith.com/morae/latest/demo/aardvark/enu/mor_coplt_cptrdcmmnt.html
Best Features video:
http://www.techsmith.com/videos/morae/projectaardvark/bestfeatures.html

One discovery the team made when watching the Morae video footage was that many of the participants mistyped the URL address. The team registered 20 separate domain names in order to capture customers, and by looking at data showing which URL address customers entered, the decision has already paid off. The most important discovery the team made had to do with the service’s key security feature. Mr. Spolsky describes it in his very popular Joel On Software blog:
http://www.joelonsoftware.com/articles/UsabilityTestingwithMorae.html

So far this morning we've run two usability test sessions, with great results: we've already realized that 2 out of 2 helpers were confused about how to get reconnected since the Fog Creek Copilot helper application deletes itself when you're done with it. This is a classic example of the user model not conforming to the program model ... most programs don't delete themselves! ... which is the source of virtually every usability problem. From the very first chapter of UI for Programmers:

The cardinal axiom of all user interface design:

A user interface is well-designed when the program behaves exactly how the user thought it would.

I should have known this. The program design violated a principle I wrote myself in big bold print in my own book: it didn't do what you expected. The great thing about usability tests is with a day of usability testing and handful of subjects, even if you're as senile as I am, you can find the biggest areas where you didn't realize where the program's behavior diverges from the user's expected behavior.

After a day of testing, the developers had valuable video to analyze, and a list of issues they needed to address, such as making certain links more obvious, adding prompts, improving the downloading instructions, adding more explanation to the purchasing page, and including more instructions in the program-generated email.

RESULTS:

Because of its superior functionality, greater flexibility, easier implementation, and affordable pricing, Morae is breaking down barriers that had inhibited organizations from conducting usability testing in the past. Morae can be deployed on standard desktop and laptop computers, eliminating the need to rent expensive testing space, assemble complex hardware solutions, and hire video experts to run the equipment and edit the footage.

TechSmith donated Morae to Project Aardvark, but even if Fog Creek had paid for the program, it would have represented an incredible savings over a traditional usability testing project – a little over $1,000 as compared with $100,000 – the cost of the last usability testing project Mr. Spolsky was part of at Juno. Morae was loaded on the PCs the employees use everyday. The only hardware requirement was a Logitech webcam with a built in microphone to capture the video and audio of the subjects.

While smaller software development shops face many challenges their larger counterparts don’t, Morae is proving that usability testing on the technical side is possible for everyone. Getting subjects to take part in usability tests is now the much bigger challenge.

“The TechSmith Morae software proved to be very useful,” said Intern Yaron Guez. “Seeing both users' screens on two side-by-side LCD monitors, with video and audio of both users was very cool, especially when the users were talking to each other over the phone. By the end of the day, we had learned a ton about Fog Creek Copilot. Issues that we never would have thought would be issues were unearthed, and I have a huge list of things to implement this week to make the service better.”

Since Morae was released in the fall of 2004, it has been adopted by organizations of all sizes, including eight of the 10 most popular websites in the world as ranked by Nieslen//Netrating. The world’s largest software developers such as SAP, Oracle, and Microsoft also use Morae, as do many other large organizations in industries such as healthcare, retail, financial services, and government and education.

ABOUT FOG CREEK SOFTWARE

Fog Creek Software is a New York City based developer of software tools for programmers. They make FogBugz, a Jolt-Award winning project tracking application, and Fog Creek Copilot, the easiest way to control another computer over the Internet. Fog Creek was founded in 2000 by software pundit Joel Spolsky of Joel on Software fame.

TechSmith Corporation was founded in 1987 and is located in Okemos, Michigan. TechSmith provides practical business software tools to capture and manipulate images, voice and video from Windows desktops. The company's flagship software packages, SnagIt and Camtasia Studio, enable customers to create visually enhanced content for teaching, training, Web sites, documentation and business presentations. Morae is the first and only all-digital solution for recording and analyzing human-computer interaction. TechSmith products are used by 99% of the Fortune 500. For additional information, visit www.techsmith.com.

TechSmith is a registered trademark. Morae is a trademark of TechSmith Corporation. All other trademarks and registered trademarks are the property of their respective companies.

Explore Morae