Language: English Deutsch Français 한국어 日本語

Online Travel Usability Study

ONLINE TRAVEL USABILITY STUDY

A Joint Study Conducted By:
Human Factors International
TechSmith Corporation
Michigan State University

Published: November 3, 2005

Travel Study

Description of the Study

Human Factors International, TechSmith, and Michigan State University collaborated to fund and conduct a research project to highlight at World Usability Day on November 3rd, 2005. The primary goal of this usability test was to study the relative ease of use for 3 travel sites when making holiday travel plans and to explore whether there was a difference in assessment between expert and novice users. The secondary goal of the study was to determine the degree to which remote tests are consistent with and as robust as the in-person tests. Half of the evaluations took place face-to-face at Michigan State University's Usability & Accessibility Center; the other half were done remotely using a combination of GoTo Meeting software and a conference telephone line. TechSmith's Usability Testing software called Morae was used to record and log data for both the in-person and remote tests. The following sites were tested in this study:

www.orbitz.com
www.priceline.com
www.southwest.com

These sites were selected because they are among the most common travel reservation sites and are used by large numbers of customers on a daily basis. Each site represents a different type of travel site (online travel agent, auction, airline). Participants were asked to complete the following task:

You are planning to visit friends and family for the holidays in Austin, Texas. You are on a budget and need to book the least expensive airline ticket from Detroit to Austin. You will also need a hotel. You would like to depart on December 23rd in the morning and return December 27th in the evening. Your budget for the entire trip is $800. Using the site(s), book your trip.

The study included 24 participants from a mix of age groups and educational backgrounds. For the in-person portion of the test, participants (12) were located in and around East Lansing, Michigan. For the remote portion of the tests, participants (12) were geographically dispersed throughout the continental United States. Half of the participants had made online travel reservations more than 4 times in the past year (expert group, 12) and half of the participants had never made online travel reservations (novice group, 12).

Summary Findings

Overall findings show that Orbitz.com was the easiest to use for both novice and expert users and Southwest.com was the most confusing site to both novice and expert users. Priceline was the site where people made the most errors, but still received a high percentage of people willing to recommend it to others.

Completion scores show that novice participants had a slightly lower completion average than expert participants. It was not as large as was expected, however. Southwest.com had the lowest completion scores for both expert and novice. In this case, ease of use (one aspect of usability), was lacking on Southwest.com so it was more difficult to use even for people with experience making flight reservations.

The overall finding regarding remote vs. in-person tests was that the remote tests were as robust and consistent with in-person tests as far as the data we were able to collect, but were significantly more difficult to administer.

Findings

Overall

Orbitz.com was the easiest site to use. 63% of participants ranked it easiest to use.
95% of participants would recommend Orbitz.com to other people
Southwest.com was the hardest to use. 67% of participants ranked it the hardest to use
62.5% of participants would NOT recommend Southwest.com to other people
The most errors were made using Priceline.com. This was due to the fact that the site switches the destination and origination fields when changing from a regular flight search to a package search so several participants made hotel reservations in the wrong city
79.2% would recommend Priceline.com to other people
There was a very slight difference in findings between people who have made travel reservations online in the past versus people who have never made travel reservations online before. As initially hypothesized, there was a slightly lower completion score for those who were novices than for those who were experts.
Completion scores on Southwest.com were lowest for both novice and expert.
The biggest difference in completion scores was on Southwest.com where the mean score for experts was 2.08 and the mean score for novices was 1.67. (3.0 = completed without difficulty, 2.0 = completed with some difficulty, 1.0 = did not complete)
The rankings novice users reported (how straightforward or confusing they thought the sites were on a scale of 1 - very straight forward to 7 - very confusing) had the largest drop from initial impression to after actual for both Priceline and Orbitz. Both sites lost a whole point. The ranking for Southwest both before and after use did not change as dramatically, but also went down. Expert users ranking for this scale remained more constant, with the exception of Priceline where it dropped over a point.

Remote Testing vs. In-person Testing

Description

Per this study's "split" in-person/remote methodology, half of the tests were done at the Usability & Accessibility Center, within University Outreach and Engagement at Michigan State University and the other half were done remotely with the use of GoTo Meeting's online meeting tool and a bridge phone line.

The in-person sessions were in a controlled environment. All test workstations were located in a quiet room, with a desktop computer and a T1 LAN connection. Participants were greeted in the lobby of the Usability and Accessibility Center, where they signed their consent forms, completed their pre-test questionnaire and enjoyed refreshments. Once their paperwork was complete, they were escorted into the testing room where they met the test facilitator and note taker. TechSmith's Morae was used to record and log the test data. Once the tasks had been completed, participants were given a paper post-test questionnaire. Once that was complete, participants verified their email address and their incentives were emailed to them.

In contrast, for the remote tests, participants were in their home or work environment. Katie Hill, the test facilitator, was located in Austin, Texas. Barb Hernandez, who was logging data using Morae, was at TechSmith headquarters in Okemos, Michigan. Prior to the testing time, participants were sent an email detailing the test procedure. This email contained a link to the GoTo Meeting site, as well as an 800 number to call to speak with the test facilitator. There was no way to control their environment, so a varied range of setups existed. For example, one participant was on a dial-up connection, one participant mistakenly called from his car phone, and one participant was an Apple user (which GoTo Meeting does not yet support, requiring us to use a different tool). These and other situations caused a variety of technical challenges for the test administrators. Once connected to GoTo Meeting, participants were given control of the browser at TechSmith through the GoTo Meeting application, and the participant, facilitator and data logger could all witness the actions and navigation of the participants.

Findings of the in-person vs. remote testing

The overall finding was that remote tests were as robust and consistent with in-person tests as far as the data was concerned, however there were many more technical and procedural challenges conducting the remote test than there were using the usability lab. Both in-person and remote tests have their pros and cons. They are highlighted below.

Pros of In-person Testing

Ability to control the environment
Easier on the facilitator and note takers logistically
Ability for observers to come to the lab and understand the process of usability
Overall stability of the equipment
Ability to control the technology (i.e. internet speed, browser used, screen resolution, operating system)
Easier to read the participants and therefore probe on facial expressions and body language

Cons of In-person Testing

Cost of travel
Cost of renting the facility
Participants have to travel, potentially increasing odds of no-shows
Potentially more stressful for the participant due to one-on-one with facilitator
Limited participant pool to geography of testing facility

Pros of Remote Testing

Ability to increase demographic pool
No cost of travel
No cost of renting testing facility
Ability to understand how users behave in their own environment
More natural environment, so you get a better understanding of outside factors in the user experience
Participants do not have to travel, potentially decreasing odds of no-shows
Morae works the same, so data integrity is preserved
Online data collection is easier to tally and saves time on data analysis

Cons of Remote Testing

Technically much more difficult to conduct the test
Inability to control user environment (speed of connection, operating system, ability to talk on the phone and be online simultaneously)
Facilitator must be very acquainted with testing tools and have flexibility to troubleshoot technical issues, including those on the participants end (screen resolution, etc)
Remote testing tools get in the way of the actual experience (connection issues, UI of the tool used, etc)
Inability to read participant's body language and facial expressions
Much more emotionally/physically taxing on facilitator/ notetaker
Environmental issues (call waiting, family members interrupting, babies crying) can't be controlled, so task times can vary
In general, test take longer because of technical issues and familiarizing participant with this process as well as the Usability Testing process
More difficult to get release form signed
Harder on participant as they have to juggle the phone, keyboard and mouse

Summary

In summary, orbitz.com was rated the easiest site to use and southwest.com was the hardest to use. Participants were most likely to make errors on Priceline.com, but still ranked it higher than southwest.com. There was only a slight difference in completion rates between novice users and expert users, with novice users having a slightly lower completion rate.

As compared to using a usability lab, conducting a usability test remotely provides the same rich and valuable data, however it is more difficult to administer and takes more knowledge and effort on the part of the test facilitator and note takers.

Contact person for questions about the test from HFI:
Katie Hill
+1.512.276.5355
katie.hill@humanfactors.com