Our study objective was to evaluate the seamlessness of the site’s common task flows and discover any difficulties or areas of confusion for the users. We conducted usability tests with eight participants to uncover the following key findings:
There are semantic and organizational problems with how customers are able to purchase tickets for passengers with disabilities.
The ticketing page has confusing elements such as unclear icons and layout.
Users encounter frustration regarding Julie, the search assistant.
This study was done in collaboration with Connor Garrett (x2@uw.edu), Reine Abubakar (reineka@uw.edu), and Ruby Davis (rkdavis@uw.edu) for HCDE 417
Read the full report here or read on for an edited version
Amtrak is the major consumer rail service in the United States, serving approximately 500 destinations and 85,000 passengers daily. For this project, a usability study was conducted on Amtrak’s flagship website, amtrak.com. While this study did not begin with an initial goal to focus resources on a specific feature, exploration of the current version of the Amtrak website led to the following overarching goals/research questions:
How can the overall user experience of the Amtrak website be improved to make it more usable?
What areas of the current Amtrak website are the biggest problem points?
Is the current version of the website accessible to all users?
Are users able to purchase tickets without assistance?
Are users able to find important information that may be hidden a few pages deep on the website?
Methods
The following sections detail how the usability study was conducted and the participants involved.
Participants
Using a screener questionnaire the research team reached out to potential participants. Out of the twenty-seven respondents, eight people qualified to be participants in the usability study. Our qualifications for study entry are as follows:
Participants should be between 18 and 70 years old.
We would like to recruit a proportion of 50% men, 50% women, but this can be adjusted based on available participants.
Participants should have traveled at least twice in the last year.
Participants should not use a travel agent, work in the travel industry, or work in UX/UI.
Participants should be Novice users, defined as having used the Amtrak site less than 5 times in the past year.
Participants need to be able to come to campus for the study and provide contact information.
The eight participants we worked with were between the ages of 20 and 22, with novice experience with the Amtrak site and no bias. Bias represents respondents who use a travel agent, work in the travel industry, or work in UX/UI.
Study Environment
The study sessions were conducted in HCDE conference rooms or lab spaces in Sieg Hall on the University of Washington campus.
Study Procedure
Prior to the participant arriving, the Moderator and the Notetaker would set up two laptops as the moderator and participant in UserZoom. When the participant arrived, they were welcomed. After they were introduced to the study and how the session would proceed, they were given a consent agreement and asked to sign it. Some instructions given with regards to the procedure of the study were “Do tasks as you would do anywhere”, “You are testing the product, you are not being tested”, “Provide both positive feedback and constructive criticism”, “Read tasks out loud before you begin”, and “Think aloud while you complete the tasks.” After they were prepared to begin the study, the participant was verbally asked some initial background questions (Appendix III) to gauge their familiarity with the site and travel more generally. At this point in the study, the participants were asked to complete the given tasks (listed below and in Appendix IV) and then respond to post-task questions within the task list. After the tasks were complete, the participants were verbally asked post-study questions about how they felt about the site and their experience of it.
Study Roles
Each member of the research team both ran study sessions as a Moderator and observed as a Note-taker. This setup was decided upon to maximize the detail of the notes and the attention given to the participant as they completed tasks.
Tasks
Task 1:
One Way Ticket w/ Passenger Specifications
Find a ticket between two cities for a given passenger specification. These specifications covered factors such as passengers with a disability, passengers with a military discount, and passengers traveling with a bike. Intended to test the main ticketing system and the different options available with it.
Task 2:
Multi City Ticket
Plan a trip between the set of given cities on the given days. Intended to test the difference in usability between the main ticketing system and the multi-city ticketing system.
Task 3:
Search Engine / Julie Chatbot
Find specific information that someone might research prior to traveling with Amtrak. Intended to test users opinion on the search engine chatbot and the information architecture of the site.
Task 4:
Train Tracking
Track a specific train and determine if it will arrive at a station as scheduled. Intended to determine the preference of users between the two train tracking systems, as well as both systems ease of use and discoverability.
Metrics
Several different types of data and metrics were collected during the actual conducting of the study. Before heading to the website, each participant was asked a series of pre-test questions to help further establish their level of experience and to keep an eye out for any variables that may be confounding. This pre-test questionnaire was conducted verbally. A similar line of questioning was conducted at the end of the tests. The post-test questions were used to learn more about the user’s overall opinions of the site and to catch feelings that might not be brought out by the mid-test conversations and post-task surveys. At the end of each task, users were asked a consistent series of questions.
Results
To process the data that was gathered during our studies, several methods were used. We began by creating affinity diagrams using issues that we noted during the sessions, grouping those issues by section and overall root cause. These diagrams were used to help focus our resources for the next steps on the most important and solvable issues. These results were also used to help classify the severity of the issues.
Below are the results of the usability tests and the severity rankings of the issues discovered. First, an overview of the key findings will be shared, ordered by severity. Then the severity criteria and the issues discovered will be presented with evidence and recommendation for improvement. These issues are presented in the following categories: purchasing tickets, accessible tickets, search / chat bot issues, tracking system issues, and miscellaneous.
Severity Rating Definitions
Severity 1: An issue that blocks a substantial number of users from using the application. E.g., Users are unable to start a game. Nearly impossible task during the initial experience.
Severity 2: An issue that blocks a substantial number of users from making use of an important feature or accomplishing an important task. E.g., Unclear solution. Undiscoverable feature.
Severity 3: An issue that hinders some users from accomplishing a task or goal. E.g., Inconsistent or misleading text. Users don't understand a non-essential aspect of the application or feature. Delays.
Severity 4: An issue that causes some users to become mildly frustrated or express minor complaints about a feature element. E.g., Suggested Feature. Minor or infrequent complaint.
Key Findings
Severity 1:
No options for passengers with disabilities on the multi-city tickets
Severity 2:
Passengers with disabilities as a ‘Discount’ passenger option
Comparing ticket prices and information while purchasing tickets
Determining multi-city trip prices while purchasing tickets
Multiple Train Tracking Systems
Severity 3:
Finding seating charts and other information while purchasing tickets
Adding tickets to cart while purchasing tickets
The Julie chatbot and search engine
Labeling of Stations
Severity 4:
Determining whether a route is a bus/train route while purchasing multi-city tickets
All Findings
The following are the issues found during this study as well as the evidence from our participants and recommendations for Amtrak to make to their site.
Purchasing Tickets
Many areas of ticket selection are unclear or challenging to process. This issue applies to both one-way/round trips and multi-city itineraries. Every participant had at least one issue with the ticket selection process. Some of the issues we found were as follows:
Add to Cart
Figure 2. Screenshot of ticket selection page with the green “Add to Cart” button displayed on the left.
Severity Ranking: 3
Evidence
Three of our participants encountered this issue, and while they were able to find the button eventually, it caused significant delays. Participants continued to scroll up and down the screen while clicking around to search for ways to add the given tickets to their cart. In one instance, a participant took four minutes and forty-eight seconds to find the add-to-cart button after selecting a train.
Recommendation
Place the ‘Add to Cart’ button at a traditional location for eCommerce websites: on the right side of the item (in this case a ticket) and in a section separated from the ticket information. This increases scannability for the overall layout of the page and increases the visibility of the ‘Add to Cart’ button. The current checkout process also only allows for the purchase of tickets from one route at a time; users cannot add multiple items to their cart. We recommend changing the name of this button to a phrase that reflects this fact, such as “select” to indicate that you may only select ticket(s) for one train at a time.
Seating Chart and Other Information
The ticket selections did not provide specific information that the participants were curious about such as seating charts. In the case for multi-way trips, the cost of the ticket was not displayed and participants wanted to learn more about their trip segments.
Severity Ranking: 3
Evidence
Five of our participants were searching for further information including train seating charts and ticket costs. This applied to both one-way and multi-way ticket purchases, and participants were unsure of where to search for further information. The term “Reserved” for each ticket led our participants to believe that there would be assigned seating, and they scrolled down the screen and clicked on icons to determine whether or not they would reveal further information about the train ride/route.
Recommendation
We recommend adding a seating chart or more detail about assigned seats. Tickets should list whether seating charts are available, and there should be visible links in case the user wants to learn more information about the type of train that they are looking at.
Page Layout/Lack of Comparisons
The ticket selections are listed with little visual informational hierarchy, making it difficult to compare prices between tickets. While browsing the list of available tickets for a given route, tickets are listed in a horizontal manner that makes it challenging to compare them while scrolling.
Severity Ranking: 2
Evidence
As users attempt to find the most appropriate ticket for their given scenario and route, they encounter a page where tickets are contained within a window within the larger page. The smaller window is scrollable, and two of our eight participants complained of this double scrolling, claiming that it was “annoying” and hindered their ability to compare tickets with one another. Additionally, the application of filters did not help as presentation of information on the page itself was still confusing after being filtered.
Recommendation
We recommend re-structuring the layout of the ticket purchasing page. The tickets should take up the full span of the page (to prevent the double scrolling) and should have clear filters at the top of the page distinct from the list of tickets. The information could be organized vertically per ticket such that similar information between tickets are on the same row and are more easily comparable.
Buses
Multi-way train routes suggest the times and routes that best fit the users’ time and location preferences. However, with transfers between trains, participants did not find it clear that a few of the transfers would actually be taken on a bus as opposed to train.
Severity Ranking: 4
Evidence
All of our participants, while asked to plan a multi-way trip, were able to complete the task and select appropriate routes for a multi-city trip. However, when asked whether they were aware of the bus transfers, some of our users said that they were not aware that the whole trip would not be conducted via train.
Recommendation
We recommend listing each trip segment with a larger icon or a clear indicator/word that indicates the mode of transportation. As it currently stands, there is a small icon that is nearly indistinguishable between trains and buses. Whether it is by bus or train, users should be able to clearly differentiate between the two available modes of transport on the site.
Multi-City Pricing
The ticket selections did not provide specific information that the participants were curious about such as ticket pricing. While constructing the various segments of the larger trip, there are no updates as to how much the trip would cost depending on the method of transportation or time the user chooses.
Severity Ranking: 2
Evidence
When asked to purchase tickets for a multi-city trip, five of our participants were searching for the pricing of each trip segment and of the trip overall. For each segment/section where the user travels from one city to another, the overall route can be updated as they select a ticket, but they are not able to see how each choice influences the overall cost of the trip. Two users noted that they would like to see the overall cost.
Recommendation
We recommend that the page have a running cost of how much the overall trip would cost. With each selection of a segment, the cost would update to reflect the price. Optionally, as users scroll through the list of possible times and methods of travel, there could be a column for costs that display the amount it would add to the overall trip.
Style Inconsistencies
The multi-city planning and booking tool looks different from the one-way selector such that it feels like a different website. This sentiment was echoed by four of the eight participants.
Severity Ranking: 4
Evidence
The disparity between the multi-city and single-city booking pages were stark, and it may have delayed our users’ tasks for booking their trip. Four out of eight of our users expressed that the page layout was unexpected, and all of our users spent additional time learning how to book the segments of their trip as it was not the same as booking a single-city trip.
Recommendation
A consistent layout for single-city and multi-city ticket purchasing will reduce the learning curve required for users. If a user is familiar with the task of booking a single-city ticket, they should be able to intuitively and easily book a multi-city ticket based on the knowledge of booking a simpler trip. This will prevent delays in completing the task, and it will prevent any additional confusion regarding how to plan trips.
Accessible Tickets
Passenger with Disability ‘Discount”
Participants had difficulty accessing Amtrak’s robust accessible ticket options because they could not locate the “Passenger with Disability” selection, which is nested in the “Discounts” drop-down menu. The categorization of accessibility options as a “discount” was also deemed not politically correct.
Figure 3. Screenshot of the Amtrak homepage. On the right, users may indicate the number of travelers and apply any discounts to their purchase. For passengers with disabilities, it is not immediately obvious how to indicate their needs.
Severity Ranking: 2
Evidence
Two out of five who booked tickets for passengers with disabilities could not locate the “passenger with disability” option at all, which had major consequences later on in the ticket purchasing process. Accessibility options are only available when this “discount” is selected, and thus users were prevented from viewing these necessary options. The three users who were able to find the “passenger with disability” selection still had difficulty locating this option. In addition, one participant, who was a person with a disability, was unhappy with their accessibility needs being trivialized as a “discount”.
Figure 4. Screenshot of the discounts drop-down menu, where the “Passenger with Disability” option is nested.
Recommendation
Consider separating accessibility options from discounts in the popup menu. As the selection of “passenger with disability” triggers more ticket options than just a discounted price, it may be more appropriate to nest these options elsewhere. One possible solution exists on the Greyhound ticket purchasing interface, where a separate checkbox has been added to allow wheelchair-accessible ticket purchasing separately from the discount drop-down menu.
Figure 5. Screenshot of a portion of the ticket purchasing interface on Greyhound.com. Instead of nesting accessibility options under “discounts”, a checkbox allows users to indicate if they will be travelling in a wheelchair.
Muli-City + Passenger with Disability was not an option
Participants were unable to book tickets for passengers with disabilities when using the Multi-City ticket booking option.
Severity Ranking: 1
Evidence
We asked participants to complete the tasks as they would if they were at home. As one of our participants uses a motorized wheelchair, they attempted to book accessible tickets for multi-city travel. At no point in the process were they able to locate a place to indicate their needs. The “Passenger with Disability” option was no longer listed in the discount drop-down menu, and thus none of the necessary selections were able to be made. Our participant did not feel comfortable with multi-city ticket booking because of this issue and indicated that if this problem were not resolved, they would not be inclined to use Amtrak’s services at all.
Recommendation
Accessibility options must be integrated into multi-city ticket booking. Consider including a selection option early on, such as a check box, which allows passengers to indicate at the beginning of the process whether they will need special accommodations. This will provide necessary accessibility options and reassure passengers that their needs will be met before they venture too far into the lengthy booking process.
Search Engine AKA Julie
Figure 6. Screenshot of the Julie chat box that appears when users attempt to use the search feature.
Severity Ranking: 3
Evidence
All six users who came into contact with Julie expressed dissatisfaction and/or frustration with her presence. No users expressed any satisfaction with Julie, other than the fact that the search engine itself often brought them to the correct page. Some of the main complaints against Julie involve the surprise of her appearing (participants often didn’t expect a chat window popup), the fact that the popup can hide important information on the page, and Julie’s tendency to drag users to the bottom of the page that they’re viewing.
Recommendation
Consider removing the “virtual assistant”/Julie feature of the search engine. Place more of a focus on the search engine’s ability to bring users to the correct page, rather than Julie’s ability to explain the page to them.
Tracking
There are two different systems for tracking train status on the Amtrak website. The second tracker is the Train Tracker Map which is accessed through a button at the bottom of the home page. This system is opened in a new tab and can track trains by searching for stations/train numbers or by clicking on stations and trains on the map.
Train Status
The first tracking system is the Train Status system, it is accessed through the header bar of the Amtrak site. It can track either between stations or with a train number and it provides the expected time of arrival.
Figure 7. Screenshot of the Schedule Train Tracker.
Severity Ranking: 3
Evidence
Three out of eight participants had to ask for assistance and be given a hint as to where the train status button was.
Recommendation
This research team suggests making the button stand out more from the surrounding bar. This would draw more attention to the button and make it easier for users to find.
Map Tracker
The second tracker is the Train Tracker Map which is accessed through a button at the bottom of the home page. This system is opened in a new tab and can track trains by searching for stations/train numbers or by clicking on stations and trains on the map.
Figure 8. Screenshot of the Map Train Tracker.
Severity Ranking: 2
Evidence
Five out of eight participants were confused by how to use the map tracking system and not confident in the results they got from it. One particularly confusing part of the map tracker to participants was which station you were supposed to enter into the search bar. In addition, two out of eight participants expressed that they did not expect to find map when they clicked this train tracking method. One participant also had difficulty navigating back to the main Amtrak site after opening the map tracker.
Recommendation
Consider either removing the map feature entirely or incorporating it into the main site and redesigning it. This could possibly be done by combining it with with the Train Status feature.
Informational Pop Ups
Throughout the site there are a number of pop ups that provide information about a wide variety of different things on the Amtrak site. There is not a clear consistency in the style of pop up, or in when pop ups are expected.
Figure 9. Screenshots of informational sections for different ticket options and dining.
Severity Ranking: 3
Evidence
Four out of eight participants found the informational pop-ups confusing or annoying. Many of those who did not comment on them did not click on the links that trigger them.
Recommendations
This team suggests that the site incorporate more of the information into the main page instead of in full screen pop ups, this can be done by replacing the single full screen pop up with individual drop downs. Additionally the style of pop-up should be made consistent throughout the site
Stations
The Amtrak site gives users station suggestions as they type out city names.
Figure 10. Screenshot of auto-complete menu when “Portland” is typed.
Severity Rating: 3
Evidence
Every one of the eight participants were confused about which station was the correct one for Portland, a city with multiple stations, and four had to be given assistance in determining the correct station. Those who had difficulty finding the correct stations frequently received error messages.
Recommendation
This team recommends clearer demarcation of the locations/types of stations. Providing information about what types of services are available from a station would help users narrow their choices. A user might not already know that greyhounds are a bus system or the different locations of stations within a single city.
Conclusion
Next steps for improving Amtrak’s usability would be to begin rolling out some of the new changes suggested in this report. Additional usability testing could help determine if these changes successfully address the issues we discovered. Preferably these tests would include a wider age range than was available for this user test. Including more participants with disabilities would also be a great way to ensure that future iterations of Amtrak’s website are accessible both in design and in function.
Overall, most participants were able to complete most tasks in this study, despite the issues they were confronted with. If Amtrak implements some key changes, such as more intuitive ticket checkout processes, more clear accessibility options, and less intrusive search features, the flow of user activities will be improved significantly. Amtrak can be an inclusive service that accommodates all users.
Comments