Prototype
Wizard of Oz prototypes are 'for testing design assumptions of HCI applications when the actual technology is either not available or too expensive to develop during the design phase of a project'. For this prototype, we wanted to test if a chat bot would be an effective way for customers to order a custom edible arrangement online. This prototype was a team effort, as multiple people are required to pull off the illusion of a wizard of oz prototype test.
To make this prototype, we each wrote a draft script of what we thought the bot should cover. Then I took the ideas and wrote them up into a final bot 'script' containing all the options and routes a user could take through the bot. This would also be what the 'man behind the curtain' would follow during the test to realistically impersonate the bot.
Testing
We conducted a test of our chat bot in a conference room in Seig. One of our team members was set up in another room as the Wizard, she was logged into an account that we had made under the name 'eddie' and was waiting for a cue that the scribe would send when the test was ready to begin. In the room with the participant was the Facilitator, who was to direct the testing, communicate with the user, and orchestrate the session, the Scribe who captured notes on the user's actions, what happened, how the prototype performed, and the Documentarian who captured the user test on video.
We had planned on testing with 2 people and had one back up, but 2 of the 3 participants were unresponsive on the day of testing and due to the limited time for this assignment we were unable to reschedule them for another time. A complete transcript of the participant and the wizard's interactions is to the right.
Results
Notes During Test
Participant didn’t read what was on the screen, and did not respond with an affirmative
Expected it to understand what he meant when he typed whole order at once
When asked if he had seen the first note he said he intentionally ignored hello message
He wanted options that are not on list
Was surprised that it understood absolutely
Typed fruit name instead of number
Post Test Interview
Doesn’t feel like a chat it feels like a menu
Feels very rigid
Why ask all these questions
Likes that it understood ‘absolutely’
Intro was too slow, wanted to get right to selection
Wants buttons rather than having to type
Analysis
Overall the results of our test showed that a chat bot is not the best format for this type of ordering process. I suspected this when I realized both the number and closeness of the different fruit options that Edible Arrangements offered. Because, for example, there were five different types of pineapple (and we cut options that were too similar) we decided that the best way to have users to differentiate would be to number all the options. This made the chat bot feel very mechanical and not much like a conversation. As our test participant said, buttons would be better for a custom order in this scenario. A simple click box ordering system would be much more efficient and effective for ordering a custom edible arrangement.
Feedback
The feedback we got on our project was that the wizardry was great and very convincing and that the prototype itself was very effective but that the video itself could explain more of the project. We had made our video not be a stand alone artifact but as evidence of our test to be shown alongside our written explanation of the project.
Comments