UX TEST Plan + FINDINGS: COIN
- Timeframe: 3 months
- Team size: 8
- Role: UX Researcher
- Target user: FinTech
- Platform: Hardware, Mobile
- Tools used: UX Testing
UX testing and findings for UT iSchool’s Usability class. This project was completed with Lauren Freeman, Matthew Horn, Bridget Lavin, Kristen Oglesbee, Juliette Seive, and Lauren Thomas. As a group we created a thorough usability test plan, involving 5 tasks, with the goal of evaluating users' ability to navigate the Coin device and accompanying mobile app.
Our test plan can be found below followed by our findings, recommendations, and results. Since performing this study production has ceased for the Coin device.
COIN USABILITY TEST PLAN
Using Coin 2.0 (a multi account credit card device) we will conduct a usability test. The goals of usability testing include establishing a baseline of user performance, establishing and validating user performance measures, and identifying potential design concerns to be addressed in order to improve the efficiency, productivity, and end-user satisfaction.
As consumer-facing hardware becomes increasingly more popular, it is important for our group to learn how hardware and software intersect. In our usability testing of Coin, we will analyze how participants use Coin’s hardware and software and document common pain points. Based on these pain points, we will be able to determine ways that the current design of Coin can be improved.
- See if participants are able to find the information they need to use Coin 2.0.
- Describe and demonstrate what, if anything, was most frustrating to participants when using Coin 2.0.
- Determine what is missing, from a usability standpoint, from the Coin 2.0 hardware and software interaction and how to fix it.
- Decide if Coin 2.0’s hardware and software user experience is easy enough for widespread adoption.
- Determine if Coin can be used without confusion or issue at a retail location.
METHODOLOGY: PARTICIPANTS, TRAINING, PROCEDURE
The test will be administered to seven participants, four males and three females aged 19-42. The recruitment process for participants was based on colleagues and current University of Texas undergraduate and graduate students and employees that we were already familiar with. Familiarity with participants gave us the option to select a diverse group of participants based on various backgrounds. Participants were also recruited based on gender, experience with technology, mobile operating system preference, and age. Participants are not required to have any knowledge of or experience using Coin coming into the study. The participants will receive an overview of the usability test procedure, equipment and software. The responsibilities of the participants are to attempt to complete a small set of tasks and to provide verbal feedback by taking the facilitator and note taker through the steps taken regarding the usability and accessibility of the user interface. The participants will be asked to provide honest opinions regarding their experience of the usability of the application.
Training (introducing Coin)
Coin 2.0 is a secure device that can hold up to 8 credit cards, debit cards, or gift cards. The device is connected to a smartphone through the Coin app. The participants will receive an overview of the usability test procedure, equipment, and software. Participants will be asked to complete a demographic and background questionnaire before beginning the test. Participants will be directed to add a gift card manually and through the card reader, add information, and name the card. Participants will then be instructed to use the coin card to make a purchase. There will be no training for this device other than mentioning the purpose of the device.
An iPhone, Coin device and reader, Coin instruction manual, and two gift cards will be provided to participants.
Each usability test will be conducted at two separate locations. Participants will take part in the first portion of the usability test at the IT Lab in UTA 1.210 at 1616 Guadalupe Street in Austin, Texas. The second half of the usability test will be completed at a nearby Starbucks store at 501 West 15th Street. To complete the first part of the usability test, users will need to interact with both an iPhone and a Coin device. They will also be provided with two Starbucks gift cards needed to complete certain tasks, as well as for a recruiting incentive. The second part requires use of just the Coin device in a retail setting.
One iPhone and one Coin 2.0 device will be provided. The same devices will be used in each session. The Coin instruction manual will also be provided for users to reference.
The tests will be conducted individually. For the first portion of the test, users will be seated in a quiet room accompanied by the facilitator and note takers. The test sessions will be audio recorded.
The facilitator will brief the participants on the Coin device and instruct the participant that they are evaluating the device and accompanying application. Participants will sign an informed consent that acknowledges that the participation is voluntary, that participation can stop at any time, and that the session will be audio recorded but their identity will be kept confidential. The facilitator will then ask the participant if they have any questions.
Participants will complete a demographic and background questionnaire before beginning the test. The facilitator will explain that the amount of time taken to complete the tasks will be measured and that the user should focus on each task until it is completed. The participant will read aloud the task description from the printed copy and begin the task. Time-on-task measurement begins when the participant starts the task.
Part I: IT Lab
The facilitator will instruct the participant to ‘think aloud’ to create a verbal record of their interaction with the device. The note takers will observe and enter user behavior, user comments, and system actions in the data logging application.
Part II: Starbucks
The user will be accompanied by the facilitator and note takers to the Starbucks location. They will be instructed to complete tasks that involve purchasing food or beverage. The user will be instructed to ‘think aloud’ throughout these tasks.
After all task scenarios are attempted, the participant will complete a user satisfaction questionnaire.
ROLES: FACILITATOR + NOTE TAKER
- Provides training/overview of study to participants
- Defines usability and purpose of usability testing to participants
- Assists in conducting of participant and note taker debriefing sessions
- Responds to participant's requests for assistance
- Records participant’s actions and comments
Silent note taker
- Assists the facilitator in identifying problems, concerns, coding bugs, and procedural errors
- Serves as data logger
All persons involved with the usability test are required to adhere to the following ethical guidelines:
- The performance of any test participant must not be individually attributable. Individual participant's name should not be used in reference outside the testing session.
- Personal information that the user enters into either of the device will be removed at the end of the session to protect their privacy.
USABILITY TASKS + METRICS
The tasks listed below are based upon real-world usability of Coin, from adding a gift card to Coin to the usage of a stored gift card at a target location.
1. Sign into Coin app using pre-made account
The facilitator will place an iPhone in front of the user along with the Coin instruction booklet, Coin device and reader. The phone will have the Coin mobile app installed, as well as having an account logged in. The facilitator will then instruct the user to open the app and sign-in.
2. Add a gift card through swipe mechanism w/nickname CRD1
The facilitator will instruct the user to use the supplied booklet and card reader to add a starbucks gift card to their Coin app and Coin card. The user must also add a nickname of CRD1 to the card.
3. Add a gift card through manual input w/nickname CRD2
The facilitator will instruct the user to use the supplied booklet to add a starbucks gift card to their Coin app and Coin card. The user must also add a nickname of CRD2 to the card.
4. Select CRD2 for use inside Starbucks
The user will be asked to navigate via the Coin card to select CRD2 for use inside of Starbucks.
5. Buy Starbucks item with CRD2
The user will be asked to use the card CRD2 to buy a beverage or food totaling less than $5.00 from Starbucks.
Usability metrics refers to user performance as measured against specific goals deemed necessary to satisfy usability requirements. Scenario completion success rates, error rates, and subjective evaluations will be used. Time-on-task data will also be collected.
- Scenario Completion
Each scenario will require that the participant obtains and input specific data that would be used in course of a typical task. The scenario is completed when the participant indicates the scenario's goal has been obtained (whether successfully or unsuccessfully) or the participant requests and receives sufficient guidance as to warrant scoring the scenario as a critical error.
- Critical Errors
Critical errors are deviations that prevent completion of scenario goals. Obtaining or otherwise reporting of the wrong data is a critical error. An example of an incorrect data critical error would include not being able to add a gift card through the swipe mechanism or an inability to buy a Starbucks item with CRD2. Participants may or may not be aware that the task goal is incorrect or incomplete.
Independent completion of the scenario is the goal; help obtained from the facilitator is a critical error. Critical errors can also be assigned when the participant initiates (or attempts to initiate) an action that will result in the goal state becoming unobtainable. In general, critical errors are unresolved errors preventing completion of the task or errors that produce an incorrect outcome.
- Non-critical Errors
Non-critical errors are errors the participant recovers from or, if undetected, do not result in processing problems or unexpected outcomes. Although non-critical errors can be undetected by the participant, when they are detected they may be frustrating to the participant.
These errors may be procedural, in which the participant does not complete a scenario with maximum efficiency (excessive steps and/or keystrokes). These errors may also stem from confusion (selecting the wrong function, or attempting to add a card from the wrong screen).
Participants can recover from noncritical errors during the process of completing the scenario. Exploratory behavior, such as opening the wrong menu while searching for a function, will not be coded as a non-critical error.
- Subjective Evaluations
Subjective evaluations regarding ease of use and satisfaction will be collected via questionnaires, and during debriefing at the conclusion of the session. The questionnaires will utilize free-form responses and a 7 point Likert scale to gauge the experience. Emoji stickers are sad, frustrated, angry, neutral, content, happy and excited. Some general questions are:
- How comfortable were you selecting a certain card? [Likert]
- How did you feel about the setup process of adding a new card? [Emoji stickers]
- What was the most memorable part of the study for you? [Comment box]
- How likely are you to use this product in your daily life? [Likert/numerical scale]
- How useful did you find this product? [Likert]
- How likely are you to recommend this product to another person? [Likert]
- How comfortable were you with the time required to set up the product? [Likert]
- Scenario Completion Time (time on task)
The time to complete each scenario, not including subjective evaluation, will be recorded.
USABILITY GOALS + REPORTING RESULTS
- Completion Rate
Completion rate is the percentage of test participants who successfully complete the task without critical errors. The completion rate represents the percentage of participants who accomplish the task completely resulting in the correct outcome. If a participant requires assistance in order to achieve a correct output then the task will be scored as a critical error and the overall completion rate for the task will be affected. Non-critical errors will be recorded as a means of identifying pain points and inefficiencies when attempting to complete the task.
A completion rate of 100% is the goal for each task in this usability test.
- Time on Task (TOT)
The time to complete a scenario is referred to as "time on task". It is measured from the time the person begins the scenario to the time he/she signals completion. TOT will be calculated for each task. The usability goal here is that as the user becomes more comfortable with Coin, their time on task will decrease with each task.
- Subjective Measures
Subjective opinions about specific tasks, time to perform each task, features, and functionality will be surveyed. At the end of the test, participants will rate their satisfaction with the overall system. Combined with the interview/debriefing session, these data are used to assess attitudes of the participants. The subjective measures will be tested using emoji stickers, as well as discussing with the participant about their overall satisfaction after each task, and then over their general experience with the product. Participants will be asked to relay the most memorable part of the study, good or bad, in order to help the facilitator gain insight into pain points, or moments of delight.
- Analyzing the Data
Completion rate will be analyzed to determine the real-world applicability of Coin and ways to improve the critical failure rate and ultimately completion rate.
While time on task may not be noticed by new users, over time it may become a critical issue and contribute to dropping use of Coin. Improving the tasks to reduce time for completion is necessary to provide a seamless user experience.
Our analysis will characterize the issues encountered by users by focusing on the frequency of unique issues, frequency of issues per participant, the percentage of participants who encountered specific issues, and unique issues for each tasks.
Using subjective measure, we will determine how users feel about the device. Many times, even if there are significant issues with usability (similar to touchscreen keyboards), people feel like they have a good experience with the device. The facilitator will inquire about the participants satisfaction after each task, and at the end of the study, discuss with the participants their overall feelings about the product. Satisfaction will include ease of use, moments of frustration, and how the participant felt, candidly, about Coin. By asking subjective questions about each facet of their experience, a comprehensive report will be made to help improve user experience.
This usability study pertains only to Coin 2.0, and does not extend into other forms of payment. By compiling the data on Coin 2.0, other researchers will be able to develop studies to compare Coin to conventional credit cards as well as other multi-account credit cards and more.
The Usability Test Report will be provided at the conclusion of the usability test. It will consist of a report and a presentation of the results. The report will include evaluation of the usability metrics against the goals, subjective evaluations, and specific usability problems and recommendations for resolution. The report will further include recommendations to aid in implementation strategy. The report shall be delivered to Professor Nordquist by March 29, 2016.
We analyzed the data from our usability testing to determine and inform our recommendations for improving Coin’s overall experience. Issues we identified regarded ambiguity in labeling and signifiers as well as animations and instructions provided to users.
- Increase information for user-onboarding
- Better labeling on hardware
- Less ambiguous swipe animation
- Address swipe mechanics
- Clarify signifiers
- Fix app glitches