At Bazaarvoice, one of my main job responsibilities is planning and conducting usabilities. In 2018, I completed 20+ different usability projects; each project involved 5 to 20 testing sessions with clients and consumers. I contributed to all the major processes of running these tests, including but limited to user recruitment, writing testing plans, modifying mock-ups, set up the testing platform, moderating testing sessions, analyzing qualitative data/quantitative metrics, presenting test results, and making product design/engineering recommendations.
The example project I share here is a test with 3 mock-ups on 10 consumers. Each session lasted about 30 minutes. My research contributed to the launch of a new review submission system, which increased review volume by 9X and click-through rate by 2X.
Background & Objectives
“Multi-Product Submission (MPS)” is a product feature that enables consumers to submit multiple ratings and reviews directly from PIE (Post Interaction Email, meaning emails sent after a consumer “interacts” with a company, i.e. purchasing a product, or completing a service such as opening an account online).
Our UX designers designed 3 product mock-ups in order to enhance the user experience of MPS. The PM came to UXR team asking for help. We proposed to use the Usability Test method in order to achieve the following goals:
- Collect user feedback for each design mock-up
- Understand user likes and dislikes for each major design components and variations
- Understand the best approach for integrating photo and social collection into the multi-product submission tool
- Make product design recommendations
- Emails v.s. text messages in terms of review collection
- General psychological factors of submitting reviews
Notes:
- The three mock-up variances were based on a previous product concept test survey. Our team received 187 valid survey responses (respondents who passed the screening). The survey verified the product concept and helped to refine the product requirements. The findings directly informed the design of mock-up
Process
1. Define Task Scenarios and Write Testing Plan
Together with a UX designer and a PM, I reviewed the 3 mock-ups to be tested, understanding the commonalities and variances. I defined 4 task scenarios for a 30-minute test. For each scenario, I specified the user task and prepared written scripts to guide observations and ask questions. The scenarios are:
- Open PIE, find the product to be rated and reviewed in the email
- Rate the product and input text reviews
- Submit review
- Move to review the next product
2. Recruiting Users and Scheduling Tests
Through a user-recruiting platform called User Interviews, I successfully screened recruited 10 users for the test and created a waiting list of 10 users to handle last-minute cancelations.
I coordinated the communication between the participants and the team, sending out welcome emails with instructions to participants.
I designed a set of screening questions to select the target user group for testing. In this scenario, our priority was given to the group who often wrote product reviews online.
3. Conducting the Usability Test
We recorded ten 30-minute real-time user interactions and took time-stamped observational notes in a tool called Lookback.
My contributions include:
- Set up the testing environment to facilitate the test session
- Taking detailed observation notes during tests
- During the tests, providing the moderator with critical information (e.g. important questions to discuss, or unexpected findings that need drilling down more)
4. Analyzing Test Data
I organized observation notes and interview transcripts to identify the emerging themes with the focus on what part of the design the users like/dislike and the reason for their preferences.
I created a spreadsheet template to organize raw data summarize findings.
5. Summarizing and Presenting Findings
Using usability test method, I helped answering the following question:
- Which mock-up is the most favorable one? Why?
- What design components should be kept/discarded? Why?
- What usabilities issues are observed and recorded in the test? How serious are they?
- Specific design recommendations for the next-step development work
I contributed to the presenting the findings to the Product and Engineering team. Furthermore, I provided support to UX designers and UI developers when they began a new round of design/development work, making sure that the research recommendations are clearly perceived.
Impact
Before:
After:
Demonstrated Business Impact:
