Summative study with unmoderated usability tests
Falabella vs Ripley
Project Overview
This case study is about a quantitative summative study conducted in March 2021, where unmoderated usability tests were performed on the websites of the Peruvian e-commerce stores Falabella and Ripley in their desktop version.
Project duration: 3 weeks.
This study was conducted to compare the usability and user experience of these platforms based on a series of specific tasks and objectives, covering both their main online shopping service, as well as other services that both platforms offer within the diversification of their business lines.
Four tasks were assigned to each participant in order to accomplish the objectives of the study. There were 52 participants in total, each randomly assigned to one of two groups.
Team: 6 UX Researchers
My role: UX Researcher
Responsibilities:
- Recluting
- Evaluator
- Design Research
- Content Creator
Challenges:
- Recruitment of people for unmoderated testing.
- Finding theoretical references to support the findings of the inspection
- Interpret the results of the emotions generated in the test and show them graphically using well-founded scientific resources.
Methodology
Research
Understanding the company and it's audience
The following question guided my use of Éxito website during the day I spent:
"What would a user want to achieve with a website like this?"
Upon doing this research, I analyzed Falabella's and Ripley's competition and also identified the company's value proposition.
Research
The type of business
Two online stores in Peru were compared using a variety of metrics and instruments that evaluate user performance through a quantitative analysis of user performance.
As the case study presented falls into the second category (summative usability tests), we chose to capture numerical data, since it was a comparison of two interfaces (Ripley, Falabella), these variables enabled us to compare the average effort for each task.
To obtain the greatest amount of qualitative data, a larger number of participants was required compared to a formative test, and with those data, a statistical analysis could be conducted to compare the results of the two interfaces.
In order to calculate this difference, we used the T-test, a statistical test used to compare the measures and averages of two normally distributed samples or groups. It is often used to determine if an item has an effect on the population of interest, or if two groups differ from each other in hypothesis testing.
Instruments
The following instruments were used to obtain quantitative data for measuring usability:
- Likert Scale. The user is able to indicate how much they agree or disagree with a statement using a five- or seven-point scale.
- SUS (System Usability Scale). In 1996, John Brooke created a widely used questionnaire whose aim was to measure user's satisfaction with the usability of systems and digital platforms, based on ten questions with a score from 1 to 5 depending on how much they agree or disagree with each statement.
User's flow
After understanding the user flow, we created tasks that focused on the possible uses of the platform in the daily lives of its users.
Tasks:
- Utilize the search engine to find an Acer Aspire 8GB RAM laptop with an undiscounted price of S/3,199.
- Find out the location and hours of operation of the store closest to you.
- Search for CMR Visa basic card benefits
- Search life insurance Effective credit.
Analyzing results
For this research, I used different formulas and tools to calculate the corresponding values. Furthermore, I analyzed the resulting graphs of each operation and the graph of users' emotions when they used these platforms.
Main Findings
Falabella website had a 63.64% success rate for tasks compared to Ripley website, which only had 88.46% success rate.
Additionally, falabella website participants took 248.5 seconds to complete the tasks, which was slower than the time taken by users who interacted with Ripley's website (182.70 seconds).
Looking foward
Takeaways
Impact
These problems could cause the image of the company or its website to be affected by the user's experience during their stay on the website.
I learned to:
- Demonstrate users' emotions with graphs and calculations.