Quantifying Customer Problems Using a Severity Rating Survey
- Christoph Nguyen
- Jun 10, 2024
- 3 min read
June, 2024 | Christopher Nguyen
Contents
Resources
Overview
Measuring customer needs is a rewarding yet complex activity. As a UX Researcher, I frequently encounter product teams asking "What type of product experience should we build?" Sometimes, they already have a problem to solve, other times they are looking for new opportunities to explore. In this case, it was to prioritize customer problems to develop the 2024 product roadmap.
Disclaimer: This document serves more as a how-to piece and focuses less on the actual project insights.
Severity Ratings: Evaluating customer problems to inform the product roadmap
This survey method doesn't have an established name because I created it myself, though inspired by Jakob Nielsen's severity ratings for usability issues. Unlike the traditional use of severity ratings, where the researcher rates the severity of a usability problem based on a single study, my repurposed method involves customers rating the severity of the problem themselves. This is done via a large-scale survey that is designed to prioritize a large number of customer problems identified through multiple research studies, not just from a single usability test.
It also takes on a resourceful approach to identify customer problems that have fallen through the cracks or been forgotten over the past years. I've used this method to help my team identify if there was more to add to the upcoming product roadmap based on any unresolved customer problems.
Key differences between traditional Nielsen's severity ratings vs. Repurposed severity ratings for survey research (Table 1).
Method | Rater | Problem origin |
Traditional Nielsen severity ratings for usability test | Researcher rates usability problem severity post usability test | Single usability study |
Repurposed severity ratings for exploratory survey | Participant/customer rates severity of problems post research repository review | Multiple studies (interviews, usability test, concept test, surveys etc.) |
Table 1 (Severity question example)
Research process for severity survey:
Research insights scraping via research repository > unresolved problem identification > severity survey
How the Severity Survey Works
This method measures the proportion of customers who perceive an issue as a problem and then compares the severity of the different problems using a paired t-test. This statistical test helps determine if there is a statistical difference in how customers rate the severity of a problem by measuring pairs of issues. The severity rating survey uses the same language as Nielsen's' ratings that is based on a likert rating scale that ranges from 0-4.
Survey method example: Say that we are focusing on customer retention and we want to know what is preventing customers from being repeat customers. Below is an example question for how the severity rating can be applied to customers who have not returned for over 6 months or more.
On a scale of 0-4, please rate the extent to which this issue has influenced your decision not to return to company X:
"Difficulty searching for products and options available"
0 - I don't consider this a problem at all
1 - Cosmetic problem: No immediate action needed
2 - Minor problem: Low-priority, but should be addressed
3 - Major problem: High-priority, important to fix
4 - Extreme problem: Critical, imperative to fix
Severity Survey Analysis and Impact
Problems were visualized as proportion percentages. A table showcasing the paired t-test revealed which problems were statistically different from each other.
This survey served as a method for evaluating unresolved customer problems from a holistic standpoint by recalling past studies, saving research resources, and informing the team if there were any research insights being overlooked. The results indicated that most unresolved problems were seen to be minor to cosmetic problems, suggesting that many of the severe problems identified from past research were already being worked on. Additionally, as a next step, we decided to run an ideation workshop to further explore areas to research and assess the gaps in our knowledge.
Severity Survey Disadvantages and Advantages
Disadvantages:
Needs careful selection of problems to evaluate (prone to selection bias)
Problem description can be misinterpreted by customer
Past research could be outdated from current customer needs (insights from past 2 years)
Advantages:
Requires little initial research prep because insights already exist within research repository
Can refer to past research insights to get a sense of customer segment
Final Thoughts
In this document, I've outlined a survey technique to help my organization better understand customer needs and problems.
When my team needed guidance on the product roadmap, the severity survey revealed there were no known unresolved problems being overlooked and that new research exploration was needed.
These use cases are some of many for these survey methods and are not limited to the cases I've explained above. I encourage researchers to experiment with different survey techniques, contributing to the evolving product research toolkit.