The 5-Star Ratings Trap: How Faulty Customer Data Can Lead Us Astray
Every company believes it genuinely cares for, and listens to, its customers. Terms like “customer-first” and “customer-driven” have become commonplace, and everyone is in the business of earning Customer Love.
On the surface, it all sounds great. Across industries, companies are moving in the right direction. They’re recognizing the wants and needs of their customers and using these newfound insights to deliver a superior customer experience.
Yet, the customer data tells a different story. True, 73% of companies believe they genuinely care for their customers—but only 36% of customers actually feel cared for. This 40-point gap, what researcher Mblox refers to as the “Care Gap,” has a pretty irrefutable implication: There is a big difference between how companies and customers perceive the customer experience.
The Mblox study proceeds to identify a number of similar perception gaps, including a 20-point gap in satisfaction rankings (67% of businesses believe their customers are satisfied, while only 47% of customers feel satisfied).
It may be tempting to pass these overinflated rankings off as hubris on the part of the companies; but in truth, their assumptions are structurally sound, and often even data-driven. They’re just driven by bad data.
The Allure of “Bad Data”
The customer care perception gap is, perhaps, no greater than in the mobile app industry, where countless app publishers struggle to grow and monetize their apps, despite genuinely claiming to having built their roadmaps around the needs of their customers.
By and large, we expect more out of our mobile apps. According to a recent Sitecore and Vanson Bourne study, 60% of consumers are disappointed by their mobile app experiences. Adding insult to injury, the app industry experiences one of the lowest average Net Promoter Scores (19), coming in behind the life insurance and cellular phone service industries and just above health insurance. With the rare exception, mobile apps fall short when it comes to earning our love and loyalty.
So why aren’t app publishers everywhere taking note and building better apps? Because their App Store and Google Play ratings instill in them a false sense of confidence, an assurance that their roadmaps are, indeed, customer-first.
The average Android app has a 4.1-star rating, and the average iOS app comes in at somewhere around 4 stars. By any conventional measure, a 4-star rating means “Very Good,” or at the very least, “Above Average.” App publishers have every reason to take these strong ratings as a sign that they’re doing everything right… which, as we just saw, may be far from the case.
App store ratings clearly paint a very different picture of the customer experience than do traditional measurements of satisfaction, but why is this the case?
First, app store ratings and reviews are biased.
Don’t get me wrong. At Apptentive, we love ratings. They’re the single greatest driver of app store conversion and the rare social proof that your app can be trusted. As customer insights, however, ratings leave a lot to be desired—and it comes down to how star ratings are distributed.
In 2014, ratings in the App Store followed a distribution categorized by a negative skew (referring to the fact that the left side of the distribution is the longer “tail”):
We’ve previously written about the psychology of app store ratings and why distributions typically take the shape above, but to sum it up quickly, it’s a matter of self-selection. The process of reviewing an app requires no small amount of effort, requiring the reviewer to go through a six-step process that includes leaving the app and signing into the app store. Frequently, the only customers willing to put this much time into leaving a rating or review are those that have a strong opinion of your app (often translating into a 1-star or 5-star review). Those who have more neutral opinions see the six-step process as a barrier to entry, and have little incentive to put that much time into reviewing an app they neither love nor hate. As a result, your app almost certainly has a disproportionate number of both five-star and one-star reviews that skew the distribution, often in favor of the five-star reviews.
This skewness wouldn’t be a problem if it weren’t for the fact that we are hardwired to see shapes as symmetrical around a clearly defined central tendency (a phenomenon known as the Law of Symmetry in Gestalt psychology). When we look at a rating on a five-star scale, we expect the average rating to fall somewhere around the midpoint of the scale (3 stars). As the chart above depicts, however, only 30% of reviews fall at or below this midpoint. So while the average review is between 4.0 and 4.5 stars, we’re accustomed to perceiving anything above 3.0 as “above average.” Thus, a 4-star review can easily trick a publisher into thinking that they have build an exceptional, customer-first app, even though they really only have an average rating.
To remedy this thinking, we need to shift our entire sense of “average” one star to the right and benchmark customer satisfaction off a new, asymmetrical and negatively skewed chart:
Second, we’re wired to dismiss negative feedback.
The second reason app store ratings can lead us astray is because we are hardwired to prefer information that confirms what we believe about ourselves. This is what psychologists call confirmation bias. Simply put, we have a pre-existing belief (e.g., “my app is great”) that biases the way we interpret feedback, leading us to inadvertently prefer data that confirms our beliefs (five-star reviews) and giving less consideration to data that refutes our beliefs (one-star reviews).
As our own Robi Ganguly told Forbes:
“There are many reasons we rationalize ignoring complaints: It’s a lost cause. It will go away if we just don’t pay any attention. Or it’s just easier.”
We view one-star reviews as rare exceptions in the customer satisfaction: One-off incidents where an individual customer had a negative experience. And of course, these reviews get to us. We want to please all of our customers. Yet, we still distance them from our greater understanding of customer satisfaction and what our “average” customers think of our app.
Unfortunately, this is another way bad data leads us astray. We think of negative experiences as one-off issues, when, in reality, they are much more common than we realize. For every one unhappy customer that leaves a review, there are 26 other, equally unhappy customers that simply uninstall your app without ever communicating any sort of feedback.
While some incidents may truly be one-off, hedge your bets by assuming that there are others who share that negative experience. Fix the experience for one, and you may just find that you’re able to win back 26 others.
Adversely, as five-star reviews start to pour in, we’re elated. They’re our long-sought proof that customers love our app just as much as we do. It doesn’t even cross our mind that these reviews might not be truly representative of our average customer (the ‘Silent Majority’ that can’t be bothered to go through the six-step process of leaving a review). We read the raving reviews and convince ourselves that there is no more work to be done. Customers love us.
Yet, as we saw with NPS and other measures of customer satisfaction, customers’ love for our app may not go as deep as we are inclined to believe. There is still work to be done, and thinking otherwise will only open a void for our competitors.
The Pursuit of Better Customer Data
So if app store ratings and reviews don’t accurately speak to the customer experience, what does?
The answer comes down to less the metric or channel we opt for and more the methodology we use to collect our customer data. Recall that app store ratings and reviews are biased because they come from only those with an extreme opinion of our app. They come from our biggest lovers and haters, those with an opinion strong enough to warrant taking the time to leave the review. This feedback comes not from our average customer but from our vocal minorities, those customers at either extreme that typically account for less than 1% of our total customers:
App store ratings and reviews lead us to evaluate our entire sense of customer satisfaction from the perspective of those at the very periphery of our engagement chart. Their opinions, while certainly valid, rarely speak to the opinions of our average customers and may even influence our “customer-driven” roadmap for the worse, as we let the squeaky wheels get the grease.
To truly understand our customers, we need to hear from those customers in the “Silent Majority,” for whom the review process presents a barrier to communication. We need to find a way to ‘give them a voice’ and make customer communication effortless enough to engage those customers who have little more to say than “meh” in an app review.
Breaking down the wall
We’ve come up with three tips, pioneered and tested by our own customers, to help turn your customer data into customer insights:
1. Design for the mobile customer in mind.
The average mobile app customer spends a mere 2.5 to 9 minutes in an app, for any one session. They launch an app with a specific task in mind, complete that task, and exit the app, all under ten minutes. That doesn’t leave a whole lot of time for brand communication, especially when the process of leaving an app store review can singlehandedly take up this entire session.
App publishers that want to engage the silent majority must be cognizant of the customer journey—and their customers’ precious time. Make communication effortless by asking customers for the least amount of their time possible. At Apptentive, we’ve found the best ways to do this are:
- Providing a channel for customer communication within the app itself, instead of requiring customers to launch your website or open their email.
- Proactively prompting customers for feedback, with well-placed surveys or feedback forms.
- Requesting only the data you need most to ascertain the customer experience.
- Limiting questionnaires to ten questions or fewer, and designing the questions for a one-click response (multiple choice, slider scale, etc.) instead of using complex matrices or free-response forms.
- Keeping questions optional, particularly those that are more time-consuming, so that a customer can still share insights even if he or she wishes to skip a particular question.
2. Segment customers for meaningful benchmarks.
If your app has thousands of monthly active users, it’s going to be difficult to survey a cross-section of customers proportionate with your average customer satisfaction levels. You’ll still experience self-selection and see some level of skewness in your response distribution. Fortunately, there’s an easy remedy: Benchmarking.
Under the same logic that we hedged our bets by moving our understanding of an “average” rating from 3 to 4 stars, it’s possible to attain a more accurate benchmark to evaluate our customer insights against.
To do this, first collect insights from a random sample of your customers. If you’re prompting customers with an in-app survey or feedback form (quantitative measurements like the Apptentive Engagement Ratio or the Net Promoter Score provide the most meaningful benchmarks), remove all targeting criteria before sending it to a small but random sample of customers. Once collected, your average datapoints for each question serve as your benchmarks. By themselves, they’re not your more reliable insights, but they can serve as a powerful reference, against which you can measure future, segment-specific responses to control for issues like self-selection.
Once you’ve established your benchmark, you can target specific customer segments to determine how responses differ between groups. For example, Apptentive’s targeting capabilities allow you to segment by loyalty (e.g., prompt only customers who have launched your app 5+ times) and usage (e.g., prompt customers only after they have explored this feature of my app).
Comparing your segmented responses to your benchmarks provides a more granular look into the customer experience, addressing such questions as:
- How does customer satisfaction evolve with usage? (e.g., how does a first-time customer’s evaluation of my app differ from that of a loyal customer?)
- Does customer satisfaction depend on customer use cases? (e.g., Do customers that use the chat feature of my app rate their overall experience differently than those who have yet to use the feature?
- Which customer actions are correlated with the highest satisfaction levels (and how do I push those features to grow satisfaction)?
Short and sweet, our final recommendation is active listening.
Today, customers can communicate with their favorite brands with the simplicity of a text, email, or tweet. But while customers have caught on to a new age of open communication, companies have been slow to follow suit. Some 70% of brand-addressed complaints continue to go ignored, either as a result of inadequate social monitoring or simply the same confirmation bias that plagues app store reviews.
In order to understand your customers, you have to listen. You have to meet your customers wherever they are (social media, discussion boards, etc.) and lend an ear to feedback of any sort, positive or negative. This unsolicited feedback, though rare, provides the best insights of all. Don’t let it go unnoticed.
Using Customer Data to Build a Better Mobile Experience
The goal of any Voice of the Customer program is to build a better customer experience. You’re now well on your way to collecting actionable, accurate, and representative customer insights, and in doing so, you’ve laid the groundwork for a customer-driven experience. In many ways, however, your work has just begun. It’s now time to use the insights—to inform your messaging and guide your product roadmap.
To help you get started on this next challenge, we’ve released a complimentary eBook on customer-driven app development. Get started today by downloading The People-First Approach to Enterprise App Development.
Have a favorite tip for measuring customer satisfaction that we missed? We’d love to hear! Share your thoughts in the comments below or tweet us @apptentive.