Likes Mean Nothing; Love Means Everything

Our 5-Step Process for Determining if Customers Really Love Your App
& Why Traditional Methods of Measuring Customer Sentiment Fall Short

Whether you’re a marketer, developer, or product manager, there’s only one question that truly matters: Am I building an experience customers love?

At the end of the day, we’re all in the business of earning Customer Love. It’s what keeps our business afloat, what keeps customers coming back day after day, and what turns your app into an experience. Yet, this single most important metric remains, for far too many brands, an abstract buzzword. It’s a riddle wrapped up in an enigma that ultimately comes down to guesswork: Of course customers love my app. After all, what’s not to love?

The unfortunate reality is that this assumption is often wrong—and may be costing you lost revenue, lost customers, and wasted development time. As our friends at SurveyMonkey found in a survey of 1,500 consumers and businesses, there’s a serious disconnect between how businesses perceive the experience they’re building and how consumers perceive the experience they’re receiving. Another recent survey echoes this sentiment, revealing that 75% of brands believe they love their customers, while only 36% of consumers feel that same love.

So what’s causing this disconnect?

Of course, saying that customer experience evaluation comes down to guesswork was a bit of an exaggeration. Every brand or app has its own way of measuring Customer Love. And often, it looks a little something like this:

Marketing: Our customers love us. Downloads are growing at 20% MoM, our star rating is at a healthy 4.3, and ARPU is 15% higher than it was this time last month.
Customer Success: The proof is in the pudding. Support email volume is down 25%, and we retained 60% of unhappy customers by acknowledging and responding to complaints.
Engineering: We’ve built a product customers love. Customers are showing higher engagement, session length is up 30%, and we’re seeing positive validation of our upcoming features based on our initial beta tests.

All good… right?

Perhaps not.

These are all important metrics that every app publisher should know, but they don’t capture that elusive concept we call Customer Love.

The metrics above show that your app is doing well, that you’re generating a positive cash flow, and that your customers generally like what you’re creating. The keyword, here, is like. When you create something that customers like, you’re meeting their needs. They’re satisfied with the experience and likely consider your app superior to your competitors’. It does not, however, denote competitive advantage. Nor does it guarantee your customers won’t leave your app the moment something bigger and better hits the app stores.

Love takes liking one step further. It denotes loyalty, sustainable competitive advantage, and not only meeting but exceeding customer expectations. When you create an app customers love, you’re creating an experience that transcends the app.

Suddenly, your mobile customer isn’t a “user” but someone just as invested in your app as you are in them. They’ve come to expect to feel valued and listened to each time they sign in.

And in exchange, you have their loyalty.

Why traditional methods of measurement fall short

Before we introduce our top strategies for gauging Customer Love and making this abstract concept a little more concrete, let’s look at what Customer Love is not.

There are five main myths about customer love:

Myth #1: Customer Love is shown in your app store rating.

Even some of the best mobile marketers are guilty of reading too much into star ratings. As our recent Consumer Survey revealed, app store ratings are an instrumental part of the mobile marketers’ toolkit. They serve many functions, from growing app store conversion to boosting in-app purchases. When it comes to measuring customer sentiment, however, ratings come up a little short.

The reason for this shortcoming lies somewhere between the realms of behavioral psychology and the design of the App Store. If you’ve ever wondered why less than 1% of your mobile customers have rated your app, simply step into the shoes of your customer. What seems to you to be a two-second job is nothing short of a herculean task in the eyes of your customer. To leave a review, customers have to respond to a rating prompt, launch the App Store, navigate to the Reviews page, select ‘Write a Review,’ sign in, and finally leave the rating and write the review. This adds up to a six-step process, lasting almost the entirety of the average five-minute app session. The Google Play Store features a more representative mix of reviews due to its simplified process, but it still sees a heavy concentration of one-star and five-star ratings. Here, it’s not uncommon to see a ratings distribution like the one below:

Measuring Customer Love by the average rating above (2.8) would seem to suggest that we’re doing alright. Not good, but alright. However, only a very small proportion of the ratings hover around 2.8. Instead, we see a very high concentration around 1-star and 5-stars. This suggests that either (a) we have a very polarizing app, or (b) our ratings simply aren’t representative of our audience or customer sentiment as a whole.

Additionally, Customer Love is a concept that can’t be aggregated and reduced to a single number. When it comes to feedback, every voice matters—especially those with opposing viewpoints. Your average rating can be high while your app is sitting on a powder keg of unsurfaced problems. Likewise, you can have the best app in the world but still have a low rating if a vocal minority of critics dictates your star rating. Customer sentiment needs to be assessed both on a group level and an individual level.

That is, you need both the vocal minority and the silent majority.

Myth #2: Customer Love is measured by revenue.

While the concept of Customer Love might be a little challenging to sell to your CEO, revenue is a language we all speak. In the case of apps, we’re concerned with the average revenue per user—how much each active customer is “worth” on a monthly basis.

It’s safe to assume, then, that if a customer is generating more money for your business, they’re exhibiting more love for your business. After all, why pay more for something unless you see the value in it?

To demonstrate why this logic isn’t as sound as it may seem, let’s leave the world of mobile for a moment and think, instead, of your latest cable bill. Cable companies are notorious for spiking prices on a regular basis. You’re now paying more for the exact same service. Your “ARPU” is high, but, if anything, you have less love for the company. You’re not happy with the increased price, but your alternatives are scant. You can either give up cable or search for an alternative provider—a search that will come up short in this monopolistic setting. Despite not wanting to pay more, you’re stuck with your provider.

That is, until a better alternative comes around.

The cable example illustrates an important distinction: The difference between short-term security and long-term security. There’s no denying that ARPU is an important metric, but it’s a short-term metric. If you’re seeing a high ARPU now, there’s no guarantee that you’ll see a high ARPU in six months. The app market is incredibly competitive, with over 1.3M apps, and you can bet customers will always be on the lookout for a better, cheaper app.

Customer Love, on the other hand, is in it for the long game. Customers who love your app are loyal. They have no interest in trying out competitors because they don’t want to give up the experience they’ve come to love. They’re also more forgiving of temporary setbacks (like bugs in a new update), knowing that their favorite app will be back in full swing momentarily. Whether you see the highest ARPU from them or not, you can count on your most loyal fans offering something even more important: lifetime value.

Myth #3: Customer Love only benefits retention.

It goes without saying the key to retaining customers is providing them with an experience they love. Yet, retention isn’t always top of mind for mobile app publishers. The focus is often on the other side of the picture: Acquisition.

For example, mobile marketers might be responsible for increasing monthly downloads by 10%, reducing the cost per install by 20%, or closing the gap between downloads and installs. These lofty goals provide little incentive to focus on retention. This, in turn, leaves marketers with neither the time nor the resources to invest in Customer Love.

Customer Love, however, affects all facets of your business. Our emphasis on retention over acquisition aside, fostering Customer Love is one of the most effective methods for increasing acquisition. Customers who love your app will be more likely to leave five-star reviews in the app stores. These reviews, in turn, provide a rare signal of quality in an otherwise blind app discovery process.

A potential customer evaluating your app against your competitors, without any existing loyalty or love, would naturally choose the app with the highest rating. And in fact, a single star might just make or break your app. If you can mobilize enough of your loyal customers to leave reviews and bump your three-star rating to four stars, you can expect app store conversion to increase by 89%. Even more powerful, this new influx of customers chose your app over your competitors whereas they’d do just the opposite if you had one less star.

Myth #4: Customer Love is one person’s job.

It’s easy to pass off the job of fostering love and loyalty to a single person or department. It seemingly lives somewhere between marketing and customer support, far away from the developers who create the very experience you want your customers to love.

Yet, when you silo Customer Love off like this, you extinguish its power. The customer becomes disentangled from the app and reduced to an anonymous data point in one team’s basket of metrics. This structure is also prone to misrepresentation. Marketing and customer support typically only hear from those who have had a negative experience with your app. Their feedback and suggestions might earn their love, but do their needs match those of the rest of your customers? Is one side of the story enough to inform your product marketing strategy?

To truly foster Customer Love, it needs to be everyone’s job. Marketing can collect customer insights and organize them by priority; engineering can use those insights to create a validated product roadmap and roll out the features their target audience wants most; and customer support can communicate with and survey customers for a process of continuous innovation. When all departments work together, it’s the customer who wins.

Myth #5: Customer Love is a cost center, not a revenue driver.

An investment in the customer experience might seem like a discretionary cost, or a “nice-to-have.” As long as you can manage to retain them, anything above and beyond to turn satisfied customers into ardent fans is a misuse of limited resources. Yet there’s nothing discretionary about Customer Love.

Recent research has shown a surprising relationship between Customer Love and your bottom line. Beyond the aforementioned effects on customer acquisition and retention, investing in your customers drives revenue through two important channels:

First, happy customers pay more. According to a study by RightNow Technologies, an astonishing 86% of customers are willing to pay more for a better experience. Invest in your customers, and they’ll invest in you.

Second, the customer experience is becoming the major differentiator. Customer experience will overtake both price and product as the leading differentiator by 2020. This is most evident in the competitive world of apps, where your app is vying for downloads against a sea of similar, and likely free, competitor apps. Create the best experience, and watch as your competitors’ customers flock to your app.

Then what’s a better way of gauging Customer Love?

No matter how you dive into the analytics, an understanding of customer sentiment will always remain tantalizingly out of reach.

The bad news is that unlike our other favorite metrics, Customer Love can’t be reduced to a nice and clean number. It’s a concept that can’t be averaged out, and one that varies greatly from segment to segment.

The good news is that we don’t have to calculate how much customers love their experience with our app. Our customers already hold the keys to this coveted data. We just have to ask them for it.

5 steps for collecting the data you need

Customer Love is the rare metric that eludes even the most sophisticated analytics tools. It’s one that necessitates breaking through the screen and connecting with your mobile customers on a human level.

There are several ways to do this, through both out-of-the-box SDKs and homegrown solutions; so instead, we’ll focus on how to approach in-app market research rather than the tactics of implementation. Working with leading enterprise apps, bootstrapped single-function apps, and everything in-between, we’ve devised a five-step process for gathering and analyzing the right data to assess Customer Love, or any other exploratory market research questions you may have.

While the first three steps can technically be done in any order, we highly recommend completing them in the order below as each step informs the direction of the next.

The first and arguably most important step is to decide the purpose of your research; what question you want to ask your customers. In this example, we’re looking at Customer Love. We’ve come up with a pretty straightforward question designed to measure sentiment: What do customers think of their in-app experience?

Of course, your research question can be different. You might, for example, be interested in exploring what features your customers would like added to the app. Or you might take a more descriptive approach and seek a greater understanding of your demographic. Whatever your objective is, keep it to a single question. Trying to collect several data points to address multiple research questions will not only muddle your results but also lead to lower response rates and risk annoying customers. After all, your customers don’t want to be stuck filling out a 20-question survey any more than they want to go through the six steps of leaving an app store rating.

The next step is to decide which segment of your mobile app’s audience can best address your research question. These criteria should map directly to your research question. For example, if your objective is to collect negative feedback or suggestions, you might consider polling your less engaged customers in an attempt to decipher why they exhibit low engagement rates. Likewise, if you’re trying to determine which features are the most marketable, you can poll recent installees to explore what they were initially looking for when they downloaded your app.

In our example, our sample is purposefully broad. We’re looking at customer sentiment across the table, and want a random sample of all customers—from first-time customers to our most loyal evangelists. Later on in the analysis step, we can break this sample apart and look for meaningful segmentations and trends at a more microscopic level; but for now, we’re just worried about collecting as much relevant data as possible.

Next, determine your sample size: How many survey responses you need to adequately answer your research question. This is one of the situations where the favored mantra “more is better” doesn’t hold true. Customers launched your app for a specific purpose, and it wasn’t to fill out a survey or share their feedback. With our clients, we recommend being respectful of your mobile customers’ time and distributing the least amount of prompts possible, while still collecting the data points you need.

To gauge Customer Love, we’d pick a larger sample size than the average research question requires, as responses are apt to vary from customer to customer. Furthermore, as we anticipate needing to segment our results by demographic or certain customer attributes, we need to collect enough data to sufficiently fill each segment.

Your research instrument is the medium from which you’ll collect your data. Common instruments include surveys (can be web-based or in-app), open-ended forms, and one-question prompts. Again, your research instrument should drive your choice of medium. Consider which kind of data will provide the most useful feedback: Do you need qualitative feedback, a list of preferences by rank, or a simple yes/no? Each of these maps to a different type of instrument best suited for collecting that data.

Regardless of your choice of instrument, it’s important to ask as few questions as possible (remember, you’re only trying to address the one research question). Just as you’re requesting data from as few people as possible, you want to make the experience as quick and seamless as possible for your respondents so as to maximize your response rate and avoid annoying customers. Likewise, try to avoid any mediums that require customers to leave the app, as these see the lowest response rates and may result in those customers not returning to your app.

Once again, your choices here should flow naturally from your research question and choice of instrument. If you’re going with a survey or questionnaire, take special care to minimize the number of questions you ask. All of the data you collect should be new and pertain to your research question. That is, if you already know a customer’s’ update version, device type, or usage statistics, there should be no reason to request this data in the survey.

For more on choosing the right questions while minimizing common biases, check out our related post, Best Practices in Survey Design.

For our example, we might go with a simple three-question survey. It cuts straight to the point, while collecting all the data we need.

1. On a scale of 1-10, how would you rate your overall satisfaction with this app?
2. How likely are you to recommend this app to a friend?
   1. Very likely
   2. Likely
   3. Unsure
   4. Unlikely
   5. Very unlikely
3. Please rate your level agreement with the following statements, on a scale of 1 (highly disagree) to 10 (highly agree).
   1. I believe [app] deserves my loyalty.
   2. I believe my loyalty for [app] is growing stronger.
   3. I believe [app] does a good job communicating with me.
   4. I believe [app] has all the features I need in an app.
   5. I believe [app] has all the features I’m looking for in an app.

It’s not a perfect survey, but it does the trick. It takes just a minute of our customer’s precious time, and gives us everything we need.

The first question provides a customer’s’ self assessment of satisfaction, a metric necessary to but equal to Customer Love. The second question (the Net Promoter Score) takes this one step further. The app might be ‘good enough’ for the respondent, but is it something to rave about? Is it something they’re confident their friends will see value in? The third question breaks down Customer Love, represented here as loyalty, into different areas to see why the customer loves (or doesn’t love) the app, and what you can improve to grow or gain their love. It also asks the important question: Is their love for your app growing or diminishing?

Once you have integrated your survey and reached your desired sample size, it’s time to close the survey and analyze the results.

The first step we recommend is to sort any qualitative (free response) data, asking questions such as: Are there any common themes? Does anything look particularly urgent? What issue keeps coming up?

By organizing your data in such a manner, you can avoid falling into the trap of letting “the squeaky wheel get the grease;” that is, adding a suggestion to the product roadmap based on one customer. Regardless of how irate or loyal that customer is, don’t let their needs supersede the needs of your less vocal customers by dictating your limited dev time. While you can never please everyone, each development on your roadmap should benefit the majority of your customers, as informed by your newly collected insights, rather than a select minority.

The next step is to dissect your data. This involves looking at each customer not as a data point, but as a person. Using whatever data you have, segment your responses in a meaningful way (by app usage/engagement, device type, gender, etc.). Once you’ve segmented your data, look for any noteworthy differences between groups.

In our example, we might want to know how the time since downloading the app correlates with Customer Love by exploring, “What differentiates an ‘evangelist’ from a ‘user’ in terms of their attitude toward my app?” If we find a difference between the two groups, we’d then hypothesize possible reasons for the disparity and do a second round of research to validate our assumptions. Knowing these differences can help us bridge the gap and convert less engaged customers into fans. This data may also reveal holes in our customer strategy: Are we listening to the needs of all of our customers, or focusing our resources on those at the top of the loyalty pyramid, shown below?

And finally, we want to establish some meaningful benchmarks by exploring how these responses change over time. This comes down to periodically sending out similar surveys, either to the same sample or a different sample. These surveys might be more granular, diving into areas highlighted in the results of the first survey, or condensed forms of the original. This time around, it’s even more important to be cognizant of your customers’ time, especially if they have already been prompted to take a survey. Consider narrowing your sample criteria, reducing your sample size, or prompting your survey in a contextually relevant manner.

Wrapping It Up

Customer Love is at once both the single most important and the single most misunderstood mobile app metrics. Considered a meaningless metric by some and an untested assumption by others, Customer Love might just prove to be your single most important asset. Coming to an understanding this enigmatic concept with our five-step approach to its measurement will unlock a world of opportunities for any app publisher — opportunities to use real insights to drive the product roadmap in a way that maximizes Customer Love; opportunities to assess the sustainability of a business model in an incredibly crowded app store marketplace; and opportunities to build loyalty and spread the love.

Our five-step approach, validated by a collection of leading apps, to collecting and analyzing customers insights in app will not only get you the data you need, but do so in the most efficient, effective, and respectful way possible. For after all, Customer Love is a product of the love you show your customers, and nothing betrays this bond faster than a barrage of poorly conceived requests on your customers’ valuable time.

Whether you’re assessing Customer Love or posing a different research question, we hope this guide provides you with some helpful tips on approaching in-app market research. As always, if you have any questions, we’re all ears.