7 Steps to Creating Effective In-App Surveys

The following is Part 2 of our three-part guide on mobile research and in-app surveys.

• Part 1 – Getting Started With Mobile Research: Why It’s Important
• Part 2 – 7 Steps to Creating Effective In-App Surveys
• Part 3 – Mobile Research Best Practices

In working with thousands of mobile developers on collecting mobile customer insights, we’ve learned a thing or two about what works – and what doesn’t work – when it comes to designing an effective in-app survey. We believe mobile research is an incredibly powerful tool for improving communication with your customers, and have made it our mission since day one to help brands create Customer Love.

We’ve therefore highlighted the steps of one of our favorite survey design roadmaps to help you think about what’s involved in creating surveys that will both deliver the data you need and enhance the overall customer experience.

1. Pick Your Survey Tool

We have an obvious bias here, but the very first thing you need to consider is your choice of survey tool. This includes both the decision of whether to use an in-app survey (like Apptentive) or a web survey (like SurveyMonkey) and the decision of whether to make your survey platform in-house or seek out a third-party solution.

We’ve done our best to highlight the pros and cons of each of these options below:

2. Determine Your Research Question

Before jumping into designing your survey on your chosen tool, take some time to identify a single concrete research question that you hope to answer with the results of your survey. This will provide a benchmark for your data analysis and can help to keep the survey short and concise if you constrain yourself to asking only those questions necessary for addressing your research question. (We’ll provide more on why shorter is better when it comes to in-app surveys in Part 3 of this guide.)

Research questions are generally classified as either attitudinal, behavioral, demographic, or technical. An example of a research question for each of these categories can be found below:

• Attitudinal: How do new users like your app?
• Behavioral: How do users interact with your app? What are the most common use cases?
• Demographic: Which age bracket is your app most popular with?
• Technical: How can this app be improved?

Once you have carefully selected your research question, several of the remaining steps will come naturally – including your target audience and the type of data you need to collect.

3. Identify Your Survey Audience

It’s now time to determine which customers are best suited to honestly and accurately address your research question. Once you’ve identified the customer segment, you can begin to seek out pain points within the app where the survey will provide the most value to both the customer and your team.

Common mobile research samples include:

A Sample Generated With Event-based Targeting

Our Recommendation!

Delivering your in-app survey to customers who engage in pre-specified ‘mobile moments,’ i.e.:

A Random Sample
Randomly choose a percentage of your mobile customers to survey.

A Sample of New Users
Customers prompted during their first time using the app.

A Sample of Loyal Users
Prompted the n-th time a customer opens the app or a few months after installing and regularly using the app.

An Opt-In Sample
Instead of identifying sampling criteria, let your customers proactively navigate your menu to find ways to voice their feedback.

4. Design Around the Research Question

When it comes to writing the actual questions for your in-app survey, start by asking yourself what sort of data best addresses your research question. Some research questions (particularly attitudinal) lend themselves to quantitative questions and rating scales. Others (particularly technical) are best addressed with qualitative, open-ended questions.

Survey questions can be classified as open-ended, closed-ended, or mixed. We’ve done our best below to address the use cases of each classification and their ability to address your anticipated needs:

Open-ended (i.e.: Providing a textbox for the respondent to type his or her answer)

• Exploratory in nature
• Less likely to result in bias from leading questions/response choices
• Provides qualitative responses similar to a focus group
• Can be time-consuming to answer, particularly when using a mobile device, leading to lower response rates

Closed-ended (i.e.: Multiple choice questions and rating scales with pre-defined response choices)

• Can provide both qualitative and quantitative responses
• Questions typically take less time to answer and experience higher response rates
• Questions and response construction requires more care remove bias

Mixed (i.e.: A multiple choice question with a fill-in ‘Other’ option)

• Allows room to write in answers that were not considered when the survey was designed – may uncover new customer needs/sentiments

For more on quantitative vs. qualitative questions, and their use cases, check out this post by our friends at SurveyMonkey.

5. Integrate Into the Mobile Experience

Now that you’ve determined your audience, the next step is to identify where and how to reach them within your app. If you’re using event-based targeting, seek out mobile moments in the customer journey – points of engagement where customers may feel happy, frustrated, or lost. Concentrating on these moments allows you to hone into a single part of the customer experience and can lead to more actionable results if the feedback directly relates to the point of engagement.

When prompting your in-app survey, be careful to not interrupt or annoy customers in the middle of a task. For example, if you want to ask about the in-app shopping experience, prompt your survey after checkout instead of after a customer adds an item to his or her cart. Remember, your survey should be designed to add value to both you and your customer.

If you’re targeting new customers, give them time to try out the app before prompting your survey. A ratings or feedback request upon first log-in is a sure way to create a negative first impression.

And finally, if you’re using an opt-in sample without any sort of prompt, insert a link to your survey into a logical place in your app’s navigation menu – such as the Help or Contact section.

6. Pre-Test Your In-App Survey

Before making your survey live, it’s important to pilot the survey internally or with a small sample of customers. This allows you to ensure that your survey is working as intended and identify any weaknesses or ambiguity.

While testing your survey, check that the following items are in order:

• Questions proceed in the intended order, and responses are mapped to the appropriate next question or prompt if conditional branching and skip logic are in place
• Questions are easily understood by those unfamiliar with the survey
• Response options are clear and do not contain unintentional overlap
• Optional and required fields are appropriately coded
• Responses are accurately collected upon completion

After you’ve tested your survey and are satisfied with the results, we’d recommend resetting the response count so that the test data does not interfere with your actual results. You’re ready to push the survey out to your intended audience.

7. Analyze the Results

By this point, you’ve launched your survey and have collected a statistically significant number of responses. Now it’s time to dig into the results of your labor.

Data analysis should be a straightforward process for those quantitative questions directly mapped to your research question. We recommend going through each question individually and asking yourself whether each result confirms or rejects your hypothesized answer to your own research question: Are the result of customer rating scales consistent with what you’re seeing in app store ratings? Has your latest version update improved customer sentiment? Do customers feel comfortable navigating your interface and new features?

In the case of qualitative questions, you may need to do a little more work to get meaningful and generalizable data. For open response questions, we recommend categorizing the responses into themes – such as feature requests, usage or interface questions, customer testimonials, and customer complaints. You can then search through your results for the most common trends and begin to incorporate these insights into your product roadmap and QA efforts.

As a last step, decide whether you want to keep the survey running or close it. This is often a matter of saturation (if you already have the sample size you need and additional responses provide little value) and relevancy (is your research question still a priority, or is there a new question worth asking in a separate survey?).

We hope these seven steps will help kickstart your in-app survey design. Let us know if there’s anything we can do to help you create your next survey and keep us updated on how customer insights shape your app.

Continue on to Part 3 of our guide on in-app surveys and mobile research, or preview our section on best practices on Slideshare: