7 Steps to Creating Effective In-App Surveys

By: Alex Walz. January 9, 2015

Share:

The following is Part 2 of our three-part guide on mobile research and in-app surveys.

• Part 1 – Getting Started With Mobile Research: Why It’s Important
• Part 2 – 7 Steps to Creating Effective In-App Surveys
• Part 3 – Mobile Research Best Practices

In working with thousands of mobile developers on collecting mobile customer insights, we’ve learned a thing or two about what works – and what doesn’t work – when it comes to designing an effective in-app survey. We believe mobile research is an incredibly powerful tool for improving communication with your customers, and have made it our mission since day one to help brands create Customer Love.

We’ve therefore highlighted the steps of one of our favorite survey design roadmaps to help you think about what’s involved in creating surveys that will both deliver the data you need and enhance the overall customer experience.

1. Pick Your Survey Tool

We have an obvious bias here, but the very first thing you need to consider is your choice of survey tool. This includes both the decision of whether to use an in-app survey (like Apptentive) or a web survey (like SurveyMonkey) and the decision of whether to make your survey platform in-house or seek out a third-party solution.

We’ve done our best to highlight the pros and cons of each of these options below:

Using an In-App Survey Tool Pros: Allows for a frictionless customer experience as customers never have to leave the app Dramatically higher response rates when compared to web surveys Easy to integrate into points of engagement within the app (‘mobile moments’) Cons: May require additional engineering time to integrate into your app

Using a Web Survey Tool Pros: Several very established survey and analytic tools available to choose from Cons: Lower response rates Requires your customers to leave the app to take the survey Branding and the customer experience are often less cohesive between the app and the externally hosted survey

Using an In-House Solution Pros: Unrivaled flexibility Cons: Very resource-intensive in terms of developer time and money Necessary testing of, and updates to, the survey tool may take time away from your product roadmap

Using a Third-Party Solution Pros: Turnkey solutions available with reporting tools and dashboards Access to existing documentation on best practices for customer success Cons: Requires some developer time, depending on ease of SDK integration

2. Determine Your Research Question

Before jumping into designing your survey on your chosen tool, take some time to identify a single concrete research question that you hope to answer with the results of your survey. This will provide a benchmark for your data analysis and can help to keep the survey short and concise if you constrain yourself to asking only those questions necessary for addressing your research question. (We’ll provide more on why shorter is better when it comes to in-app surveys in Part 3 of this guide.)

Research questions are generally classified as either attitudinal, behavioral, demographic, or technical. An example of a research question for each of these categories can be found below:

• Attitudinal: How do new users like your app?
• Behavioral: How do users interact with your app? What are the most common use cases?
• Demographic: Which age bracket is your app most popular with?
• Technical: How can this app be improved?

Once you have carefully selected your research question, several of the remaining steps will come naturally – including your target audience and the type of data you need to collect.

3. Identify Your Survey Audience

It’s now time to determine which customers are best suited to honestly and accurately address your research question. Once you’ve identified the customer segment, you can begin to seek out pain points within the app where the survey will provide the most value to both the customer and your team.

Common mobile research samples include:

A Sample Generated With Event-based Targeting

Our Recommendation!

Delivering your in-app survey to customers who engage in pre-specified ‘mobile moments,’ i.e.:

• The third time a customer uses the Search function
• The first time a customer shares content via the app
• After a customer updates to a new version
• After a customer uses a new/beta feature

Advantages: Can be used to address narrow, feature-specific research questions Can be used to refine and beta test new versions and rollouts Respondents inherently have familiarity with the app event they’re providing feedback on

Disadvantages: Resource-intensive, unless using third-party mobile engagement solutions since activity is a necessary condition of the sampling selection

A Random Sample
Randomly choose a percentage of your mobile customers to survey.

Advantages: Diversity of responses makes the data highly representative of your overall audience Surveys can be prompted even if mobile analytic capabilities do not collect information on customers and customer activity

Disadvantages: Sample may be too broad to address narrow research questions that require a high level of familiarity with the app Responses may be too generalized to uncover trends based on customers’ familiarity with the app, device used to access the app, etc.

A Sample of New Users
Customers prompted during their first time using the app.

Advantages: Allows you to collect information, unbiased by existing loyalty, that can be used to improve the customer experience and retention

Disadvantages: Requesting customer information from first-time users may create a negative initial experience with the app, and has been shown to be a leading factor in app exits

A Sample of Loyal Users
Prompted the n-th time a customer opens the app or a few months after installing and regularly using the app.

Advantages: Can be used to uncover which features loyal users find the most valuable / what is bringing them back to your app Can be used to gather suggestions from those already familiar with those apps – and customers can be further delighted if those suggestions are acted on

Disadvantages: Responses may have an upward bias and have difficulty capturing equally important negative experiences with the app

An Opt-In Sample
Instead of identifying sampling criteria, let your customers proactively navigate your menu to find ways to voice their feedback.

Advantages: Survey can be easily integrated as a link in your app’s navigation with no third party tools or prompts Least intrusive – avoids potentially annoying notifications and prompts

Disadvantages: Can lead to selection bias as those who opt-in may not share fundamentally different views/attitudes than those who do not opt-in Opt-ins have the lowest response rate and the survey link can be difficult for potential respondents to find, making it hard to create open dialog with customers

4. Design Around the Research Question

When it comes to writing the actual questions for your in-app survey, start by asking yourself what sort of data best addresses your research question. Some research questions (particularly attitudinal) lend themselves to quantitative questions and rating scales. Others (particularly technical) are best addressed with qualitative, open-ended questions.

Survey questions can be classified as open-ended, closed-ended, or mixed. We’ve done our best below to address the use cases of each classification and their ability to address your anticipated needs:

Open-ended (i.e.: Providing a textbox for the respondent to type his or her answer)

• Exploratory in nature
• Less likely to result in bias from leading questions/response choices
• Provides qualitative responses similar to a focus group
• Can be time-consuming to answer, particularly when using a mobile device, leading to lower response rates

Closed-ended (i.e.: Multiple choice questions and rating scales with pre-defined response choices)

• Can provide both qualitative and quantitative responses
• Questions typically take less time to answer and experience higher response rates
• Questions and response construction requires more care remove bias

Mixed (i.e.: A multiple choice question with a fill-in ‘Other’ option)

• Allows room to write in answers that were not considered when the survey was designed – may uncover new customer needs/sentiments

For more on quantitative vs. qualitative questions, and their use cases, check out this post by our friends at SurveyMonkey.

5. Integrate Into the Mobile Experience

Now that you’ve determined your audience, the next step is to identify where and how to reach them within your app. If you’re using event-based targeting, seek out mobile moments in the customer journey – points of engagement where customers may feel happy, frustrated, or lost. Concentrating on these moments allows you to hone into a single part of the customer experience and can lead to more actionable results if the feedback directly relates to the point of engagement.

When prompting your in-app survey, be careful to not interrupt or annoy customers in the middle of a task. For example, if you want to ask about the in-app shopping experience, prompt your survey after checkout instead of after a customer adds an item to his or her cart. Remember, your survey should be designed to add value to both you and your customer.

If you’re targeting new customers, give them time to try out the app before prompting your survey. A ratings or feedback request upon first log-in is a sure way to create a negative first impression.

And finally, if you’re using an opt-in sample without any sort of prompt, insert a link to your survey into a logical place in your app’s navigation menu – such as the Help or Contact section.

6. Pre-Test Your In-App Survey

Before making your survey live, it’s important to pilot the survey internally or with a small sample of customers. This allows you to ensure that your survey is working as intended and identify any weaknesses or ambiguity.

While testing your survey, check that the following items are in order:

• Questions proceed in the intended order, and responses are mapped to the appropriate next question or prompt if conditional branching and skip logic are in place
• Questions are easily understood by those unfamiliar with the survey
• Response options are clear and do not contain unintentional overlap
• Optional and required fields are appropriately coded
• Responses are accurately collected upon completion

After you’ve tested your survey and are satisfied with the results, we’d recommend resetting the response count so that the test data does not interfere with your actual results. You’re ready to push the survey out to your intended audience.

7. Analyze the Results

By this point, you’ve launched your survey and have collected a statistically significant number of responses. Now it’s time to dig into the results of your labor.

Data analysis should be a straightforward process for those quantitative questions directly mapped to your research question. We recommend going through each question individually and asking yourself whether each result confirms or rejects your hypothesized answer to your own research question: Are the result of customer rating scales consistent with what you’re seeing in app store ratings? Has your latest version update improved customer sentiment? Do customers feel comfortable navigating your interface and new features?

In the case of qualitative questions, you may need to do a little more work to get meaningful and generalizable data. For open response questions, we recommend categorizing the responses into themes – such as feature requests, usage or interface questions, customer testimonials, and customer complaints. You can then search through your results for the most common trends and begin to incorporate these insights into your product roadmap and QA efforts.

As a last step, decide whether you want to keep the survey running or close it. This is often a matter of saturation (if you already have the sample size you need and additional responses provide little value) and relevancy (is your research question still a priority, or is there a new question worth asking in a separate survey?).

We hope these seven steps will help kickstart your in-app survey design. Let us know if there’s anything we can do to help you create your next survey and keep us updated on how customer insights shape your app.

Continue on to Part 3 of our guide on in-app surveys and mobile research, or preview our section on best practices on Slideshare:

Mobile research best practices from Apptentive