Behavioural economics

Five ways behavioural insights can improve survey design

| Blog
  • Behavioural economics
BETA blog

Surveys and questionnaires are indispensable tools for research. At their best, they are a quick, efficient way to gather feedback and opinions from a study population, and they can be an enormously valuable tool for re-designing government processes to better serve people. However, people are now bombarded with market research surveys in their everyday life, like customer satisfaction surveys in emails and popup surveys on websites. Surveys are now everywhere, and people are increasingly annoyed by them: the Pew Research Centre has found that response rates for their public opinion surveys fell dramatically from 36% in 1997 to just 9% in 2012. This decline poses a problem for researchers and policymakers, as non‑response not only decreases the amount of data gathered, it may also bias the results.

Leveraging behavioural insights can mitigate the effects of these challenges.

Survey response rates, US public opinion surveys, 1997-2012
 1997
%		2000
%		2003
%		2006
%		2009
%		2012
%
Contact rate
(% of households in which an adult was reached)		907779737262
Cooperation rate
(% of households contacted that yielded an interview)		434034312114
Response rate
(% of households sampled that yielded an interview)		36282521159

Notes: Rates computed according to American Association for Public Opinion Research (AAPOR) standard definitions for CON2, COOP3 and RR3. Rates are typical for surveys conducted in each year.

Source: Pew Research Centre (2012).

Keeping behavioural insights in mind throughout your survey design process can help to increase response rates and improve the quality of the data you gather, while reducing the burden on respondents. We think of this as respondent-centred survey design—it’s crucial to remember that these are real people, who live incredibly busy lives, giving their time to take your survey.

In this post, we’ve gathered 5 ways you can use behavioural insights to design better surveys and maximise your response rates.

First, be deliberate about how you personalise your survey invitations. Wherever possible, personalise this invite: convey the message that this specific person’s input is important. Marketers have found that personalised invites increase response rates, as have other researchers. Here in Australia, emails recruiting participants for a study about breast cancer were 50% more likely to get a response when they included the woman’s name. If you’re targeting a specific population or group, use that in your invite. Making group affiliations salient—“we’re interested in the experience of first-generation university students at the Australian National University” versus “we’re interested in your experiences”—can also help to signal the relevance of your survey to their interests. Letting your study population know they’re part of a smaller group also avoids the diffusion of responsibility, where a potential respondent might assume that since enough other people will probably respond, they don’t have to.

Second, framing matters too. We know from Kahneman and Tversky’s prospect theory that people’s emotional response to potential losses is greater than to equally-sized potential gains. This is an insight known as loss aversion, and can be applied to research invitations. In recruiting surveyed participants for a follow-up interview, researchers compared gain framing and loss framing:

  1. (Gain frame) “The information you’ve already provided to us will be a lot more valuable if you complete the second interview,” or
  2. (Loss frame) “Unfortunately, the information you’ve already provided to us will be much less valuable unless you complete the second interview.”

The results? The loss framing saw an 87.5% response rate, compared to 77.9% in the gain framing condition.

Third, understand that people often fail to follow through on their intentions to fill out your survey, or procrastinate if the invite arrives at an inconvenient time: leverage behavioural insights to make it easy for people to participate. In one study, researchers used behavioural insights to design a survey invitation that condensed several possible steps into a single, clear action to take, within a specific time frame. It generated 53% more call-ins to participate in the study than the original letter, 7.2% compared to 4.7%. Following the initial invitation, reminders can be incredibly effective in encouraging responses from people who may have procrastinated the first time. In another study, reminders increased the response rate by over 500%, from 6.2% to 31.2%!

Figure: Impact of reminders on survey response rate

This figure shows the impact of reminders on survey response rates. The initial response rate was 6.2%. Three successive reminders increased the response rate to 16.5%, then 25.1 and finally 31.2%.

Source: Van Mol (2017, p322)

Fourth, since thoughtfully answering survey questions involves mental effort, people often seek to satisfice (a combination of “satisfy” and “suffice”). In surveys, when faced with cognitively demanding questions, Krosnik’s research suggests that respondents seek mental shortcuts and use satisficing behaviours like: choosing the first reasonable or acceptable answer, simply agreeing, just picking the same answer for a list of questions (rather than considering each individually, also termed “straightlining”), choosing “don’t know” answers when available, skipping items, or just plugging in random answers. Similarly, researchers have found that towards the end of a survey, people are more likely to skip questions, spend less time answering them and write fewer responses to open-ended questions. All of this can seriously undermine your data.

What can be done? For starters, strive to keep surveys short and sweet, for the sake of survey completion and answer quality. More generally, researchers believe the likelihood of satisficing is related to ability, motivation, and task difficulty. The thoughtfulness that goes into your survey design can influence both motivation and task difficulty: your goal should be to maximise respondent motivation, and minimise task difficulty. Explain the purpose and value of your study, ask respondents to think carefully, write questions in simple, clear language, and ask respondents to evaluate only one thing at a time where possible (comparisons are more cognitively demanding).

Finally, embrace rigorous evaluation! Behavioural scientists use rigorous evaluation and randomised controlled trials in other parts of their work, and should do the same in their surveys! When timing and funding permits, a pilot study with embedded A/B tests can help to determine exactly what invitation message, framing, incentives, phrasing, and questionnaire length get the best response rates in your particular context, which can inform the final survey instrument sent out to your study population.

In 2017, BETA partnered with the Australian Department of Social Services to test the impact of behaviourally-informed changes to their surveys on completion rates. We found that including progress updates in the form of a gradually completing pie-chart, with text encouragements, increased survey completion rates from 84.7% to 91.5%. In this case we were able to use a rigorous evaluation to find concrete evidence that the new survey design was more effective.

Emma Klugman is a statistician and education policy researcher working on her PhD at Harvard. Gabe Cederberg is an undergraduate student at Harvard studying government and economics. Emma and Gabe were interns at BETA in 2019.

References

General

Personalisation

Framing

Procrastination and follow-through

Satisficing

Explore more blogs and podcasts