Improving reports to sensitive survey questions

An experimental project designed to test whether changes to questionnaire design will remove bias and improve answers to sensitive questions.

Background

Social surveys often ask questions on sensitive topics. Getting answers which are thoughtful and accurate can be difficult. Often, where sensitive topics are concerned respondents may give answers which are more positive than their true state (for example when asked how much alcohol they consume). The result is that the survey tends to underestimate socially unpleasant behaviour.

At the same time, recent research suggests that respondents do not always fully read through a list of response options, especially where respondents rush their answers. In addition respondents tend to align themselves with the top options in a vertical list, known as the primacy effect.

By inverting a vertical scale so that the most negative response is at the top of the scale respondents will be aware of the negative options and will consider these before moving down the scale. At the same time it is when respondents are highly motivated they take more care in completing survey questions and this leads to higher quality responses.

Project aims

The aims of this experimental project are:

1. To test whether a top negative scale is less likely to obtain socially desirable
   answers than questions with a top positive scale.
2. To see whether motivating respondents to respond carefully increases the reporting of socially undesirable answers and to see whether this effect varies with the type of scale used
3. To assess the effect of the above on respondents who spend less time on survey questions

Data was drawn from the result of a randomised experiment using 90 volunteers who were asked to complete a questionnaire using questions selected from the Understanding Society adult self-completion questionnaire. The experiment consisted of two related tests.

In the first test, half the respondents were asked to complete the questionnaire using a response scale which starts with positive category at the top (top positive condition). The other half will complete the same questionnaire using a reversed version of the scale (top-negative condition).

The second test is related to motivation. During the questionnaire, half the respondents were asked to make a special effort to take more care when responding to the survey questions.

The data was analysed at question level with account for clustering within respondents in Stata.