Assessing the Effect of Data Collection Mode on Measurement

Abstract

In order to make informed decisions about suitable data collection designs, survey practitioners need to evaluate and quantify the potential impact of mode on data quality. Generating this type of information is however a difficult task. We review the methods typically used to assess the effects of mode on measurement and data comparability and then discuss some of the challenges, including 1) the need to avoid confounding effects, 2) the sensitivity of conclusions to methods of analysing experimental mode comparison data, 3) the difficulty of assessing whether measurement differences matter in practice, and 4) the assessment of which mode provides better measurement. We illustrate the challenges and implications of mixed modes research for survey design with examples from mode experiments conducted in the context of the European Social Survey (ESS). The paper concludes with implications for mixed mode research.