First equals most important? Order effects in vignette-based measurement

Abstract

A vignette typically describes a hypothetical
situation or object which respondents are asked to judge. The object is
described as having different dimensions, the values of which are
experimentally varied, so that their impact on respondents’ judgments can be
estimated. We examine 1) whether the order in which dimensions are presented
impacts estimates, and 2) under which conditions order effects are mostly
likely. Using data from a web survey of students we analyze several possible
conditions features of the vignette design, characteristics of respondents,
and interactions between these. Our results show that strong order effects can
occur, but only when the vignettes are complex.