Paradata versus self-reports for the collection of technical data about smartphones: trade-offs between completeness and accuracy of data on smartphone operating system versions

Summary

Smartphones play a central role in everyday life and offer significant potential for survey data collection. However, their usefulness may be constrained by the smartphone’s operating system (OS) and its version, which determine app compatibility. This article investigates methods for accurately determining smartphone OS version, an important factor when evaluating device compatibility for data collection tools. We compare three approaches: (1) passive collection of paradata (user agent strings; UASs) from web respondents, (2) self-reported smartphone make and model matched to OS version information from an online database, and (3) direct self-reporting of OS version by respondents following step-by-step instructions. Using data from the UK Household Longitudinal Study COVID-19 web survey, a probability sample of UK households, we assess the completeness and accuracy of each method. Our findings suggest the paradata were in some respects inferior to the methods based on self-reports. First, the UAS data only provided valid smartphone OS version data for about 50 percent of respondents; the other 50 percent did not use a smartphone to complete the web survey (compared to 90 percent and 71 percent of valid cases based on methods (2) and (3)). Second, the subset of respondents who completed the survey using their smartphone was not representative of the full set of respondents, while the respondents for whom OS version data were obtained from self-report methods were broadly representative. Third, the UAS data under-represented older OS versions. Respondents with older smartphones seem unlikely to use them to complete the survey. Our findings also show differences between OSes, suggesting it was easier for iPhone users to provide relevant and valid information than for Android users: they were more likely to report a valid make and model that could be linked to technical data, to report the OS version, and to use their smartphone to complete the survey.