Evaluating the statistical realism of LLM-generated social science data

Summary

Large language models (LLMs) enable the generation of data that could potentially be analyzed for social research. While the need for assessing the validity of such AI-generated data is widely recognized, we do not yet have a coherent framework for assessment. This work introduces SSDataBench, a systematic benchmark of population-level statistical realism in LLM-generated social science data. By evaluating 15 LLMs in generating data across five types of statistical patterns against real data from four longitudinal and three cross-sectional social science surveys, we quantify how well LLM-generated data reproduce key statistical moments in actual surveys and reveal representational limitations in current LLMs under sparse conditioning settings. We further provide preliminary evidence that domain-specific training can significantly improve population-level statistical realism.