The Experimenter’s Dilemma: Online Survey Experiments and Organizational Processes

Online panels and crowdsourcing platforms have gained widespread use in social science research due to their cost-effectiveness, simplicity, and speed. However, their validity depends on the assumption that online responses reflect real-world decision-making behavior. To test this assumption, we rerun seven prominent field experiments on various forms of discrimination using online survey experiments. Four field experiments examined organizational decisions (e.g., corporate recruiters’ job candidate interview selections), while three examined personal decisions (e.g., individual choice of communication or exchange partners). Varying the details of the survey experimental design, we conducted 23 studies (N = 10,110). Most findings were sobering. There was a significant disparity between online survey experiments and field results in the context of organizational decisions. Traditional data quality controls, such as excluding non-naïve participants, adding attention checks, or recruiting domain experts failed to align online survey findings with real-world outcomes. However, a more positive trend emerged in the realm of individual choices. Online survey experiments consistently mirrored field results in cases of personal decision-making. For example, while online survey experiments did not reflect field findings on racial bias in organizational hiring, they did correspond with results on racial discrimination in individual transactions. Overall, online survey experiments should be used with caution in research on organizational decisions—where individual decisions must often align with the anticipated reactions of others in the organization—but they may be appropriate for personal decision-making—where individuals make decisions for themselves.