The Validity of Qualtrics Panel Data for Research on Video Gaming and Gaming Disorder

Belliveau, Jacob
Soucy, Krystal I.
Yakovenko, Igor

Department of Psychology and Neuroscience, Dalhousie University
Department of Psychiatry, Dalhousie University

This research was not supported by any funding source. There are no conflicts of interest to declare of any kind for any of the authors.

All authors contributed in a significant way to the article, and all authors have read and approved the final article.

Jacob Belliveau played supporting role in conceptualization and equal role in data curation, formal analysis, funding acquisition, investigation, methodology, project administration, resources, software, supervision, validation, visualization, writing of original draft and writing of review and editing. Krystal I. Soucy played equal role in conceptualization, methodology, project administration and writing of review and editing. Igor Yakovenko played lead role in conceptualization, funding acquisition, investigation, methodology, project administration, resources and supervision, supporting role in writing of original draft and writing of review and editing and equal role in data curation, formal analysis, software, validation and visualization.

Study materials and analysis code are available upon request from the corresponding author.

This study was not preregistered.

Igor Yakovenko, Department of Psychology and Neuroscience, Dalhousie University, 1355 Oxford Street, Halifax, NS B3H 4R2, Canada [email protected]

Received November 10, 2021

revised February 26, 2022

Accepted April 6, 2022

Experimental and Clinical Psychopharmacology 30(4):p 424-431, August 2022. | DOI: 10.1037/pha0000575

Crowdsourced samples are increasing in popularity, particularly within psychological and addictive behaviors research. The trend has resulted in significant interest in the use of panel samples for the examination of behavioral and substance addictions. One newer panel platform, Qualtrics, has seen an increase in usage in recent years despite lack of research examining the validity of Qualtrics-produced data for addictive behaviors. The goal of the present study was to evaluate the validity of Qualtrics-obtained data for the most recently classified behavioral addiction—video gaming. The evaluation compared a Qualtrics-recruited video gamers sample (n = 586) to traditional community (n = 108) and student (n = 217) samples on demographics and key outcomes relevant to gaming disorder researchers (average playtime, frequency of gaming, and gaming disorder risk scores) to evaluate the generalizability of Qualtrics panel data. The results revealed that Qualtrics samples were comparable to a traditionally recruited community sample, but different from a student sample on gaming frequency (p < .001) and risk for gaming disorder (p < .001). The Qualtrics sample also had longer durations of average gaming time relative to the student sample (p = .01), with some differences in demographics between the all three sources of recruitment. The findings suggest that Qualtrics may provide a suitable method of convenience panel recruitment, generalizable to the broader North American community, for research examining video gaming behaviors and gaming disorder.