An official website of the United States government
A non-response follow-up study by mail in a national sample of U.S. households had five embedded experiments to test the effects of an advance mailing, alternate survey titles, 1- or 2-page questionnaire length, the inclusion or exclusion of political questions on the 1-page questionnaire, and the position of political content on the first or second page of the 2-page questionnaire. None of these design elements affected the payout of escalated postpaid incentives. Advance mailings had no effect on response rate. A short title (National Survey of Households) had a slightly higher response rate than a longer, more descriptive one (National Survey of Households, Families, and Covid-19). Political question content, whether by inclusion, exclusion, or position, had no discernable effect on response, even among prior-study non-respondents. Questionnaire length was inversely related to response: the 2-page questionnaire depressed the overall response rate by 3.7 points (58.5 compared to 54.8 percent, weighted) and depressed response for the critical sample group of prior non-respondents by 6.9 points (36.9 compared to 29.9).
more »
« less
The Visible Cash Effect with Prepaid Incentives: Evidence for Data Quality, Response Rates, Generalizability, and Cost
Abstract A prior study found that mailing prepaid incentives with $5 cash visible from outside the envelope increased the response rate to a mail survey by 4 percentage points compared to cash that was not externally visible. This “visible cash effect” suggests opportunities to improve survey response at little or no cost, but many unknowns remain. Among them: Does the visible cash effect generalize to different survey modes, respondent burdens, and cash amounts? Does it differ between fresh samples and reinterview samples? Does it affect data quality or survey costs? This article examines these questions using two linked studies where incentive visibility was randomized in a large probability sample for the American National Election Studies. The first study used $10 incentives with invitations to a long web questionnaire (median 71 minutes, n = 17,849). Visible cash increased response rates in a fresh sample for both screener and extended interview response (by 6.7 and 4.8 percentage points, respectively). Visible cash did not increase the response rate in a reinterview sample where the baseline reinterview response rate was very high (72 percent). The second study used $5 incentives with invitations to a mail-back paper questionnaire (n = 8,000). Visible cash increased the response rate in a sample of prior nonrespondents by 4.0 percentage points (from 31.5 to 35.5), but it did not increase the response rate in a reinterview sample where the baseline reinterview rate was very high (84 percent). In the two studies, several aspects of data quality were investigated, including speeding, non-differentiation, item nonresponse, nonserious responses, noncredible responses, sample composition, and predictive validity; no adverse effects of visible cash were detected, and sample composition improved marginally. Effects on survey costs were either negligible or resulted in net savings. Accumulated evidence now shows that visible cash can increase incentives’ effectiveness in several circumstances.
more »
« less
- Award ID(s):
- 1835022
- PAR ID:
- 10383031
- Publisher / Repository:
- Oxford University Press
- Date Published:
- Journal Name:
- Journal of Survey Statistics and Methodology
- Volume:
- 11
- Issue:
- 5
- ISSN:
- 2325-0984
- Format(s):
- Medium: X Size: p. 991-1010
- Size(s):
- p. 991-1010
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Despite the growing popularity of digital payment transactions in the United States, most survey participation incentives are still paid through cash or check and then distributed to respondents or potential sample members via direct mail. Though survey researchers have explored alternative incentives, such as e-gift cards, for online samples, there has been no study of electronic cash incentives—specifically paid through mobile pay applications—to date. In this article, we briefly review the literature on incentives used in online surveys and then examine survey incentive payment preferences among respondents using a small, web-based survey of younger adults. Our results suggest a greater preference for cash incentives paid through mobile applications than through direct mail, further highlighting the need for more research on the efficacy of electronically-delivered monetary incentives.more » « less
-
While paper mail-based surveys avoid much of the risk of bots and fraudulent data, they suffer from lower response rates and ever-inflating material and logistical costs. In response, there is a nascent, but growing literature investigating a lower cost, explicitly anonymous, mail-based survey distribution method called Every Door Direct Mail (EDDM). This study contributes to this growing body of literature by using EDDM to disseminate a sequential mixed-mode census-style survey that meets best use-case recommendations per past research. We make several design alterations to elicit higher response rates including using an outer envelope and cash incentive. The survey, distributed near large-scale solar developments in three urban Michigan communities (~1554 households), was geographically based, targeted a specific and limited population, and covered the potentially sensitive topic of local solar development, which may have also led to a higher response rate. The survey achieved an overall response rate of 10.2% with 158 complete surveys returned, demonstrating this work’s usefulness, use case, and flexibility.more » « less
-
Elite surveys are increasingly common in political science, but how best to motivate participation in them remains poorly understood. This study compares the effect of three treatments designed to increase participation in an online survey of international non-profit professionals: a monetary reward, an altruistic appeal emphasizing the study’s benefits, and a promise to give the respondent access to the study’s results. Only the monetary incentive increased the survey response rate. It did not decrease response quality as measured in terms of straight-lining or skipped questions, although it may have produced a pool of respondents more likely to speed through the survey. The findings suggest that monetary incentives reduce total survey error even in the context of an elite survey, perhaps especially with elite populations frequently contacted by researchers. However, such incentives may not be without trade-offs in terms of how carefully respondents engage with the survey.more » « less
-
With increasing incidence of harmful algal blooms (HABs) and their associated illnesses such as ciguatera poisoning (CP), there is need for educating current and future clinicians. This study sought to assess medical and physician assistant (PA) students' knowledge, attitudes, and beliefs toward HABs and their related illnesses. A survey of medical students and PA students at the University of South Alabama (USA) was conducted using an online questionnaire on climate change, HABs/associated illnesses, and CP. Response rate was calculated using fully executed questionnaires. Frequency data was utilized for demographics and knowledge-based questions; stratified analysis was used for associations. Three hundred three medical students received the questionnaire; 27% (n= 81) completed it. One hundred seventy-one PA students received the questionnaire; 19% (n= 33) completed the survey. These students were demographically representative of their student bodies. Out of 10 questions regarding knowledge of HABs, the percent correct was 30.7% for medical students and 20.3% for PA students. 34.6% of medical students and 47.1% of PA students had never heard of HABs. 90.1% of medical students and 84.8% of PA students believed climate change will impact human health in the future and more knowledge is needed about the relationship between health and climate change. Seventy six percentage of medical students and 51.6% of PA students expressed that HAB education should be part of health professional school curriculum. Future clinicians have little knowledge of HABs and their associated illnesses despite recognizing that climate change is a vital health issue. Medical and health professional schools should consider adding HAB education for future clinicians.more » « less
