An official website of the United States government
A non-response follow-up study by mail in a national sample of U.S. households had five embedded experiments to test the effects of an advance mailing, alternate survey titles, 1- or 2-page questionnaire length, the inclusion or exclusion of political questions on the 1-page questionnaire, and the position of political content on the first or second page of the 2-page questionnaire. None of these design elements affected the payout of escalated postpaid incentives. Advance mailings had no effect on response rate. A short title (National Survey of Households) had a slightly higher response rate than a longer, more descriptive one (National Survey of Households, Families, and Covid-19). Political question content, whether by inclusion, exclusion, or position, had no discernable effect on response, even among prior-study non-respondents. Questionnaire length was inversely related to response: the 2-page questionnaire depressed the overall response rate by 3.7 points (58.5 compared to 54.8 percent, weighted) and depressed response for the critical sample group of prior non-respondents by 6.9 points (36.9 compared to 29.9).
more »
« less
The Visible Cash Effect with Prepaid Incentives: Evidence for Data Quality, Response Rates, Generalizability, and Cost
Abstract A prior study found that mailing prepaid incentives with $5 cash visible from outside the envelope increased the response rate to a mail survey by 4 percentage points compared to cash that was not externally visible. This “visible cash effect” suggests opportunities to improve survey response at little or no cost, but many unknowns remain. Among them: Does the visible cash effect generalize to different survey modes, respondent burdens, and cash amounts? Does it differ between fresh samples and reinterview samples? Does it affect data quality or survey costs? This article examines these questions using two linked studies where incentive visibility was randomized in a large probability sample for the American National Election Studies. The first study used $10 incentives with invitations to a long web questionnaire (median 71 minutes, n = 17,849). Visible cash increased response rates in a fresh sample for both screener and extended interview response (by 6.7 and 4.8 percentage points, respectively). Visible cash did not increase the response rate in a reinterview sample where the baseline reinterview response rate was very high (72 percent). The second study used $5 incentives with invitations to a mail-back paper questionnaire (n = 8,000). Visible cash increased the response rate in a sample of prior nonrespondents by 4.0 percentage points (from 31.5 to 35.5), but it did not increase the response rate in a reinterview sample where the baseline reinterview rate was very high (84 percent). In the two studies, several aspects of data quality were investigated, including speeding, non-differentiation, item nonresponse, nonserious responses, noncredible responses, sample composition, and predictive validity; no adverse effects of visible cash were detected, and sample composition improved marginally. Effects on survey costs were either negligible or resulted in net savings. Accumulated evidence now shows that visible cash can increase incentives’ effectiveness in several circumstances.
more »
« less
- Award ID(s):
- 1835022
- PAR ID:
- 10383031
- Publisher / Repository:
- Oxford University Press
- Date Published:
- Journal Name:
- Journal of Survey Statistics and Methodology
- Volume:
- 11
- Issue:
- 5
- ISSN:
- 2325-0984
- Format(s):
- Medium: X Size: p. 991-1010
- Size(s):
- p. 991-1010
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Despite the growing popularity of digital payment transactions in the United States, most survey participation incentives are still paid through cash or check and then distributed to respondents or potential sample members via direct mail. Though survey researchers have explored alternative incentives, such as e-gift cards, for online samples, there has been no study of electronic cash incentives—specifically paid through mobile pay applications—to date. In this article, we briefly review the literature on incentives used in online surveys and then examine survey incentive payment preferences among respondents using a small, web-based survey of younger adults. Our results suggest a greater preference for cash incentives paid through mobile applications than through direct mail, further highlighting the need for more research on the efficacy of electronically-delivered monetary incentives.more » « less
-
Elite surveys are increasingly common in political science, but how best to motivate participation in them remains poorly understood. This study compares the effect of three treatments designed to increase participation in an online survey of international non-profit professionals: a monetary reward, an altruistic appeal emphasizing the study’s benefits, and a promise to give the respondent access to the study’s results. Only the monetary incentive increased the survey response rate. It did not decrease response quality as measured in terms of straight-lining or skipped questions, although it may have produced a pool of respondents more likely to speed through the survey. The findings suggest that monetary incentives reduce total survey error even in the context of an elite survey, perhaps especially with elite populations frequently contacted by researchers. However, such incentives may not be without trade-offs in terms of how carefully respondents engage with the survey.more » « less
-
The Study of Trauma, Resilience, and Opportunity among Neighborhoods in the Gulf III, hereafter referred to as STRONG III, is a survey conducted among a randomly selected, representative sample of adult residents of 56 counties located in the coastal areas along the Gulf of Mexico, spanning 5 states (Texas, Louisiana, Mississippi, Alabama, and Florida). This is a re-contact study of STRONG I and STRONG II, the data for which are archived on GRIIDC, a Gulf Science Data Repository. The original STRONG I sample comprised 2,520 respondents, where as the re-contact efforts in STRONG III yielded responses from 599 participants in the Gulf region. The survey evaluates a broad range of topics including living conditions, neighborhood satisfaction and safety, social cohesion, neighborhood walkability, home environment, COVID-19 experiences and risks, COVID-19 disruptions to routine behaviors, COVID-19 service impacts, COVID-19 employment impacts, social resources, storm experiences, physical health, alcohol consumption, mental health, healthcare access, trauma, food security, political efficacy, and sociodemographics. mail questionnaire; telephone interview; web-based survey;more » « less
-
null (Ed.)Abstract This study investigates what role, if any, nonresponse plays in inflating survey estimates of religious behavior, using a multimode survey designed to allow estimation of nonresponse bias. A sample of 3,000 Boston-area households drawn from an address-based frame was randomly divided into two subsamples, contacted by mail, and invited to participate in a survey. The first subsample was asked to complete an interactive voice response interview. The second subsample was asked to complete a survey by telephone if a number was available for the address or by personal interview if not. Finally, random samples of nonrespondents were recontacted for a personal interview. Comparison of attendance estimates from initial interviews with nonrespondent interviews within sample segments yields minor or minimal differences that are not statistically significant. Findings suggest that the mechanism generating survey nonresponse is unlikely to be a major cause of bias in religious service attendance estimates in this study.more » « less
