More Like this

1. The Effects of a Targeted “Early Bird” Incentive Strategy on Response Rates, Fieldwork Effort, and Costs in a National Panel Study

   https://doi.org/10.1093/jssam/smab042  

   Mcgonagle, Katherine A; Sastry, Narayan; Freedman, Vicki A (February 2022, Journal of Survey Statistics and Methodology)

   Abstract Adaptive survey designs are increasingly used by survey practitioners to counteract ongoing declines in household survey response rates and manage rising fieldwork costs. This paper reports findings from an evaluation of an early-bird incentive (EBI) experiment targeting high-effort respondents who participate in the 2019 wave of the US Panel Study of Income Dynamics. We identified a subgroup of high-effort respondents at risk of nonresponse based on their prior wave fieldwork effort and randomized them to a treatment offering an extra time-delimited monetary incentive for completing their interview within the first month of data collection (treatment group; N = 800) or the standard study incentive (control group; N = 400). In recent waves, we have found that the costs of the protracted fieldwork needed to complete interviews with high-effort cases in the form of interviewer contact attempts plus an increased incentive near the close of data collection are extremely high. By incentivizing early participation and reducing the number of interviewer contact attempts and fieldwork days to complete the interview, our goal was to manage both nonresponse and survey costs. We found that the EBI treatment increased response rates and reduced fieldwork effort and costs compared to a control group. We review several key findings and limitations, discuss their implications, and identify the next steps for future research. 

   more » « less
2. Selection in Surveys: Using Randomized Incentives to Detect and Account for Nonresponse Bias

   https://doi.org/10.1093/restud/rdag003  

   Dutz, Deniz; Huitfeldt, Ingrid; Lacouture, Santiago; Mogstad, Magne; Torgovitsky, Alexander; van_Dijk, Winnie (February 2026, Review of Economic Studies)

   Abstract We show how to use randomized participation incentives to test and account for nonresponse bias in surveys. We first use data from a survey about labour market conditions, linked to full-population administrative data, to provide evidence of large differences in labour market outcomes between survey participants and nonparticipants, differences which would not be observable to an analyst who only has access to the survey data. These differences persist even after correcting for observable characteristics. We then use the randomized incentives in our survey to directly test for nonresponse bias and find evidence of substantial bias. Next, we apply a range of existing methods that account for nonresponse bias and find they produce bounds (or point estimates) that are either wide or far from the ground truth. We investigate the failure of these methods by taking a closer look at the determinants of participation, finding that the composition of participants changes in opposite directions in response to incentives and reminder emails. We develop a model of participation that allows for two dimensions of unobserved heterogeneity in the participation decision. Applying the model to our data produces bounds (or point estimates) that are narrower and closer to the ground truth than the other methods. Our results highlight the benefits of including randomized participation incentives in surveys. Both the testing procedure and the methods for bias adjustment may be attractive tools for researchers who are able to embed randomized incentives into their survey. 

   more » « less
3. The Effects of Nonresponse and Sampling Omissions on Estimates on Various Topics in Federal Surveys: Telephone and IVR Surveys of Address-Based Samples

   https://doi.org/10.2478/jos-2020-0032  

   Fowler, Floyd J.; Brenner, Philip; Roman, Anthony M.; Hargraves, J. Lee (July 2020, Journal of Official Statistics)

   null (Ed.)

   Abstract With declining response rates and challenges of using RDD sampling for telephone surveys, collecting data from address-based samples has become more attractive. Two approaches are doing telephone interviews at telephone numbers matched to addresses and asking those at sampled addresses to call into an Interactive Voice Response (IVR) system to answer questions. This study used in-person interviewing to evaluate the effects of nonresponse and problems matching telephone numbers when telephone and IVR were used as the initial modes of data collection. The survey questions were selected from major US federal surveys covering a variety of topics. Both nonresponse and, for telephone, inability to find matches result in important nonresponse error for nearly half the measures across all topics, even after adjustments to fit the known demographic characteristics of the residents. Producing credible estimates requires using supplemental data collection strategies to reduce error from nonresponse. 

   more » « less
4. “Accepting Evolution Means You Can’t Believe in God”: Atheistic Perceptions of Evolution among College Biology Students

   https://doi.org/10.1187/cbe.19-05-0106  

   Barnes, M. Elizabeth; Dunlop, Hayley M.; Sinatra, Gale M.; Hendrix, Taija M.; Zheng, Yi; Brownell, Sara E. (June 2020, CBE—Life Sciences Education)

   Nehm, Ross (Ed.)

   Although many scientists agree that evolution does not make claims about God/god(s), students might assume that evolution is atheistic, and this may lead to lower evolution acceptance. In study 1, we surveyed 1081 college biology students at one university about their religiosity and evolution acceptance and asked what religious ideas someone would have to reject if that person were to accept evolution. Approximately half of students wrote that a person cannot believe in God/religion and accept evolution, indicating that these students may have atheistic perceptions of evolution. Religiosity was not related to whether a student wrote that evolution is atheistic, but writing that evolution is atheistic was associated with lower evolution acceptance among the more religious students. In study 2, we collected data from 1898 students in eight states in the United States using a closed-ended survey. We found that 56.5% of students perceived that evolution is atheistic even when they were given the option to choose an agnostic perception of evolution. Further, among the most religious students, those who thought evolution is atheistic were less accepting of evolution, less comfortable learning evolution, and perceived greater conflict between their personal religious beliefs and evolution than those who thought evolution is agnostic. 

   more » « less
5. Multisite causal mediation analysis in the presence of complex sample and survey designs and non-random nonresponse

   Qin, X.; Hong, G.; Deutsch, J.; & Bein, E. (January 2019, Journal of the Royal Statistical Society)

   This study provides a template for multisite causal mediation analysis using a comprehensive weighting-based analytic procedure that enhances external and internal validity. The template incorporates a sample weight to adjust for complex sample and survey designs, adopts an IPTW weight to adjust for differential treatment assignment probabilities, employs an estimated nonresponse weight to account for non-random nonresponse, and utilizes a propensity score-based weighting strategy to flexibly decompose not only the population average but also the between-site heterogeneity of the total program impact. Because the identification assumptions are not always warranted, a weighting-based balance checking procedure assesses the remaining overt bias, while a weighting-based sensitivity analysis further evaluates the potential bias related to omitted confounding or to propensity score model misspecification. We derive the asymptotic variance of the estimators for the causal effects that account for the sampling uncertainty in the estimated weights. The method is applied to a re-analysis of the data from the National Job Corps Study. 

   more » « less