More Like this

1. Selection in Surveys: Using Randomized Incentives to Detect and Account for Nonresponse Bias

   https://doi.org/10.1093/restud/rdag003  

   Dutz, Deniz; Huitfeldt, Ingrid; Lacouture, Santiago; Mogstad, Magne; Torgovitsky, Alexander; van_Dijk, Winnie (February 2026, Review of Economic Studies)

   Abstract We show how to use randomized participation incentives to test and account for nonresponse bias in surveys. We first use data from a survey about labour market conditions, linked to full-population administrative data, to provide evidence of large differences in labour market outcomes between survey participants and nonparticipants, differences which would not be observable to an analyst who only has access to the survey data. These differences persist even after correcting for observable characteristics. We then use the randomized incentives in our survey to directly test for nonresponse bias and find evidence of substantial bias. Next, we apply a range of existing methods that account for nonresponse bias and find they produce bounds (or point estimates) that are either wide or far from the ground truth. We investigate the failure of these methods by taking a closer look at the determinants of participation, finding that the composition of participants changes in opposite directions in response to incentives and reminder emails. We develop a model of participation that allows for two dimensions of unobserved heterogeneity in the participation decision. Applying the model to our data produces bounds (or point estimates) that are narrower and closer to the ground truth than the other methods. Our results highlight the benefits of including randomized participation incentives in surveys. Both the testing procedure and the methods for bias adjustment may be attractive tools for researchers who are able to embed randomized incentives into their survey. 

   more » « less
2. Does Survey Nonresponse Bias Estimates of Religious Service Attendance? Evidence from an Address-Based Sample from the Boston Area

   https://doi.org/10.1093/socrel/sry031  

   Brenner, Philip S (August 2018, Sociology of Religion)

   null (Ed.)

   Abstract This study investigates what role, if any, nonresponse plays in inflating survey estimates of religious behavior, using a multimode survey designed to allow estimation of nonresponse bias. A sample of 3,000 Boston-area households drawn from an address-based frame was randomly divided into two subsamples, contacted by mail, and invited to participate in a survey. The first subsample was asked to complete an interactive voice response interview. The second subsample was asked to complete a survey by telephone if a number was available for the address or by personal interview if not. Finally, random samples of nonrespondents were recontacted for a personal interview. Comparison of attendance estimates from initial interviews with nonrespondent interviews within sample segments yields minor or minimal differences that are not statistically significant. Findings suggest that the mechanism generating survey nonresponse is unlikely to be a major cause of bias in religious service attendance estimates in this study. 

   more » « less
3. Why Ecological Momentary Assessment Surveys Go Incomplete: When It Happens and How It Impacts Data

   https://doi.org/10.1055/s-0040-1719135  

   Wu, Yu-Hsiang; Xu, Jingjing; Stangl, Elizabeth; Pentony, Shareka; Vyas, Dhruv; Chipara, Octav; Gudjonsdottir, Anna; Oleson, Jacob; Galster, Jason (January 2021, Journal of the American Academy of Audiology)

   Abstract Background Ecological momentary assessment (EMA) often requires respondents to complete surveys in the moment to report real-time experiences. Because EMA may seem disruptive or intrusive, respondents may not complete surveys as directed in certain circumstances. Purpose This article aims to determine the effect of environmental characteristics on the likelihood of instances where respondents do not complete EMA surveys (referred to as survey incompletion), and to estimate the impact of survey incompletion on EMA self-report data. Research Design An observational study. Study Sample Ten adults hearing aid (HA) users. Data Collection and Analysis Experienced, bilateral HA users were recruited and fit with study HAs. The study HAs were equipped with real-time data loggers, an algorithm that logged the data generated by HAs (e.g., overall sound level, environment classification, and feature status including microphone mode and amount of gain reduction). The study HAs were also connected via Bluetooth to a smartphone app, which collected the real-time data logging data as well as presented the participants with EMA surveys about their listening environments and experiences. The participants were sent out to wear the HAs and complete surveys for 1 week. Real-time data logging was triggered when participants completed surveys and when participants ignored or snoozed surveys. Data logging data were used to estimate the effect of environmental characteristics on the likelihood of survey incompletion, and to predict participants' responses to survey questions in the instances of survey incompletion. Results Across the 10 participants, 715 surveys were completed and survey incompletion occurred 228 times. Mixed effects logistic regression models indicated that survey incompletion was more likely to happen in the environments that were less quiet and contained more speech, noise, and machine sounds, and in the environments wherein directional microphones and noise reduction algorithms were enabled. The results of survey response prediction further indicated that the participants could have reported more challenging environments and more listening difficulty in the instances of survey incompletion. However, the difference in the distribution of survey responses between the observed responses and the combined observed and predicted responses was small. Conclusion The present study indicates that EMA survey incompletion occurs systematically. Although survey incompletion could bias EMA self-report data, the impact is likely to be small. 

   more » « less
4. Correlates of Differences in Interactional Patterns among Black and White Respondents

   https://doi.org/10.1007/978-3-030-47256-6_12  

   Dykema, Jennifer; Garbarski, Dana; Schaeffer, Nora Cate; Anadon, Isabel; Edwards, Dorothy F. (October 2020, Understanding Survey Methodology. Frontiers in Sociology and Social Research)

   Brenner, P. S. (Ed.)

   Features of the survey measurement process may affect responses from respondents in various racial, ethnic, or cultural groups in different ways. When responses from multiethnic populations are combined, such variability in responding could increase variable error or bias results. The current study examines the survey response process among Black and White respondents answering questions about trust in medical researchers and participation in medical research. Using transcriptions from telephone interviews, we code a rich set of behaviors produced by respondents that past research has shown to be associated with measurement error, including long question-answer sequences, uncodable answers, requests for repetition or clarification, affective responses, and tokens. In analysis, we test for differences between Black and White respondents in the likelihood with which behaviors occur and examine whether the behaviors vary by specific categorizations of the questions, including whether the questions are racially focused. Overall, we find that White respondents produce more behaviors that indicate cognitive processing problems for racially focused questions, which may be interpreted as demonstrating a “cultural” difference in the display of cognitive processing and interaction. Data are provided by the 2013–2014 Voices Heard Survey, a computer-assisted telephone survey designed to measure respondents’ perceptions of barriers and facilitators to participating in medical research. 

   more » « less
5. Transitioning to a Mixed-Mode Study Design in a National Household Panel Study: Effects on Fieldwork Outcomes, Sample Composition and Costs

   https://doi.org/10.18148/srm/2023.v17i4.8172  

   McGonagle, Katherine A; Sastry, Narayan (December 2023, Survey research methods)

   The U.S. Panel Study of Income Dynamics (PSID) made a planned transition to a web-first mixed-mode data collection design in 2021 (web and computer-assisted telephone interviewing [CATI]), following nearly five decades of collecting data primarily using CATI with professional interviewers. To evaluate potential effects of mode on fieldwork outcomes, two sequential mixed-mode protocols were introduced using an experimental design. One protocol randomized sample families to a “web-first” treatment, which encouraged response through an online interview, followed by an offer of telephone to complete the interview; a second protocol randomized sample families to a “CATI-first” treatment until the last phase of fieldwork when the option to complete a web interview was offered. This paper examines the comparative effects of the two protocols on fieldwork outcomes, including response rates, interviewer contact attempts, fieldwork duration, and cost. Comparisons are also made with fieldwork outcomes and characteristics of non-responding sample members from the prior-wave when a traditional telephone design was used. We found that the web-first design compared to the CATI-first design led to comparably high response rates, and faster interview completion with lower effort and cost. With some notable exceptions, compared to the prior wave, the mixed-mode design reduced effort and had generally similar patterns of non-response among key respondent subgroups. The results provide new empirical evidence on the effects of mixing modes on fieldwork outcomes and costs and contribute to the small body of experimental evidence on the use of mixed-mode designs in household panel studies. 

   more » « less