skip to main content

Title: Response Times as an Indicator of Data Quality: Associations with Question, Interviewer, and Respondent Characteristics in a Health Survey of Diverse Respondents
Response time (RT) – the time elapsing from the beginning of question reading for a given question until the start of the next question – is a potentially important indicator of data quality that can be reliably measured for all questions in a computer-administered survey using a latent timer (i.e., triggered automatically by moving on to the next question). In interviewer-administered surveys, RTs index data quality by capturing the entire length of time spent on a question–answer sequence, including interviewer question-asking behaviors and respondent question-answering behaviors. Consequently, longer RTs may indicate longer processing or interaction on the part of the interviewer, respondent, or both. RTs are an indirect measure of data quality; they do not directly measure reliability or validity, and we do not directly observe what factors lengthen the administration time. In addition, either too long or too short RTs could signal a problem (Ehlen, Schober, and Conrad 2007). However, studies that link components of RTs (interviewers’ question reading and response latencies) to interviewer and respondent behaviors that index data quality strengthen the claim that RTs indicate data quality (Bergmann and Bristle 2019; Draisma and Dijkstra 2004; Olson, Smyth, and Kirchner 2019). In general, researchers tend to consider longer RTs more » as signaling processing problems for the interviewer, respondent, or both (Couper and Kreuter 2013; Olson and Smyth 2015; Yan and Olson 2013; Yan and Tourangeau 2008). Previous work demonstrates that RTs are associated with various characteristics of interviewers (where applicable), questions, and respondents in web, telephone, and face-to-face interviews (e.g., Couper and Kreuter 2013; Olson and Smyth 2015; Yan and Tourangeau 2008). We replicate and extend this research by examining how RTs are associated with various question characteristics and several established tools for evaluating questions. We also examine whether increased interviewer experience in the study shortens RTs for questions with characteristics that impact the complexity of the interviewer’s task (i.e., interviewer instructions and parenthetical phrases). We examine these relationships in the context of a sample of racially diverse respondents who answered questions about participation in medical research and their health. « less
Authors:
; ; ;
Award ID(s):
1853094
Publication Date:
NSF-PAR ID:
10186206
Journal Name:
Interviewer Effects from a Total Survey Error Perspective
Page Range or eLocation-ID:
252-265
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    Asking questions fluently, exactly as worded, and at a reasonable pace is a fundamental part of a survey interviewer’s role. Doing so allows the question to be asked as intended by the researcher and may decrease the risk of measurement error and contribute to rapport. Despite the central importance placed on reading questions exactly as worded, interviewers commonly misread questions, and it is not always clear why. Thus, understanding the risk of measurement error requires understanding how different interviewers, respondents, and question features may trigger question reading problems. In this article, we evaluate the effects of question features on question asking behaviors, controlling for interviewer and respondent characteristics. We also examine how question asking behaviors are related to question-asking time. Using two nationally representative telephone surveys in the United States, we find that longer questions and questions with transition statements are less likely to be read exactly and fluently, that questions with higher reading levels and parentheticals are less likely to be read exactly across both surveys and that disfluent readings decrease as interviewers gain experience across the field period. Other question characteristics vary in their associations with the outcomes across the two surveys. We also find that inexactmore »and disfluent question readings are longer, but read at a faster pace, than exact and fluent question reading. We conclude with implications for interviewer training and questionnaire design.

    « less
  2. We conducted an experiment to evaluate the effects on fieldwork outcomes and interview mode of switching to a web-first mixed-mode data collection design (self-administered web interview and interviewer-administered telephone interview) from a telephone-only design. We examine whether the mixed-mode option leads to better survey outcomes, based on response rates, fieldwork outcomes, interview quality and costs. We also examine respondent characteristics associated with completing a web interview rather than a telephone interview. Our mode experiment study was conducted in the 2019 wave of the Transition into Adulthood Supplement (TAS) to the US Panel Study of Income Dynamics (PSID). TAS collects information biennially from approximately 3,000 young adults in PSID families. The shift to a mixed-mode design for TAS was aimed at reducing costs and increasing respondent cooperation. We found that for mixed-mode cases compared to telephone only cases, response rates were higher, interviews were completed faster and with lower effort, the quality of the interview data appeared better, and fieldwork costs were lower. A clear set of respondent characteristics reflecting demographic and socioeconomic characteristics, technology availability and use, time use, and psychological health were associated with completing a web interview rather than a telephone interview.
  3. Live video (LV) communication tools (e.g., Zoom) have the potential to provide survey researchers with many of the benefits of in-person interviewing, while also greatly reducing data collection costs, given that interviewers do not need to travel and make in-person visits to sampled households. The COVID-19 pandemic has exposed the vulnerability of in-person data collection to public health crises, forcing survey researchers to explore remote data collection modes—such as LV interviewing—that seem likely to yield high-quality data without in-person interaction. Given the potential benefits of these technologies, the operational and methodological aspects of video interviewing have started to receive research attention from survey methodologists. Although it is remote, video interviewing still involves respondent–interviewer interaction that introduces the possibility of interviewer effects. No research to date has evaluated this potential threat to the quality of the data collected in video interviews. This research note presents an evaluation of interviewer effects in a recent experimental study of alternative approaches to video interviewing including both LV interviewing and the use of prerecorded videos of the same interviewers asking questions embedded in a web survey (“prerecorded video” interviewing). We find little evidence of significant interviewer effects when using these two approaches, which is amore »promising result. We also find that when interviewer effects were present, they tended to be slightly larger in the LV approach as would be expected in light of its being an interactive approach. We conclude with a discussion of the implications of these findings for future research using video interviewing.« less
  4. Researchers and practitioners in marketing, economics, and public policy often use preference elicitation tasks to forecast real-world behaviors. These tasks typically ask a series of similarly structured questions. The authors posit that every time a respondent answers an additional elicitation question, two things happen: (1) they provide information about some parameter(s) of interest, such as their time preference or the partworth for a product attribute, and (2) the respondent increasingly “adapts” to the task—that is, using task-specific decision processes specialized for this task that may or may not apply to other tasks. Importantly, adaptation comes at the cost of potential mismatch between the task-specific decision process and real-world processes that generate the target behaviors, such that asking more questions can reduce external validity. The authors used mouse and eye tracking to trace decision processes in time preference measurement and conjoint choice tasks. Respondents increasingly relied on task-specific decision processes as more questions were asked, leading to reduced external validity for both related tasks and real-world behaviors. Importantly, the external validity of measured preferences peaked after as few as seven questions in both types of tasks. When measuring preferences, less can be more.

  5. A diagnostic of thirty questions administered to incoming STEM students in Fall 2013 and Fall 2015 - Fall 2018 reflects that their spatial visualization skills (SVS) need to be improved. Previous studies in the SVS subject [1], [2], [3] report that well-developed SVS skills lead to students’ success in Engineering and Technology, Computer Science, Chemistry, Computer Aided Design and Mathematics. Authors [4], [5] mention that aptitude in spatial skills is gradually becoming a standard assessment of an individual’s likelihood to succeed as an engineer. This research reports the qualitative and quantitative results of a project designed to improve SVS’s for STEM students managed under two strategies. The first strategy utilized was a series of face-to-face (FtF), two-hour training sessions taught over six weeks to all majors in STEM. This strategy was offered in Spring 2014 and every semester from Fall 2015 - Spring 2018. The second strategy was an embedded training (ET) implemented by one faculty from Fall 2017- Fall 2018. The faculty embedded the training in the US 1100 freshman seminar and was highly motivated to increase awareness of students on the importance and applicability of SVS in their fields of study. As reported by Swail et al. [6],more »cognitive, social, and institutional factors are key elements to best support students’ persistence and achievement. Both interventions used in this project encompassed all these factors and were supported by an NSF IUSE grant (2015-2019) to improve STEM retention. The FtF training was taken by 34 students majoring in diverse STEM fields. Its effectiveness was statistically assessed through a t-test to compare the results in the Purdue Spatial Visualization Skills Test - Rotations before and after the training and through analysis of surveys. Results were very positive; 85.29% of the participants improved their scores. The average change in scores was 5.29 (from 16.85 to 22.15; 17.65% improvement) and it was statistically significant (p-value 3.9E-8). On the surveys, 90% of students answered that they were satisfied with the training. Several students reported that they appreciated a connection between SVS, Calculus II and Engineering Graphics classes while others based the satisfaction on perceiving the critical role SVS will play in their careers. Results from the ET strategy were also encouraging. Teaching methods, curriculum and results are discussed in this paper. Adjustments to the teaching methods were done over 3 semesters. In the last semester, the faculty found that covering the modules at a slower pace than in the FtF training, asking the students to complete the pre-and post-diagnostic in class, and introducing the Spatial VisTM app to provide students with additional practice were key elements to assure students success and satisfaction. In conclusion, both strategies were demonstrated to be powerful interventions to increase students’ success because they not only offer students, particularly freshman, a way to refine SVS but also increase motivation in STEM while creating a community among students and faculty. The ET is effective and apt to be institutionalized. Lastly, this experimental research strengthens the literature on SVS.« less