An official website of the United States government
This Research paper discusses the opportunities that utilizing a computer program can present in analyzing large amounts of qualitative data collected through a survey tool. When working with longitudinal qualitative data, there are many challenges that researchers face. The coding scheme may evolve over time requiring re-coding of early data. There may be long periods of time between data analysis. Typically, multiple researchers will participate in the coding, but this may introduce bias or inconsistencies. Ideally the same researchers would be analyzing the data, but often there is some turnover in the team, particularly when students assist with the coding. Computer programs can enable automated or semi-automated coding helping to reduce errors and inconsistencies in the coded data. In this study, a modeling survey was developed to assess student awareness of model types and administered in four first-year engineering courses across the three universities over the span of three years. The data collected from this survey consists of over 4,000 students’ open-ended responses to three questions about types of models in science, technology, engineering, and mathematics (STEM) fields. A coding scheme was developed to identify and categorize model types in student responses. Over two years, two undergraduate researchers analyzed a total of 1,829 students’ survey responses after ensuring intercoder reliability was greater than 80% for each model category. However, with much data remaining to be coded, the research team developed a MATLAB program to automatically implement the coding scheme and identify the types of models students discussed in their responses. MATLAB coded results were compared to human-coded results (n = 1,829) to assess reliability; results matched between 81%-99% for the different model categories. Furthermore, the reliability of the MATLAB coded results are within the range of the interrater reliability measured between the 2 undergraduate researchers (86-100% for the five model categories). With good reliability of the program, all 4,358 survey responses were coded; results showing the number and types of models identified by students are presented in the paper.
more »
« less
Who Broke Amazon Mechanical Turk?: An Analysis of Crowdsourcing Data Quality over Time
We present the results of a survey fielded in June of 2022 as a lens to examine recent data reliability issues on Amazon Mechanical Turk. We contrast bad data from this survey with bad data from the same survey fielded among US workers in October 2013, April 2018, and February 2019. Application of an established data cleaning scheme reveals that unusable data has risen from a little over 2% in 2013 to almost 90% in 2022. Through symptomatic diagnosis, we attribute the data reliability drop not to an increase in bad faith work, but rather to a continuum of English proficiency levels. A qualitative analysis of workers’ responses to open-ended questions allows us to distinguish between low fluency workers, ultra-low fluency workers, satisficers, and bad faith workers. We go on to show the effects of the new low fluency work on Likert scale data and on the study’s qualitative results. Attention checks are shown to be much less effective than they once were at identifying survey responses that should be discarded.
more »
« less
- Award ID(s):
- 1816923
- PAR ID:
- 10462328
- Date Published:
- Journal Name:
- WebSci '23: Proceedings of the 15th ACM Web Science Conference 2023
- Page Range / eLocation ID:
- 335 to 345
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The COVID-19 pandemic has transformed where paid work is done. Workers able to do so have been required to work remotely. We draw on survey data collected in October 2020 from a nationally representative sample of 3,017 remote workers, as well as qualitative survey data collected from 231 remote workers, to examine perceived changes in work hours from before to during the pandemic. Results indicate women are at greater risk of change (either a major decrease or a major increase)—rather than stability—in work hours. Gender also intersects with caregiving, race/ethnicity, prior remote work experiences, and socioeconomic status to shape changes in hours. Women and men in the sandwich generation, as well as women (but not men) with pre-school children, are the most likely to report a decrease in work hours, whereas women with older children at home or caring for adults (but not both) are the most likely to have an increase in hours. Remote working Black women and women moving into remote work are more likely to experience a major increase in hours worked, even as Hispanic women and Black men are the most likely to report somewhat of a reduction in work hours. Gender also intersects with SES, such that women without a college degree are more likely to have a decrease in work hours, while women with an advanced degree and women managers report a considerable increase in work hours. Qualitative data further illuminate why work hours change or remain stable for remote workers during COVID-19.more » « less
-
The future of work is ambiguous at best. Despite widespread shifts to remote/hybrid work during the COVID-19 lockdown, there is a paucity of knowledge about changing job conditions in tandem with different work locales. Is the move to remote/hybrid work a disrupter or accentuator of existing norms and inequalities? Drawing on nationally representative, four-wave panel survey data (October 2020 to April 2022) collected from U.S. workers who spent at least some time working from home since the pandemic onset, we examine effects of within-person changes in where respondents work on changes in job conditions (psychological job demands, job control, coworker support, and monitoring). Estimates from fixed-effects models show that, compared with returning to working at work, ongoing remote and moving to hybrid work lead to greater reductions in psychological job demands, especially among older women and men. Black and Hispanic women moving back to the office experience the greatest loss of decision latitude and schedule control. While white workers see increased coworker support when returning to the office, returning Black and Hispanic men report a decline in coworker support. Family caregivers’ job conditions do not improve whether remote/hybrid or returning to work. Qualitative data collected from Amazon Mechanic Turk illuminate mechanisms leading to salutary effects of remote work, but also the stress of combining jobs with family carework.more » « less
-
In this study we use latent class analysis, distractor analysis, and qualitative analysis of cognitive interviews of student responses to questions on an algebra concept inventory, in order to generate theories about how students’ selections of specific answer choices may reflect different stages or types of algebraic conceptual understanding. Our analysis reveals three groups of students in elementary algebra courses, which we label as “mostly random guessing”, “some procedural fluency with key misconceptions”, and “procedural fluency with emergent conceptual understanding”. Student responses also revealed high rates of misconceptions that stem from misuse or misunderstanding of procedures, and whose prevalence often correlates with higher levels of procedural fluency.more » « less
-
null (Ed.)Crowd work has the potential of helping the financial recovery of regions traditionally plagued by a lack of economic opportunities, e.g., rural areas. However, we currently have limited information about the challenges facing crowd workers from rural and super rural areas as they struggle to make a living through crowd work sites. This paper examines the challenges and advantages of rural and super rural Amazon Mechanical Turk (MTurk) crowd workers and contrasts them with those of workers from urban areas. Based on a survey of 421 crowd workers from differing geographic regions in the U.S., we identified how across regions, people struggled with being onboarded into crowd work. We uncovered that despite the inequalities and barriers, rural workers tended to be striving more in micro-tasking than their urban counterparts. We also identified cultural traits, relating to time dimension and individualism, that offer us an insight into crowd workers and the necessary qualities for them to succeed on gig platforms. We finish by providing design implications based on our findings to create more inclusive crowd work platforms and tools.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3578503.3583622
