An official website of the United States government
Adaptive comparative judgment (ACJ) is a holistic judgment approach used to evaluate the quality of something (e.g., student work) in which individuals are presented with pairs of work and select the better item from each pair. This approach has demonstrated high levels of reliability with less bias than other approaches, hence providing accurate values in summative and formative assessment in educational settings. Though ACJ itself has demonstrated significantly high reliability levels, relatively few studies have investigated the validity of peer-evaluated ACJ in the context of design thinking. This study explored peer-evaluation, facilitated through ACJ, in terms of construct validity and criterion validity (concurrent validity and predictive validity) in the context of a design thinking course. Using ACJ, undergraduate students ( n = 597) who took a design thinking course during Spring 2019 were invited to evaluate design point-of-view (POV) statements written by their peers. As a result of this ACJ exercise, each POV statement attained a specific parameter value, which reflects the quality of POV statements. In order to examine the construct validity, researchers conducted a content analysis, comparing the contents of the 10 POV statements with highest scores (parameter values) and the 10 POV statements with the lowest scores (parameter values)—as derived from the ACJ session. For the criterion validity, we studied the relationship between peer-evaluated ACJ and grader’s rubric-based grading. To study the concurrent validity, we investigated the correlation between peer-evaluated ACJ parameter values and grades assigned by course instructors for the same POV writing task. Then, predictive validity was studied by exploring if peer-evaluated ACJ of POV statements were predictive of students’ grades on the final project. Results showed that the contents of the statements with the highest parameter values were of better quality compared to the statements with the lowest parameter values. Therefore, peer-evaluated ACJ showed construct validity. Also, though peer-evaluated ACJ did not show concurrent validity, it did show moderate predictive validity.
more »
« less
Comparing cultural theory and cultural cognition theory survey measures to each other and as explanations for judged risk
Different approaches to operationalizing the cultural theory (CT) developed by Douglas, Thompson, Wildavsky, and others in survey research on risk perceptions are rarely compared, never for the same people. We compare for US respondents the construct validity of cultural worldview measures developed by Jenkins-Smith and colleagues—including both indices of items refining the Wildavsky and Dake approach, and short paragraphs (cultural “statements”)—to those developed by Kahan and colleagues based on cultural cognition theory (CCT). Correlational analyses reveal moderate convergent and discriminant validity among these measures, and along with regression analyses controlling for demographic variables similarly moderate predictive validity across measures for judgments of personal risk for ten hazards. CT statements better discriminate between individualists and hierarchists, and CT indices explain more variance in judged risk (predictive validity) when controlling for demographic variables in regression analyses. We discuss theoretical and methodological implications of our findings to foster further scholarly comparisons of and improvements in these survey-based cultural approaches to explaining risk judgments.
more »
« less
- Award ID(s):
- 1644853
- PAR ID:
- 10096557
- Date Published:
- Journal Name:
- Journal of risk research
- ISSN:
- 1366-9877
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Isaacowitz, Derek (Ed.)Abstract Objectives Theories of aging posit that older adult age is associated with less negative emotions, but few studies have examined age differences at times of novel challenges. As COVID-19 spread in the United States, this study therefore aimed to examine age differences in risk perceptions, anxiety, and depression. Method In March 2020, a nationally representative address-based sample of 6,666 U.S. adults assessed their perceived risk of getting COVID-19, dying if getting it, getting quarantined, losing their job (if currently working), and running out of money. They completed a mental health assessment for anxiety and depression. Demographic variables and precrisis depression diagnosis had previously been reported. Results In regression analyses controlling for demographic variables and survey date, older adult age was associated with perceiving larger risks of dying if getting COVID-19, but with perceiving less risk of getting COVID-19, getting quarantined, or running out of money, as well as less depression and anxiety. Findings held after additionally controlling for precrisis reports of depression diagnosis. Discussion With the exception of perceived infection-fatality risk, U.S. adults who were relatively older appeared to have a more optimistic outlook and better mental health during the early stages of the pandemic. Interventions may be needed to help people of all ages maintain realistic perceptions of the risks, while also managing depression and anxiety during the COVID-19 crisis. Implications for risk communication and mental health interventions are discussed.more » « less
-
Abstract Multiple imputation (MI) is a popular and well-established method for handling missing data in multivariate data sets, but its practicality for use in massive and complex data sets has been questioned. One such data set is the Panel Study of Income Dynamics (PSID), a longstanding and extensive survey of household income and wealth in the United States. Missing data for this survey are currently handled using traditional hot deck methods because of the simple implementation; however, the univariate hot deck results in large random wealth fluctuations. MI is effective but faced with operational challenges. We use a sequential regression/chained-equation approach, using the software IVEware, to multiply impute cross-sectional wealth data in the 2013 PSID, and compare analyses of the resulting imputed data with those from the current hot deck approach. Practical difficulties, such as non-normally distributed variables, skip patterns, categorical variables with many levels, and multicollinearity, are described together with our approaches to overcoming them. We evaluate the imputation quality and validity with internal diagnostics and external benchmarking data. MI produces improvements over the existing hot deck approach by helping preserve correlation structures, such as the associations between PSID wealth components and the relationships between the household net worth and sociodemographic factors, and facilitates completed data analyses with general purposes. MI incorporates highly predictive covariates into imputation models and increases efficiency. We recommend the practical implementation of MI and expect greater gains when the fraction of missing information is large.more » « less
-
ABSTRACT CONTEXT Culture influences the dynamics and outcomes of organizations in profound ways, including individual-level outcomes (like the quality of work products) and collective impacts (such as reputation or influence). As such, understanding organizational culture is a crucial element of understanding performance; from an anthropological perspective, ‘performance’ is not an outcome of culture, it is a part of culture. A key challenge in understanding organizational culture, especially in complex academic organizations, is the lack of a flexible, scalable approach for data collection and analysis. PURPOSE OR GOAL In this study, we report on our development of a survey-based cultural characterization tool that leverages both lightweight data collection from stakeholders in the organization and public information about that organization. We also integrate perspectives from prior literature about faculty, students, and staff in academic departments. Taken together, the resulting survey covers key elements of culture and allows for scalable data collection across settings via customizations and embedded logic in the survey itself. The outcome of this work is a design process for a new and promising tool for scalable cultural characterization, and we have deployed this tool across two institutions. APPROACH OR METHODOLOGY/METHODS We leverage prior research, our own preliminary data collection, and our experience with this approach in a different setting to develop a cultural characterization survey suitable for delivery to multiple engineering department stakeholders (faculty, staff, and students). We start with a modest number of interviews, stratified by these three groups and achieving saturation of responses, to understand their views on their organization, its strengths and weaknesses, and their perceptions of how it ‘works’. We merge this information with public data (for instance, departmental vision or mission statements, which convey a sense of priorities or values) as well as prior literature about higher education culture. We also draw upon our experience in another setting as well as pilot testing data, and the result is a carefully-constructed set of dichotomous items that are offered to department stakeholders in survey form using an electronic survey platform. We also collect background and demographic information in the survey. The resulting data are analyzed using Cultural Consensus Theory (CCT) to extract meaningful information about the departmental culture from the perspectives of the stakeholder groups. ACTUAL OR ANTICIPATED OUTCOMES The resulting survey consists of two parts, each with sub-components. The two top level survey parts contain: (i) items common to all respondents in all settings (i.e. all institutions in this study), and (ii) a set of institution-specific items. Within those sections, the framing of the items is calibrated for the stakeholder groups so that items make sense to them within the context of their experience. The survey has been administered, and the data are being analyzed and interpreted presently. We expect the results to capture the specific elements of local culture within these institutions, as well as differences in perspectives and experience among the three primary stakeholder groups. CONCLUSIONS/RECOMMENDATIONS/SUMMARY This study demonstrates a scalable approach to survey development for the purposes of cultural characterization, and its use across settings and with multiple stakeholder groups. This work enables a very nuanced view of culture within a department, and these results can be used within academic departments to enable discussion about change, priorities, performance, and the work environment.more » « less
-
Abstract BackgroundEmerging evidence indicates an elevated risk of post-concussion musculoskeletal (MSK) injuries in collegiate athletes; however, identifying athletes at highest risk remains to be elucidated. ObjectiveThe purpose of this study was to model post-concussion MSK injury risk in collegiate athletes by integrating a comprehensive set of variables by machine learning. MethodsA risk model was developed and tested on a dataset of 194 athletes (155 in the training set and 39 in the test set) with 135 variables entered into the analysis, which included participant’s heath and athletic history, concussion injury and recovery specific criteria, and outcomes from a diverse array of concussions assessments. The machine learning approach involved transforming variables by the Weight of Evidence method, variable selection using L1-penalized logistic regression, model selection via the Akaike Information Criterion, and a final L2-regularized logistic regression fit. ResultsA model with 48 predictive variables yielded significant predictive performance of subsequent MSK injury with an area under the curve of 0.82. Top predictors included cognitive, balance, and reaction at Baseline and Acute timepoints. At a specified false positive rate of 6.67%, the model achieves a true positive rate (sensitivity) of 79% and a precision (positive predictive value) of 95% for identifying at-risk athletes via a well calibrated composite risk score. ConclusionThese results support the development of a sensitive and specific injury risk model using standard data combined with a novel methodological approach that may allow clinicians to target high injury risk student-athletes. The development and refinement of predictive models, incorporating machine learning and utilizing comprehensive datasets, could lead to improved identification of high-risk athletes and allow for the implementation of targeted injury risk reduction strategies by identifying student-athletes most at risk for post-concussion MSK injury. Key PointsThere is a well-established elevated risk of post-concussion subsequent musculoskeletal injury; however, prior efforts have failed to identify risk factors.This study developed a composite risk score model with an AUC of 0.82 from common concussion clinical measures and participant demographics.By identifying athletes at elevated risk, clinicians may be able to reduce injury risk through targeted injury risk reduction programs.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
