Title: An Objective Measure of Decisional Clarity to Assess Decision Aid Effectiveness in Situations with Equipoise: A Randomized Trial
Background
Decision aids can help patients make medical decisions, which is especially advantageous in situations with equipoise. However, when there is no correct answer, it is difficult to assess whether a decision aid is helpful. The goal of this research is to propose and validate an objective method for measuring decision aid effectiveness by quantifying the clarity participants achieved when making decisions.
Design
The measure of decisional clarity was tested in a convenience sample of 131 college-aged students making hypothetical decisions about 2 treatment options for depression and anxiety. The treatments varied with respect to potential benefits and harms. Information was presented numerically or with an accompanying data visualization (an icon array) that is known to aid decision making.
Results
Decisional clarity was better with the icon arrays. Furthermore, the results showed that decisional clarity can be used to identify situations for which patients will be more likely to struggle making their decision. These included situations for which financial considerations were relevant to the decision and situations for which the probabilities of potential benefits were higher.
Limitations
The measure of decisional clarity and the situations identified as lacking clarity should be validated with a larger, more representative sample.
Conclusions
These findings demonstrate that decisional clarity can be used to both empirically evaluate the effectiveness of a decision aid as well as test factors that can cloud clarity and disrupt medical decision making.
Implications
Researchers and medical providers interested in developing decision aids for situations with equipoise can use decisional clarity as an objective measure to assess the effectiveness of their decision aid. Financial considerations and higher probabilities may also cloud judgments.
Highlights
An objective measure of decisional clarity is supported. Decisional clarity can be used to evaluate decision aids in the context of equipoise for which there is no objectively correct choice. Decisional clarity can also be used to identify scenarios for which patients are likely to struggle to make a medical decision.
Peters, Ellen; Shoots-Reinhard, Brittany(
, Medical Decision Making)
Background
Objective numeracy appears to support better medical decisions and health outcomes. The more numerate generally understand and use numbers more and make better medical decisions, including more informed medical choices. Numeric self-efficacy—an aspect of subjective numeracy that is also known as numeric confidence—also relates to decision making via emotional reactions to and inferences from experienced difficulty with numbers and via persistence linked with numeric comprehension and healthier behaviors over time. Furthermore, it moderates the effects of objective numeracy on medical outcomes.
Purpose
We briefly review the numeracy and decision-making literature and then summarize more recent literature on 3 separable effects of numeric self-efficacy. Although dual-process theories can account for the generally superior decision making of the highly numerate, they have neglected effects of numeric self-efficacy. We discuss implications for medical decision-making (MDM) research and practice. Finally, we propose a modification to dual-process theories, adding a “motivational mind” to integrate the effects of numeric self-efficacy on decision-making processes (i.e., inferences from experienced difficulty with numbers, greater persistence, and greater use of objective-numeracy skills) important to high-quality MDM.
Conclusions
The power of numeric self-efficacy (confidence) has been little considered in MDM, but many medical decisions and behaviors require persistence to be successful over time (e.g., comprehension, medical-recommendation adherence). Including numeric self-efficacy in research and theorizing will increase understanding of MDM and promote development of better decision interventions.
Highlights
Research demonstrates that objective numeracy supports better medical decisions and health outcomes. The power of numeric self-efficacy (aka numeric confidence) has been little considered but appears critical to emotional reactions and inferences that patients and others make when encountering numeric information (e.g., in decision aids) and to greater persistence in medical decision-making tasks involving numbers. The present article proposes a novel modification to dual-process theory to account for newer findings and to describe how numeracy mechanisms can be better understood. Because being able to adapt interventions to improve medical decisions depends in part on having a good theory, future research should incorporate numeric self-efficacy into medical decision-making theories and interventions.
McCowan, K.(
, Annual Conference of the American Psychology-Law Society.)
Abstract
Expert testimony varies in scientific quality and jurors have a difficult time evaluating evidence quality (McAuliff et al., 2009). In the current study, we apply Fuzzy Trace Theory principles, examining whether visual and gist aids help jurors calibrate to the strength of scientific evidence. Additionally we were interested in the role of jurors’ individual differences in scientific reasoning skills in their understanding of case evidence. Contrary to our preregistered hypotheses, there was no effect of evidence condition or gist aid on evidence understanding. However, individual differences between jurors’ numeracy skills predicted evidence understanding.
Summary
Poor-quality expert evidence is sometimes admitted into court (Smithburn, 2004). Jurors’ calibration to evidence strength varies widely and is not robustly understood. For instance, previous research has established jurors lack understanding of the role of control groups, confounds, and sample sizes in scientific research (McAuliff, Kovera, & Nunez, 2009; Mill, Gray, & Mandel, 1994). Still others have found that jurors can distinguish weak from strong evidence when the evidence is presented alone, yet not when simultaneously presented with case details (Smith, Bull, & Holliday, 2011). This research highlights the need to present evidence to jurors in a way they can understand. Fuzzy Trace Theory purports that people encode information in exact, verbatim representations and through “gist” representations, which represent summary of meaning (Reyna & Brainerd, 1995). It is possible that the presenting complex scientific evidence to people with verbatim content or appealing to the gist, or bottom-line meaning of the information may influence juror understanding of that evidence. Application of Fuzzy Trace Theory in the medical field has shown that gist representations are beneficial for helping laypeople better understand risk and benefits of medical treatment (Brust-Renck, Reyna, Wilhelms, & Lazar, 2016). Yet, little research has applied Fuzzy Trace Theory to information comprehension and application within the context of a jury (c.f. Reyna et. al., 2015). Additionally, it is likely that jurors’ individual characteristics, such as scientific reasoning abilities and cognitive tendencies, influence their ability to understand and apply complex scientific information (Coutinho, 2006).
Methods
The purpose of this study was to examine how jurors calibrate to the strength of scientific information, and whether individual difference variables and gist aids inspired by Fuzzy Trace Theory help jurors better understand complicated science of differing quality. We used a 2 (quality of scientific evidence: high vs. low) x 2 (decision aid to improve calibration - gist information vs. no gist information), between-subjects design. All hypotheses were preregistered on the Open Science Framework.
Jury-eligible community participants (430 jurors across 90 juries; Mage = 37.58, SD = 16.17, 58% female, 56.93% White). Each jury was randomly assigned to one of the four possible conditions. Participants were asked to individually fill out measures related to their scientific reasoning skills prior to watching a mock jury trial. The trial was about an armed bank robbery and consisted of various pieces of testimony and evidence (e.g. an eyewitness testimony, police lineup identification, and a sweatshirt found with the stolen bank money). The key piece of evidence was mitochondrial DNA (mtDNA) evidence collected from hair on a sweatshirt (materials from Hans et al., 2011). Two experts presented opposing opinions about the scientific evidence related to the mtDNA match estimate for the defendant’s identification. The quality and content of this mtDNA evidence differed based on the two conditions. The high quality evidence condition used a larger database than the low quality evidence to compare to the mtDNA sample and could exclude a larger percentage of people. In the decision aid condition, experts in the gist information group presented gist aid inspired visuals and examples to help explain the proportion of people that could not be excluded as a match. Those in the no gist information group were not given any aid to help them understand the mtDNA evidence presented. After viewing the trial, participants filled out a questionnaire on how well they understood the mtDNA evidence and their overall judgments of the case (e.g. verdict, witness credibility, scientific evidence strength). They filled this questionnaire out again after a 45-minute deliberation.
Measures
We measured Attitudes Toward Science (ATS) with indices of scientific promise and scientific reservations (Hans et al., 2011; originally developed by National Science Board, 2004; 2006). We used Drummond and Fischhoff’s (2015) Scientific Reasoning Scale (SRS) to measure scientific reasoning skills. Weller et al.’s (2012) Numeracy Scale (WNS) measured proficiency in reasoning with quantitative information. The NFC-Short Form (Cacioppo et al., 1984) measured need for cognition. We developed a 20-item multiple-choice comprehension test for the mtDNA scientific information in the cases (modeled on Hans et al., 2011, and McAuliff et al., 2009). Participants were shown 20 statements related to DNA evidence and asked whether these statements were True or False. The test was then scored out of 20 points.
Results
For this project, we measured calibration to the scientific evidence in a few different ways. We are building a full model with these various operationalizations to be presented at APLS, but focus only on one of the calibration DVs (i.e., objective understanding of the mtDNA evidence) in the current proposal. We conducted a general linear model with total score on the mtDNA understanding measure as the DV and quality of scientific evidence condition, decision aid condition, and the four individual difference measures (i.e., NFC, ATS, WNS, and SRS) as predictors. Contrary to our main hypotheses, neither evidence quality nor decision aid condition affected juror understanding. However, the individual difference variables did: we found significant main effects for Scientific Reasoning Skills, F(1, 427) = 16.03, p <.001, np2 = .04, Weller Numeracy Scale, F(1, 427) = 15.19, p <.001, np2 = .03, and Need for Cognition, F(1, 427) = 16.80, p <.001, np2 = .04, such that those who scored higher on these measures displayed better understanding of the scientific evidence. In addition there was a significant interaction of evidence quality condition and scores on the Weller’s Numeracy Scale, F(1, 427) = 4.10, p = .04, np2 = .01. Further results will be discussed.
Discussion
These data suggest jurors are not sensitive to differences in the quality of scientific mtDNA evidence, and also that our attempt at helping sensitize them with Fuzzy Trace Theory-inspired aids did not improve calibration. Individual scientific reasoning abilities and general cognition styles were better predictors of understanding this scientific information. These results suggest a need for further exploration of approaches to help jurors differentiate between high and low quality evidence.
Note: The 3rd author was supported by an AP-LS AP Award for her role in this research.
Learning Objective:
Participants will be able to describe how individual differences in scientific reasoning skills help jurors understand complex scientific evidence.
Theory—understanding mental processes that drive decisions—is important to help patients and providers make decisions that reflect medical advances and personal values. Building on a 2008 review, we summarize current tenets of fuzzy-trace theory (FTT) in light of new evidence that provides insight regarding mental representations of options and how such representations connect to values and evoke emotions. We discuss implications for communicating risks, preventing risky behaviors, discouraging misinformation, and choosing appropriate treatments. Findings suggest that simple, fuzzy but meaningful gist representations of information often determine decisions. Within minutes of conversing with their doctor, reading a health-related web post, or processing other health information, patients rely on gist memories of that information rather than verbatim details. This fuzzy-processing preference explains puzzles and paradoxes in how patients (and sometimes providers) think about probabilities (e.g., “50-50” chance), outcomes of treatment (e.g., with antibiotics), experiences of pain, end-of-life decisions, memories for medication instructions, symptoms of concussion, and transmission of viruses (e.g., in AIDS and COVID-19). As examples, participation in clinical trials or seeking treatments with low probabilities of success (e.g., with antibiotics or at the end of life) may indicate a defensibly different categorical gist perspective on risk as opposed to simply misunderstanding probabilities or failing to make prescribed tradeoffs. Thus, FTT explains why people avoid precise tradeoffs despite computing them. Facilitating gist representations of information offers an alternative approach that goes beyond providing uninterpreted “neutral” facts versus persuading or shifting the balance between fast versus slow thinking (or emotion vs. cognition). In contrast to either taking mental shortcuts or deliberating about details, gist processing facilitates application of advanced knowledge and deeply held values to choices. Highlights Fuzzy-trace theory (FTT) supports practical approaches to improving health and medicine. FTT differs in important respects from other theories of decision making, which has implications for how to help patients, providers, and health communicators. Gist mental representations emphasize categorical distinctions, reflect understanding in context, and help cue values relevant to health and patient care. Understanding the science behind theory is crucial for evidence-based medicine.
Patients have a poor understanding of outcomes related to total knee replacement (TKR) surgery, with most patients underestimating the potential benefits and overestimating the risk of complications. In this study, we sought to compare the impacts of descriptive information alone or in combination with an icon array, experience condition (images), or spinner on participants’ preference forTKR.
Methods
A total of 648 members of an online arthritis network were randomized to 1 of 4 outcome presentation formats: numeric only, numeric with an icon array, numeric with a set of 50 images, or numeric with a functional spinner. Preferences forTKRwere measured before and immediately after viewing the outcome information using an 11‐point numeric rating scale. Knowledge was assessed by asking participants to report the frequency of each outcome.
Results
Participants randomized to the icon array, images, and spinner had stronger preferences forTKR(after controlling for baseline preferences) compared to those viewing the numeric only format (P< 0.05 for all mean differences). Knowledge scores were highest in participants randomized to the icon array; however, knowledge did not mediate the association between format and change in preference forTKR.
Conclusion
Decision support at the point‐of‐care is being increasingly recognized as a vital component of care. Our findings suggest that adding graphic information to descriptive statistics strengthens preferences forTKR. Although experience formats using images may be too complex to use in clinical practice, icon arrays and spinners may be a viable and easily adaptable decision aid to support communication of probabilistic information.
Witt, Jessica K. An Objective Measure of Decisional Clarity to Assess Decision Aid Effectiveness in Situations with Equipoise: A Randomized Trial. Medical Decision Making 42.6 Web. doi:10.1177/0272989X221085489.
Witt, Jessica K. An Objective Measure of Decisional Clarity to Assess Decision Aid Effectiveness in Situations with Equipoise: A Randomized Trial. Medical Decision Making, 42 (6). https://doi.org/10.1177/0272989X221085489
Witt, Jessica K.
"An Objective Measure of Decisional Clarity to Assess Decision Aid Effectiveness in Situations with Equipoise: A Randomized Trial". Medical Decision Making 42 (6). Country unknown/Code not available: SAGE Publications. https://doi.org/10.1177/0272989X221085489.https://par.nsf.gov/biblio/10368918.
@article{osti_10368918,
place = {Country unknown/Code not available},
title = {An Objective Measure of Decisional Clarity to Assess Decision Aid Effectiveness in Situations with Equipoise: A Randomized Trial},
url = {https://par.nsf.gov/biblio/10368918},
DOI = {10.1177/0272989X221085489},
abstractNote = {BackgroundDecision aids can help patients make medical decisions, which is especially advantageous in situations with equipoise. However, when there is no correct answer, it is difficult to assess whether a decision aid is helpful. The goal of this research is to propose and validate an objective method for measuring decision aid effectiveness by quantifying the clarity participants achieved when making decisions. DesignThe measure of decisional clarity was tested in a convenience sample of 131 college-aged students making hypothetical decisions about 2 treatment options for depression and anxiety. The treatments varied with respect to potential benefits and harms. Information was presented numerically or with an accompanying data visualization (an icon array) that is known to aid decision making. ResultsDecisional clarity was better with the icon arrays. Furthermore, the results showed that decisional clarity can be used to identify situations for which patients will be more likely to struggle making their decision. These included situations for which financial considerations were relevant to the decision and situations for which the probabilities of potential benefits were higher. LimitationsThe measure of decisional clarity and the situations identified as lacking clarity should be validated with a larger, more representative sample. ConclusionsThese findings demonstrate that decisional clarity can be used to both empirically evaluate the effectiveness of a decision aid as well as test factors that can cloud clarity and disrupt medical decision making. ImplicationsResearchers and medical providers interested in developing decision aids for situations with equipoise can use decisional clarity as an objective measure to assess the effectiveness of their decision aid. Financial considerations and higher probabilities may also cloud judgments. HighlightsAn objective measure of decisional clarity is supported. Decisional clarity can be used to evaluate decision aids in the context of equipoise for which there is no objectively correct choice. Decisional clarity can also be used to identify scenarios for which patients are likely to struggle to make a medical decision.},
journal = {Medical Decision Making},
volume = {42},
number = {6},
publisher = {SAGE Publications},
author = {Witt, Jessica K.},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.