skip to main content


Title: Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement
When groups of people are tasked with making a judgment, the issue of uncertainty often arises. Existing methods to reduce uncertainty typically focus on iteratively improving specificity in the overall task instruction. However, uncertainty can arise from multiple sources, such as ambiguity of the item being judged due to limited context, or disagreements among the participants due to different perspectives and an under-specified task. A one-size-fits-all intervention may be ineffective if it is not targeted to the right source of uncertainty. In this paper we introduce a new workflow, Judgment Sieve, to reduce uncertainty in tasks involving group judgment in a targeted manner. By utilizing measurements that separate different sources of uncertainty during an initial round of judgment elicitation, we can then select a targeted intervention adding context or deliberation to most effectively reduce uncertainty on each item being judged. We test our approach on two tasks: rating word pair similarity and toxicity of online comments, showing that targeted interventions reduced uncertainty for the most uncertain cases. In the top 10% of cases, we saw an ambiguity reduction of 21.4% and 25.7%, and a disagreement reduction of 22.2% and 11.2% for the two tasks respectively. We also found through a simulation that our targeted approach reduced the average uncertainty scores for both sources of uncertainty as opposed to uniform approaches where reductions in average uncertainty from one source came with an increase for the other.  more » « less
Award ID(s):
2137469
PAR ID:
10487973
Author(s) / Creator(s):
;
Publisher / Repository:
ACM
Date Published:
Journal Name:
Proceedings of the ACM on Human-Computer Interaction
Volume:
7
Issue:
CSCW2
ISSN:
2573-0142
Page Range / eLocation ID:
1 to 26
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    The management of biological invasions is a worldwide conservation priority. Unfortunately, decision‐making on optimal invasion management can be impeded by lack of information about the biological processes that determine invader success (i.e. biological uncertainty) or by uncertainty about the effectiveness of candidate interventions (i.e. operational uncertainty). Concurrent assessment of both sources of uncertainty within the same framework can help to optimize control decisions.

    Here, we present a Value of Information (VoI) framework to simultaneously analyse the effects of biological and operational uncertainties on management outcomes. We demonstrate this approach with a case study: minimizing the long‐term population growth of musk thistleCarduus nutans, a widespread invasive plant, using several insects as biological control agents, includingTrichosirocalus horridus,Rhinocyllus conicusandUrophora solstitialis.

    The ranking of biocontrol agents was sensitive to differences in the target weed's demography and also to differences in the effectiveness of the different biocontrol agents. This finding suggests that accounting for both biological and operational uncertainties is valuable when making management recommendations for invasion control. Furthermore, our VoI analyses show that reduction of all uncertainties across all combinations of demographic model and biocontrol effectiveness explored in the current study would lead, on average, to a 15.6% reduction in musk thistle population growth rate. The specific growth reduction that would be observed in any instance would depend on how the uncertainties actually resolve. Resolving biological uncertainty (across demographic model combinations) or operational uncertainty (across biocontrol effectiveness combinations) alone would reduce expected population growth rate by 8.5% and 10.5% respectively.

    Synthesis and applications. Our study demonstrates that intervention rank is determined both by biological processes in the targeted invasive populations and by intervention effectiveness. Ignoring either biological uncertainty or operational uncertainty may result in a suboptimal recommendation. Therefore, it is important to simultaneously acknowledge both sources of uncertainty during the decision‐making process in invasion management. The framework presented here can accommodate diverse data sources and modelling approaches, and has wide applicability to guide invasive species management and conservation efforts.

     
    more » « less
  2. Abstract

    A left digit effect has been broadly observed across judgment and decision‐making contexts ranging from product evaluation to medical treatment decisions to number line estimation. For example, $3.00 is judged to be a much greater cost than $2.99, and “801” is estimated strikingly too far to the right of “798” on a number line. Although the consequences of the effects for judgment and decision behavior have been documented, the sources of the effects are not well established. The goal of the current work is to extend investigations of the left digit effect to a new complex judgment activity and to assess whether the magnitude of the effect at the individual level can be predicted from performance on a simpler number skills task on which the left digit effect has also recently been observed. In three experiments (N = 434), adults completed a judgment task in which they rated the strength of hypothetical applicants for college admission and a self‐paced number line estimation task. In all experiments, a small or medium left digit effect was found in the college admissions task, and a large effect was found in number line estimation. Individual‐level variation was observed, but there was no relationship between the magnitudes of the effects in the two tasks. These findings provide evidence of a left digit effect in a novel multiattribute judgment task but offer no evidence that such performance can be predicted from a simple number skills task such as number line estimation.

     
    more » « less
  3. Abstract

    How do learners gather new information during word learning? One possibility is that learners selectively sample items that help them reduce uncertainty about new word meanings. In a series of cross‐situational word learning tasks with adults and children, we manipulated the referential ambiguity of label‐object pairs experienced during training and subsequently investigated which words participants chose to sample additional information about. In the first experiment, adult learners chose to receive additional training on object‐label associations that reduce referential ambiguity during cross‐situational word learning. This ambiguity‐reduction strategy was related to improved test performance. In two subsequent experiments, we found that, at least in some contexts, children (3–8 years of age) show a similar preference to seek information about words experienced in ambiguous word learning situations. In Experiment 2, children did not preferentially select object‐label associations that remained ambiguous during cross‐situational word learning. However, in a third experiment that increased the relative ambiguity of two sets of novel object‐label associations, we found evidence that children preferentially make selections that reduce ambiguity about novel word meanings. These results carry implications for understanding how children actively contribute to their own language development by seeking information that supports learning.

     
    more » « less
  4. Uncertainty decomposition refers to the task of decomposing the total uncertainty of a predictive model into aleatoric (data) uncertainty, resulting from inherent randomness in the data-generating process, and epistemic (model) uncertainty, resulting from missing information in the model’s training data. In large language models (LLMs) specifically, identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability, but remains an important open research question. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling, which can be applied to any pre-trained LLM. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions. We show that, when aleatoric uncertainty arises from ambiguity or under-specification in LLM inputs, this approach makes it possible to factor an (un-clarified) LLM’s predictions into separate aleatoric and epistemic terms, using a decomposition similar to the one employed by Bayesian neural networks. Empirical evaluations demonstrate that input clarification ensembling provides accurate and reliable uncertainty quantification on several language processing tasks. 
    more » « less
  5. Abstract

    Emerging infectious diseases have caused population declines and biodiversity loss. The ability of pathogens to survive in the environment, independent of their host, can exacerbate disease impacts and increase the likelihood of species extinction. Control of pathogens with environmental stages remains a significant challenge for conservation and effective management strategies are urgently needed.

    We examined the effectiveness of managing environmental exposure to reduce the impacts of an emerging infectious disease of bats, white‐nose syndrome (WNS). We used a chemical disinfectant, chlorine dioxide (ClO2), to experimentally reducePseudogymnoascus destructans, the fungal pathogen causing WNS, in the environment. We combined laboratory experiments with 3 years of field trials at four abandoned mines to determine whether ClO2could effectively removeP. destructansfrom the environment, reduce host infection and limit population impacts.

    ClO2was effective at killingP. destructansin vitro across multiple concentrations. In field settings, higher concentrations of ClO2treatment were needed to sufficiently reduce viableP. destructansconidia in the environment.

    The reduction in the environmental reservoir at treatment sites resulted in lower fungal loads on bats compared to untreated control populations. Survival following treatment was also higher in little brown bats (Myotis lucifugus), and trended higher for tricolored bats (Perimyotis subflavus).

    Synthesis and applications. Our results highlight that targeted management of sources for environmental transmission can be an effective control strategy for wildlife disease. We found that successfully reducing pathogen in the environment decreased disease severity and increased survival, but required higher treatment exposure than was effective in laboratory experiments, and the effects varied among species. More broadly, our findings have implications for other emerging wildlife diseases with free‐living pathogen stages by highlighting how the degree of environmental contamination can have cascading impacts on hosts, presenting an opportunity for intervention.

     
    more » « less