An official website of the United States government
Multiverse analyses involve conducting all combinations of reasonable choices in a data analysis process. A reader of a study containing a multiverse analysis might question—are all the choices included in the multiverse reasonable and equally justifiable? How much do results vary if we make different choices in the analysis process? In this work, we identify principles for validating the composition of, and interpreting the uncertainty in, the results of a multiverse analysis. We present Milliways, a novel interactive visualisation system to support principled evaluation of multiverse analyses. Milliways provides interlinked panels presenting result distributions, individual analysis composition, multiverse code specification, and data summaries. Milliways supports interactions to sort, filter and aggregate results based on the analysis specification to identify decisions in the analysis process to which the results are sensitive. To represent the two qualitatively different types of uncertainty that arise in multiverse analyses—probabilistic uncertainty from estimating unknown quantities of interest such as regression coefficients, and possibilistic uncertainty from choices in the data analysis—Milliways uses consonance curves and probability boxes. Through an evaluative study with five users familiar with multiverse analysis, we demonstrate how Milliways can support multiverse analysis tasks, including a principled assessment of the results of a multiverse analysis.
more »
« less
Paths Explored, Paths Omitted, Paths Obscured: Decision Points & Selective Reporting in End-to-End Data Analysis
Drawing reliable inferences from data involves many, sometimes arbitrary, decisions across phases of data collection, wrangling, and modeling. As different choices can lead to diverging conclusions, understanding how researchers make analytic decisions is important for supporting robust and replicable analysis. In this study, we pore over nine published research studies and conduct semi-structured interviews with their authors. We observe that researchers often base their decisions on methodological or theoretical concerns, but subject to constraints arising from the data, expertise, or perceived interpretability. We confirm that researchers may experiment with choices in search of desirable results, but also identify other reasons why researchers explore alternatives yet omit findings. In concert with our interviews, we also contribute visualizations for communicating decision processes throughout an analysis. Based on our results, we identify design opportunities for strengthening end-to-end analysis, for instance via tracking and meta-analysis of multiple decision paths.
more »
« less
- Award ID(s):
- 1901386
- PAR ID:
- 10171985
- Date Published:
- Journal Name:
- Human factors in computing systems
- ISSN:
- 1062-9432
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Expanding and Refining a Decision-Making Competency Inventory for Undergraduate Engineering Studentsnull (Ed.)This Full Research Paper discusses ongoing work to develop a survey instrument to reliably assess undergraduate engineering student self-regulated decision-making. This work focuses on a second round of item expansion and refinement to the Decision-Making Competency Inventory (DMCI) to develop items related to learning from past decisions. The refined instrument was distributed to first-year engineering students enrolled in a large, public, land-grant institution located in the southeastern United States in the Fall of 2018. Of the approximately 1,200 students in first-year engineering courses, 883 valid surveys were randomly split into two separate samples for exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). EFA results indicated a viable four-factor solution, which was explored with the CFA. The CFA results also indicated a four-factor model was appropriate. Improving this instrument will help researchers document and understand students’ decision-making skills and how they relate to observed decisions like initial choice of major or change of major. A decision-making instrument will also be valuable in evaluating the effectiveness of interventions to help students build their decision-making competency and make adaptive choices.more » « less
-
Background The decision making process undertaken during wildfire responses is complex and prone to uncertainty. In the US, decisions federal land managers make are influenced by numerous and often competing factors. Aims To assess and validate the presence of decision factors relevant to the wildfire decision making context that were previously known and to identify those that have emerged since the US federal wildfire policy was updated in 2009. Methods Interviews were conducted across the US while wildfires were actively burning to elucidate time-of-fire decision factors. Data were coded and thematically analysed. Key results Most previously known decision factors as well as numerous emergent factors were identified. Conclusions To contextualise decision factors within the decision making process, we offer a Wildfire Decision Framework that has value for policy makers seeking to improve decision making, managers improving their process and wildfire social science researchers. Implications Managers may gain a better understanding of their decision environment and use our framework as a tool to validate their deliberations. Researchers may use these data to help explain the various pressures and influences modern land and wildfire managers experience. Policy makers and agencies may take institutional steps to align the actions of their staff with desired wildfire outcomes.more » « less
-
What happens to a person’s data after they pass away? Designing for digital legacy requires input from individuals across all life stages, as motivations to plan vary with age. Yet, the specific perspectives that older adults have on end-of-life data management have not been investigated in depth. Through interviews with 16 older adults, we examine their preferences and motivations for managing everyday digital data (e.g., text messages, social media, photos) after death. Our findings surface several implications for end-of-life data planning, including creating awareness about digital legacy and associated risks. We also unpack and discuss how older adults’ life stage and familiarity with end-of-life planning uniquely positions them to identify barriers and opportunities in managing digital legacy, such as how post-mortem data can encode societal norms of a period or be donated for the greater good.more » « less
-
Many high-stakes policies can be modeled as a sequence of decisions along a pipeline. We are interested in auditing such pipelines for both Our empirical focus is on policy decisions made by the New efficiency and equity. Using a dataset of over 100,000 crowdsourced resident requests for po- life-tentially hazardous tree maintenance in New York City, we observe a sequence of city government decisions about whether to inspect and work on a reported incident. At each decision in the pipeline, we define parity definitions and tests to identify inefficient, inequitable treatment. Disparities in resource allocation and scheduling across census tracts are reported as preliminary results.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3313831.3376533
