An official website of the United States government
Abstract Research on best practices in theory assessment highlights that testing theories is challenging because they inherit a new set of assumptions as soon as they are linked to a specific methodology. In this article, we integrate and build on this work by demonstrating the breadth of these challenges. We show that tracking auxiliary assumptions is difficult because they are made at different stages of theory testing and at multiple levels of a theory. We focus on these issues in a reanalysis of a seminal study and its replications, both of which use a simple working-memory paradigm and a mainstream computational modeling approach. These studies provide the main evidence for “all-or-none” recognition models of visual working memory and are still used as the basis for how to measure performance in popular visual working-memory tasks. In our reanalysis, we find that core practical auxiliary assumptions were unchecked and violated; the original model comparison metrics and data were not diagnostic in several experiments. Furthermore, we find that models were not matched on “theory general” auxiliary assumptions, meaning that the set of tested models was restricted, and not matched in theoretical scope. After testing these auxiliary assumptions and identifying diagnostic testing conditions, we find evidence for the opposite conclusion. That is, continuous resource models outperform all-or-none models. Together, our work demonstrates why tracking and testing auxiliary assumptions remains a fundamental challenge, even in prominent studies led by careful, computationally minded researchers. Our work also serves as a conceptual guide on how to identify and test the gamut of auxiliary assumptions in theory assessment, and we discuss these ideas in the context of contemporary approaches to scientific discovery.
more »
« less
Four Internal Inconsistencies in Tversky and Kahneman’s (1992) Cumulative Prospect Theory Article: A Case Study in Ambiguous Theoretical Scope and Ambiguous Parsimony
Scholars heavily rely on theoretical scope as a tool to challenge existing theory. We advocate that scientific discovery could be accelerated if far more effort were invested into also overtly specifying and painstakingly delineating the intended purview of any proposed new theory at the time of its inception. As a case study, we consider Tversky and Kahneman (1992). They motivated their Nobel-Prize-winning cumulative prospect theory with evidence that in each of two studies, roughly half of the participants violated independence, a property required by expected utility theory (EUT). Yet even at the time of inception, new theories may reveal signs of their own limited scope. For example, we show that Tversky and Kahneman’s findings in their own test of loss aversion provide evidence that at least half of their participants violated their theory, in turn, in that study. We highlight a combination of conflicting findings in the original article that make it ambiguous to evaluate both cumulative prospect theory’s scope and its parsimony on the authors’ own evidence. The Tversky and Kahneman article is illustrative of a social and behavioral research culture in which theoretical scope plays an extremely asymmetric role: to call existing theory into question and motivate surrogate proposals.
more »
« less
- Award ID(s):
- 2049896
- PAR ID:
- 10405277
- Date Published:
- Journal Name:
- Advances in Methods and Practices in Psychological Science
- Volume:
- 5
- Issue:
- 1
- ISSN:
- 2515-2459
- Page Range / eLocation ID:
- 251524592210746
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Soot is known for its enormous and pervasive negative impacts on human health and the environmental, but there much about soot that is not well known, including the precursors and chemical mechanisms involved in its formation. Many studies have characterized species associated with incipient particles, i.e., the first particles produced during soot formation. These studies provide insight into inception mechanisms, the pathways leading from gas-phase precursors to condensed-phase particles. Potential inception mechanisms involve one (or a combination) of two classes of pathways: physical nucleation, in which precursors undergo a thermodynamic phase change and are bound together by electrostatic forces, and chemical clustering, in which precursors react to form covalently bound clusters. In a recent paper, Shao et al.1 concluded that soot inception occurs through physical nucleation and claimed to have provided direct evidence of such a mechanism. We demonstrate that this conclusion is inconsistent with (1) the consensus of published work, (2) the data, theory, and analysis on which this conclusion is nominally based, and (3) the second law of thermodynamics. We show that, contrary to their conclusions, their experimental and theoretical results provide evidence for a chemical-clustering soot-inception mechanism.more » « less
-
Contextual bandits, which leverage baseline features of sequentially arriving individuals to optimize cumulative rewards while balancing exploration and exploitation, are critical for online decision-making. Existing approaches typically assume no interference, where each individual’s action affects only their own reward. Yet, such an assumption can be violated in many practical scenarios, and the oversight of interference can lead to short-sighted policies that focus solely on maximizing the immediate outcomes for individuals, which further results in suboptimal decisions and potentially increased regret over time. To address this significant gap, we introduce the foresighted online policy with interference (FRONT) that innovatively considers the long-term impact of the current decision on subsequent decisions and rewards.more » « less
-
null (Ed.)Abstract The Earth's population will become more than 80% urban during this century. This threshold is often regarded as sufficient justification for pursuing urban ecology. However, pursuit has primarily focused on building empirical richness, and urban ecology theory is rarely discussed. The Baltimore Ecosystem Study (BES) has been grounded in theory since its inception and its two decades of data collection have stimulated progress toward comprehensive urban theory. Emerging urban ecology theory integrates biology, physical sciences, social sciences, and urban design, probes interdisciplinary frontiers while being founded on textbook disciplinary theories, and accommodates surprising empirical results. Theoretical growth in urban ecology has relied on refined frameworks, increased disciplinary scope, and longevity of interdisciplinary interactions. We describe the theories used by BES initially, and trace ongoing theoretical development that increasingly reflects the hybrid biological–physical–social nature of the Baltimore ecosystem. The specific mix of theories used in Baltimore likely will require modification when applied to other urban areas, but the developmental process, and the key results, will continue to benefit other urban social–ecological research projects.more » « less
-
We present a speculative application of model estimates from Fudenberg and Puri (2021) to prize-linked savings in South Africa. The models used include one combining simplicity theory (Puri 2018, 2022), a preference for lotteries with fewer possible outcomes, with cumulative prospect theory. The results and those of prior literature indicate that both simplicity and probability weighting have a role to play in understanding behavior in choice under risk. We discuss the properties of these models and their implications for behavior.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1177/25152459221074653
