An official website of the United States government
Large language models (LLMs) have achieved widespread success on a variety of in-context few shot tasks, but this success is typically evaluated via correctness rather than consistency. We argue that self-consistency is an important criteria for valid multi-step reasoning in tasks where the solution is composed of the answers to multiple sub-steps. We propose two types of self consistency that are particularly important for multi-step reasoning – hypothetical consistency (a model’s ability to predict what its output would be in a hypothetical other context) and compositional consistency (consistency of a model’s final outputs when intermediate sub-steps are replaced with the model’s outputs for those steps). We demonstrate that multiple variants of the GPT-3/-4 models exhibit poor consistency rates across both types of consistency on a variety of tasks.
more »
« less
On the Consistency strength of MM(\omega_1)
We prove that the consistency of Martin's Maximum restricted to partial orders of cardinality \omega_1 follows from the consistency of ZFC.
more »
« less
- Award ID(s):
- 2300896
- PAR ID:
- 10535394
- Publisher / Repository:
- American Mathematical Society
- Date Published:
- Journal Name:
- Proceedings of the American Mathematical Society
- Volume:
- 152
- Issue:
- 5
- ISSN:
- 0002-9939
- Page Range / eLocation ID:
- 2229-2237
- Subject(s) / Keyword(s):
- 03E35 03E50
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Programming concurrent, distributed systems is hard---especially when these systems mutate shared, persistent state replicated at geographic scale. To enable high availability and scalability, a new class of weakly consistent data stores has become popular. However, some data needs strong consistency. To manipulate both weakly and strongly consistent data in a single transaction, we introduce a new abstraction: mixed-consistency transactions, embodied in a new embedded language, MixT. Programmers explicitly associate consistency models with remote storage sites; each atomic, isolated transaction can access a mixture of data with different consistency models. Compile-time information-flow checking, applied to consistency models, ensures that these models are mixed safely and enables the compiler to automatically partition transactions. New run-time mechanisms ensure that consistency models can also be mixed safely, even when the data used by a transaction resides on separate, mutually unaware stores. Performance measurements show that despite their stronger guarantees, mixed-consistency transactions retain much of the speed of weak consistency, significantly outperforming traditional serializable transactions.more » « less
-
This paper unifies the theory of consistent-set maximization for robust outlier detection in a simultaneous localization and mapping framework. We first describe the notion of pairwise consistency before discussing how a consistency graph can be formed by evaluating pairs of measurements for consistency. Finding the largest set of consistent measurements is transformed into an instance of the maximum clique problem and can be solved relatively quickly using existing maximum-clique solvers. We then generalize our algorithm to check consistency on a group- k basis by using a generalized notion of consistency and using generalized graphs. We also present modified maximum clique algorithms that function over generalized graphs to find the set of measurements that is internally group- k consistent. We address the exponential nature of group- k consistency and present methods that can substantially decrease the number of necessary checks performed when evaluating consistency. We extend our prior work to perform data association, and to multi-agent systems in both simulation and hardware, and provide a comparison with other state-of-the-art methods.more » « less
-
A dimer model is a quiver with faces embedded in a surface. We define and investigate notions of consistency for dimer models on general surfaces with boundary which restrict to well-studied consistency conditions in the disk and torus case. We define weak consistency in terms of the associated dimer algebra and show that it is equivalent to the absence of bad configurations on the strand diagram. In the disk and torus case, weakly consistent models are nondegenerate, meaning that every arrow is contained in a perfect matching; this is not true for general surfaces. Strong consistency is defined to require weak consistency as well as nondegeneracy. We prove that the completed as well as the noncompleted dimer algebra of a strongly consistent dimer model are bimodule internally 3-Calabi-Yau with respect to their boundary idempotents. As a consequence, the Gorenstein-projective module category of the completed boundary algebra of suitable dimer models categorifies the cluster algebra given by their underlying quiver. We provide additional consequences of weak and strong consistency, including that one may reduce a strongly consistent dimer model by removing digons and that consistency behaves well under taking dimer submodels.more » « less
-
Since the early days of relational databases, it was realized that acyclic hypergraphs give rise to database schemas with desirable structural and algorithmic properties. In a bynow classical paper, Beeri, Fagin, Maier, and Yannakakis established several different equivalent characterizations of acyclicity; in particular, they showed that the sets of attributes of a schema form an acyclic hypergraph if and only if the local-to-global consistency property for relations over that schema holds, which means that every collection of pairwise consistent relations over the schema is globally consistent. Even though real-life databases consist of bags (multisets), there has not been a study of the interplay between local consistency and global consistency for bags. We embark on such a study here and we first show that the sets of attributes of a schema form an acyclic hypergraph if and only if the local-to-global consistency property for bags over that schema holds. After this, we explore algorithmic aspects of global consistency for bags by analyzing the computational complexity of the global consistency problem for bags: given a collection of bags, are these bags globally consistent? We show that this problem is in NP, even when the schema is part of the input. We then establish the following dichotomy theorem for fixed schemas: if the schema is acyclic, then the global consistency problem for bags is solvable in polynomial time, while if the schema is cyclic, then the global consistency problem for bags is NP-complete. The latter result contrasts sharply with the state of affairs for relations, where, for each fixed schema, the global consistency problem for relations is solvable in polynomial time.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
