skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Exploring Prompting Approaches in Legal Textual Entailment
We report explorations into prompt engineering with large pre-trained language models that were not fine-tuned to solve the legal entailment task (Task 4) of the 2023 COLIEE competition. Our most successful strategy used simple text similarity measures to retrieve articles and queries from the training set. We report on our efforts to optimize performance with both OpenAI’s GPT-4 and FLaN-T5. We also used an ensemble approach to find the best combination of models and prompts. Finally, we analyze our results and suggest ideas for future improvements.  more » « less
Award ID(s):
2311286
PAR ID:
10535280
Author(s) / Creator(s):
; ; ; ; ; ;
Publisher / Repository:
The Review of Socionetwork Strategies
Date Published:
Journal Name:
The Review of Socionetwork Strategies
Volume:
18
Issue:
1
ISSN:
2523-3173
Page Range / eLocation ID:
75 to 100
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. In this chapter, we address the problem of why blockages occur during mathematization by introducing a method for studying mathematizing based in quantitative reasoning. We report on interview data with six tertiary STEM majors as they developed models of the population dynamics of cats and birds in a backyard habitat. Our analysis focused on real-world relationships participants tried to express when using a given arithmetic operation in a predator-prey modelling task. Our results reveal the conceptions of × participants used to justify their models when constructing an expression for the decrease in the bird population. We conclude by discussing the method’s utility for studying mathematization and with conjectures on how instructors might leverage participants’ justifications to scaffold their emergent models towards a conventionally correct model. 
    more » « less
  2. In this paper, we provide comparison of language features and runtime systems of commonly used threading parallel programming models for high performance computing, including OpenMP, Intel Cilk Plus, Intel TBB, OpenACC, Nvidia CUDA, OpenCL, C++11 and PThreads. We then report our performance comparison of OpenMP, Cilk Plus and C++11 for data and task parallelism on CPU using benchmarks. The results show that the performance varies with respect to factors such as runtime scheduling strategies, overhead of enabling parallelism and synchronization, load balancing and uniformity of task workload among threads in applications. Our study summarizes and categorizes the latest development of threading programming APIs for supporting existing and emerging computer architectures, and provides tables that compare all features of different APIs. It could be used as a guide for users to choose the APIs for their applications according to their features, interface and performance reported. 
    more » « less
  3. Complex neural network architectures are being increasingly used to learn to compute the semantic resemblances among natural language texts. It is necessary to establish a lower bound of performance that must be met in or- der for new complex architectures to be not only novel, but also worthwhile in terms of implementation. This paper focuses on the specific task of determin- ing semantic textual similarity (STS). We construct a number of models from simple to complex within a framework and report our results. Our findings show that a small number of LSTM stacks with an LSTM stack comparator produces the best results. We use Se- mEval 2017 STS Competition Dataset for evaluation. 
    more » « less
  4. Important natural resources in the Arctic rely heavily on sea ice, making it important to forecast Arctic sea ice changes. Arctic sea ice forecasting often involves two connected tasks: sea ice concentration at each pixel and overall sea ice extent. Instead of having two separate models for two forecasting tasks, in this report, we study how to use multi-task learning techniques and leverage the connections between ice concentration and ice extent to improve accuracy for both prediction tasks. Because of the spatiotemporal nature of the data, we designed two novel multi-task learning models based on CNNs and ConvLSTMs, respectively. We also developed a custom loss function which trains the models to ignore land pixels when making predictions. Our experiments show our models can have better accuracies than separate models that predict sea ice extent and concentration separately, and that our accuracies are better than or comparable with results in the state-of-the-art studies. 
    more » « less
  5. In the realm of collaborative learning, extracting the beliefs shared within a group is a critical capability to navigate complex tasks. Inherent in this problem is the fact that in naturalistic collaborative discourse, the same propositional content may be expressed in radically different ways. This difficulty is exacerbated when speech overlaps and other communicative modalities are used, as would be the case in a co-situated collaborative task. In this paper, we conduct a comparative methodological analysis of extraction techniques for task-relevant propositions from natural speech dialogues in a challenging shared task setting where participants collaboratively determine the weights of five blocks using only a balance scale. We encode utterances and candidate propositions through language models and compare a cross-encoder method, adapted from coreference research, to a vector similarity baseline. Our cross-encoder approach outperforms both a cosine similarity baseline and zero-shot inference by both the GPT-4 and LLaMA 2 language models, and we establish a novel baseline on this challenging task on two collaborative task datasets---the Weights Task and DeliData---showing the generalizability of our approach. Furthermore, we explore the use of state of the art large language models for data augmentation to enhance performance, extend our examination to transcripts generated by Google's Automatic Speech Recognition system to assess the potential for automating the propositional extraction process in real-time, and introduce a framework for live propositional extraction from natural speech and multimodal signals. This study not only demonstrates the feasibility of detecting collaboration-relevant content in unstructured interactions but also lays the groundwork for employing AI to enhance collaborative problem-solving in classrooms, and other collaborative settings, such as the workforce. Our code may be found at: (https://github.com/csu-signal/PropositionExtraction). 
    more » « less