Search for: All records

As large language models (LLMs) expand the power of natural language processing to handle long inputs, rigorous and systematic analyses are necessary to understand their abilities and behavior. A salient application is summarization, due to its ubiquity and controversy (e.g., researchers have declared the death of summarization). In this paper, we use financial report summarization as a case study because financial reports are not only long but also use numbers and tables extensively. We propose a computational framework for characterizing multimodal long-form summarization and investigate the behavior of Claude 2.0/2.1, GPT-4/3.5, and Cohere. We find that GPT-3.5 and Cohere fail to perform this summarization task meaningfully. For Claude 2 and GPT-4, we analyze the extractiveness of the summary and identify a position bias in LLMs. This position bias disappears after shuffling the input for Claude, which suggests that Claude seems to recognize important information. We also conduct a comprehensive investigation on the use of numeric data in LLM-generated summaries and offer a taxonomy of numeric hallucination. We employ prompt engineering to improve GPT-4's use of numbers with limited success. Overall, our analyses highlight the strong capability of Claude 2 in handling long multimodal inputs compared to GPT-4. The generated summaries and evaluation code are available at https://github.com/ChicagoHAI/characterizing-multimodal-long-form-summarization. 

Increases in the deployment of machine learning algorithms for applications that deal with sensitive data have brought attention to the issue of fairness in machine learning. Many works have been devoted to applications that require different demographic groups to be treated fairly. However, algorithms that aim to satisfy inter-group fairness (also called group fairness) may inadvertently treat individuals within the same demographic group unfairly. To address this issue, this article introduces a formal definition of within-group fairness that maintains fairness among individuals from within the same group. A pre-processing framework is proposed to meet both inter- and within-group fairness criteria with little compromise in performance. The framework maps the feature vectors of members from different groups to an inter-group fair canonical domain before feeding them into a scoring function. The mapping is constructed to preserve the relative relationship between the scores obtained from the unprocessed feature vectors of individuals from the same demographic group, guaranteeing within-group fairness. This framework has been applied to the Adult, COMPAS risk assessment, and Law School datasets, and its performance is demonstrated and compared with two regularization-based methods in achieving inter-group and within-group fairness. 

Market simulation is an increasingly important method for evaluating and training trading strategies and testing "what if" scenarios. The extent to which results from these simulations can be trusted depends on how realistic the environment is for the strategies being tested. As a step towards providing benchmarks for realistic simulated markets, we enumerate measurable stylized facts of limit order book (LOB) markets across multiple asset classes from the literature. We apply these metrics to data from real markets and compare the results to data originating from simulated markets. We illustrate their use in five different simulated market configurations: The first (market replay) is frequently used in practice to evaluate trading strategies; the other four are interactive agent based simulation (IABS) configurations which combine zero intelligence agents, and agents with limited strategic behavior. These simulated agents rely on an internal "oracle" that provides a fundamental value for the asset. In traditional IABS methods the fundamental originates from a mean reverting random walk. We show that markets exhibit more realistic behavior when the fundamental arises from historical market data. We further experimentally illustrate the effectiveness of IABS techniques as opposed to market replay.