Search for: All records

Generative large language models (LLMs) exhibit impressive capabilities, which can be further augmented by integrating a pre-trained vision model into the original LLM to create a multimodal LLM (MLLM). However, this integration often significantly decreases performance on natural language understanding and generation tasks, compared to the original LLM. This study investigates this issue using the LLaVA MLLM, treating the integration as a continual learning problem. We evaluate five continual learning methods to mitigate forgetting and identify a technique that enhances visual understanding while minimizing linguistic performance loss. Our approach reduces linguistic performance degradation by up to 15% over the LLaVA recipe, while maintaining high multimodal accuracy. We also demonstrate the robustness of our method through continual learning on a sequence of vision-language tasks, effectively preserving linguistic skills while acquiring new multimodal capabilities. 

Schistosomiasis is a parasitic disease with significant global health and socio-economic implications. Drug discovery for schistosomiasis typically involves high-content whole-organism screening. In this approach, parasites are ex-posed to various chemical compounds and their systemic, whole-organism-level responses are captured via microscopy and analyzed to obtain a quanti-tative assessment of chemical effect. These effects are multidimensional and time-varying, impacting shape, appearance, and behavior. Accurate identifi-cation of object boundaries is essential for preparing images for subsequent analysis in high-content studies. Object segmentation is one of the most deeply studied problems in computer vision where recent efforts have incor-porated deep learning. Emerging results indicate that acquiring robust fea-tures in spectral domain using Fast Fourier Transform (FFT) within Deep Neural Networks (DNNs) can enhance segmentation accuracy. In this paper, we explore this direction further and propose a latent space Phase-Gating (PG) method that builds upon FFT and leverages phase information to effi-ciently identify globally significant features. While the importance of phase in analyzing signals has long been known, technical difficulties in calculat-ing phase in manners that are invariant to imaging parameters has limited its use. A key result of this paper is to show how phase information can be in-corporated in neural architectures that are compact. Experiments conducted on complex HCS datasets demonstrate how this idea leads to improved seg-mentation accuracy, while maintaining robustness against commonly en-countered noise (blurring) in HCS. The compactness of the proposed method also makes it well-suited for application specific architectures (ASIC) de-signed for high-content screening. 

We establish the first finite-time logarithmic regret bounds for the self-tuning regulation problem. We introduce a modified version of the certainty equivalence algorithm, which we call PIECE, that clips inputs in addition to utilizing probing inputs for exploration. We show that it has a ClogT upper bound on the regret after T time-steps for bounded noise, and Clog3T in the case of sub-Gaussian noise, unlike the LQ problem where logarithmic regret is shown to be not possible. The PIECE algorithm is also designed to address the critical challenge of poor initial transient performance of reinforcement learning algorithms for linear systems. Comparative simulation results illustrate the improved performance of PIECE. 

Social media has become an important method for information sharing. This has also created opportunities for bad actors to easily spread disinformation and manipulate public opinion. This paper explores the possibility of applying Authorship Verification on online communities to mitigate abuse by analyzing the writing style of online accounts to identify accounts managed by the same person. We expand on our similarity-based authorship verification approach, previously applied on large fanfictions, and show that it works in open-world settings, shorter documents, and is largely topic-agnostic. Our expanded model can link Reddit accounts based on the writing style of only 40 comments with an AUC of 0.95, and the performance increases to 0.98 given more content. We apply this model on a set of suspicious Reddit accounts associated with the disinformation campaign surrounding the 2016 U.S. presidential election and show that the writing style of these accounts are inconsistent, indicating that each account was likely maintained by multiple individuals. We also apply this model to Reddit user accounts that commented on the WallStreetBets subreddit around the 2021 GameStop short squeeze and show that a number of account pairs share very similar writing styles. We also show that this approach can link accounts across Reddit and Twitter with an AUC of 0.91 even when training data is very limited. 

Debiased machine learning is a meta-algorithm based on bias correction and sample splitting to calculate confidence intervals for functionals, i.e., scalar summaries, of machine learning algorithms. For example, an analyst may seek the confidence interval for a treatment effect estimated with a neural network. We present a non-asymptotic debiased machine learning theorem that encompasses any global or local functional of any machine learning algorithm that satisfies a few simple, interpretable conditions. Formally, we prove consistency, Gaussian approximation and semiparametric efficiency by finite-sample arguments. The rate of convergence is $$n^{-1/2}$$ for global functionals, and it degrades gracefully for local functionals. Our results culminate in a simple set of conditions that an analyst can use to translate modern learning theory rates into traditional statistical inference. The conditions reveal a general double robustness property for ill-posed inverse problems. 

Opioid addiction constitutes a significant contemporary health crisis that is multifarious in its complexity. Modeling the epidemiology of any addiction is challenging in its own right. For opioid addiction, the challenge is exacerbated due to the difficulties in collecting real-time data and the circumscribed nature of information opioid users may disclose owing to stigma associated with prescription misuse. Given this context, identifying the progression of individuals through the stages of (opioid) addiction is one of the more acute problems in epidemiological modeling whose solution is crucial for designing specific interventions at both personal and population levels. We describe a computational approach for determining and characterizing addiction stages of opioid users from their social media posts. The proposed approach combines recurrent neural network learning with information-theoretic analysis of word-associations and context-based word embedding to determine addiction stage-specific language usage. Users who have a high likelihood for relapsing back to drug-use are identified and characterized using propensity score matching and logistic regression. Experimental evaluations indicate that the proposed approach can distinguish between various addiction stages and identify users prone to relapse with high accuracy as evidenced by F1 scores of 0.88 and 0.79 respectively