Search for: All records

Contrastive learning has served as a powerful framework in the early development of vision–language models (VLMs), demonstrating remarkable effectiveness in learning generalizable representations and establishing itself as the foundation for many state-of-the-art systems. However, despite these advances, its theoretical understanding remains limited, particularly under imbalanced data distributions that are prevalent in real-world settings. Such imbalance can degrade representation quality and induce biased model behavior, yet a rigorous characterization of these effects is still lacking. In this work, we develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders under imbalanced data. Our results reveal that neuron weights evolve differently across three stages of training, with distinct dynamics for majority features, minority features, and the noise. We further show that minority features diminish neurons’ representational capacity, increase the need for more complex architectures, and impair the separation of ground-truth features from noise. These findings offer new theoretical insights into how data imbalance shapes learning in contrastive frameworks and serve as an early step towards principled modifications for developing more robust and unbiased representations. 

While Transformers rely on a distinctive attention mechanism, the recent emergence of Mamba and other selective state space models (SSMs) offers a strong alternative. These models incorporate attention-like mechanisms with hardware-aware efficiency and a unique selection strategy, yet their theoretical properties remain poorly understood. In this work, we present a first-step theoretical analysis of the selection mechanism in Mamba. We study a simplified single-layer Mamba block trained with gradient descent on structured data containing both label-relevant and irrelevant tokens. Our results show that the gating vector dynamically aligns with label-relevant features while negating irrelevant ones, formalizing its role as an implicit feature selector. Moreover, we prove that training achieves guaranteed generalization, with explicit bounds on sample size and convergence rate. These f indings offer principled insight into when and why Mamba’s selection mechanism enables efficient learning, offering a theoretical counterpoint to Transformer-centric explanations of generalization. 

Contrastive learning has served as a powerful framework in the early development of vision–language models (VLMs), demonstrating remarkable effectiveness in learning generalizable representations and establishing itself as the foundation for many state-of-the-art systems. However, despite these advances, its theoretical understanding remains limited, particularly under imbalanced data distributions that are prevalent in real-world settings. Such imbalance can degrade representation quality and induce biased model behavior, yet a rigorous characterization of these effects is still lacking. In this work, we develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders under imbalanced data. Our results reveal that neuron weights evolve differently across three stages of training, with distinct dynamics for majority features, minority features, and the noise. We further show that minority features diminish neurons’ representational capacity, increase the need for more complex architectures, and impair the separation of ground-truth features from noise. These findings offer new theoretical insights into how data imbalance shapes learning in contrastive frameworks and serve as an early step towards principled modifications for developing more robust and unbiased representations. 

Radiomap characterizes geographical radio spectrum coverage and can facilitate resource allocation and management of wireless networks. One practical radiomap estimation (RME) task is to form a full radiomap from sparse samples collected by sensors or mobile devices. Often, traditional RME approaches focus on statistical data distributions without exploiting the underlying spatial correlations among sparse observations. Utilizing geometric/geographical path correlation, this letter proposes a novel dual-phase RME method based on graph neural networks. In this Dual-phase Graph-based Radiomap Estimation (Dual-GRE) framework, the first phase integrates graph attention (GAT) networks with radio propagation models to construct a coarse-resolution (CR) radiomap to embed the spatial information and physical principles. Phase 2 utilizes a deep convolution neural network that uses the CR radiomap and landscape information to derive fine-resolution radiomaps. Our experimental results demonstrate the power of physics-integrated GAT in capturing the spatial spectrum information, together with the efficiency of the proposed Dual-GRE in radiomap estimation.