Search for: All records

Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs sit idle, waiting for slow CPU updates and limited-bandwidth PCIe transfers. We present ZenFlow, a new offloaded training framework that decouples updates between GPU and CPU by prioritizing important gradients. ZenFlow performs in-place updates of important gradients on GPU while offloading and accumulating less important ones on CPU without blocking training iterations, fully overlapping CPU work with GPU computation. To make this importance-aware design scalable, ZenFlow leverages a novel spatio-temporal locality of important gradients—concentrated in specific positions that remain stable across iterations—to make important gradient identification efficient and lightweight while preserving fidelity. Evaluation shows that ZenFlow achieves up to 5×end-to-end speedup, 2×lower PCIe traffic, 85% less GPU stalls, and up to 72% lower cloud GPU cost, all while preserving accuracy. 

Abstract We analyzed 19,123 natural language processing-related studies to explore the differences in task distributions and application contexts between large language models (LLMs) and non-LLM methods in health care. Through topic modeling analysis, we found that LLMs demonstrate advantages in open-ended tasks, while non-LLM methods dominate in information extraction tasks. These findings highlight the complementary strengths of the two technical paradigms and provide reference for their integration strategies in future health care applications. 

Large language models (LLMs) have shown exceptional capabilities in code understanding and generation. However, they still face significant challenges in analyzing and debugging code. Most existing works rely on a single model, which often struggles to detect and fix bugs in complex program structures and semantic logic. This paper presents DoTA, a novel debugging framework that enhances LLMs' debugging capabilities through multiagent collaboration. Our key innovation is two-fold. First, we enhance code understanding through automated hierarchical documentation analysis, enabling more effective bug detection and localization based on comprehensive program context. Second, we leverage a delta-of-thoughts process where multi LLM agents analyze different aspects of program correctness and iteratively contribute complementary insights to identify bugs. Our experiments show that combining different LLM based agents with enriched documentation context significantly improves LLMs' debugging capabilities. DoTA has been extensively evaluated on the Debug- Bench dataset of 4,253 debugging instances. It achieves an average improvement of 15% in bug detection accuracy across languages compared to GPT-3.5 with task background prompting. The framework shows particular strength in handling complex logical errors (+18.3%) and multiple bugs (+18.9%). On open-source models, DoTA enhances bug detection capabilities by 10.4–13.5%. 

Surfactants at the air–sea interface are known to alter surface wave dynamics by modifying surface tension and Marangoni stresses. In this study, we perform two-dimensional direct numerical simulations of gravity-capillary waves with insoluble surfactants using a coupled phase field and volume-of-fluid method. We consider a nonlinear equation of state for surface tension and resolve Marangoni stresses induced by surfactant concentration gradients. We explore a broad parameter space characterised by initial wave steepness, Bond number Bo (comparing gravity and surface tension), Reynolds number Re (comparing inertia and viscosity), and the importance of surfactant concentration and strength of the gradient, characterised by a surfactant parameter beta. We analyse the impact of surfactants on wave patterns, surface roughness, wave breaking, energy dissipation and surface vorticity. Our results reveal a non-monotonic dependence of wave shape, roughness, vorticity and energy dissipation on beta, which is found to be governed by Marangoni effects that peak at intermediate surfactant concentrations. Wave regime transition at high is governed by an effective Bo number, which accounts for the reduction in surface tension induced by surfactants. We further introduce a rescaled parameter based on force balance, which collapses the transition boundaries across different=Re. These findings provide a systematic understanding of surfactant-modulated wave dynamics for both laboratory and geophysical applications. 

In this paper, we introduce DREAM, Distributed Regional Efficient Agent Management, a novel method using Large Language Models (LLMs) to solve Multi-Agent Pathfinding (MAPF) problems in complicated environments. Our approach splits up the area into various local regions and an LLM agent handles each one of them intelligently in reasoning and decision making. We present some novel designs in our system: 1) Adaptive region management and allocation to regions, supporting the dynamic partitioning of different complexity or density areas. 2) The multi-level LLM-driven agents collaboration framework that enables peer-peer, interLLM coordination and controls for effective monitoring intelligence across a hierarchical path planning organization hierarchy level ensures autonomy whilst improving overall understanding among LLM agents, leading to more accurate planning decisions from real-time analysis. (3) Failurereflection- replanning mechanism integrated within an individual LLM's management scope eventually results continual improvement. (4) LLM agents can do function calling to interact with the typical algorithms also. Our system successfully processes complex and large-scale MAPF scenarios by merging the higher-orderality of reasoning capabilities in LLMs with this novel distributed framework. For instance, the distributed and hierarchical nature of this approach helps to break a high-dimensional MAPF problem into several groups of smaller dimension. As such, this approach also opens up the development of AI language models in more complex robotics and logistics scenarios, potentially changing how multi-agent coordination is done for actual situations. 

We report multiple negative differential conductance (NDC) features observed under both forward and reverse bias voltages in hybrid-cladding interband cascade lasers (ICLs). The ICL with an emission wavelength of 2.77 μm at 300 K has a threshold current density of 183 A/cm2, which is lower than previously reported values from type-II ICLs at similar and somewhat longer wavelengths. The number of NDC peaks aligns with the number of cascade stages and the current densities of the peaks are below 100 and 1 mA/cm2 for reverse and forward bias voltages, respectively, with a peak to valley current ratio of 1.8 and 1.7, respectively, at 80 K. These NDC features are observable at temperatures above 400 K under reverse bias and up to 160 K under forward bias. Hysteresis was observed with these NDC features. Since the unique features discovered in this work are inherently correlated with resonant tunneling in quantum well structures and interband transitions in ICLs, they may help improve understanding of carrier transport dynamics and provide insights into further development of ICLs as well as other potential applications. 

We report a detailed examination of multiple negative differential conductance (NDC) features with hysteresis, observed in interband cascade lasers and light emitting diodes made from six different wafers covering an emission wavelength range from 2.7 to about 7 μm. These features arise from resonant tunneling in quantum wells (QWs), can be observed in both forward and reverse biases, and persist up to 160 K and over 300 K, respectively. By comparing features from wafers with a variable number of cascade stages (Ns) and different electron injector doping concentrations, strong correlations are established between the Ns and the number of NDC peaks, as well as between the appearance of forward-bias NDC features and the doping concentration. Under reverse bias, two distinct groupings of NDC peaks appear, which can be plausibly attributed to resonant tunneling through both ground and excited states in QWs. By comparing different internal structures, this tunneling can be isolated to the cascade stages of the emitters. NDC analysis is also shown to be useful as a tool for diagnosing partial internal damage within devices, providing a simple non-destructive way to test device performance.