An official website of the United States government
High-velocity workloads and intricate task dependencies inherent in distributed stream-processing systems pose a fundamental challenge to efficient resource allocation. Traditional heuristic and single-agent reinforcement learning (RL) schedulers frequently fail to recognize these complex network and data-flow interactions, leading to severe resource fragmentation and catastrophic tail latency spikes. In order to accomplish coordinated, low-latency scheduling, we propose a Topology-Aware Multi-Agent Reinforcement Learning (TAMARL) framework utilizing a Centralized Training and Decentralized Execution (CTDE) architecture. TAMARL allows distributed agents to optimize task placement across heterogeneous cluster nodes and prevent backpressure cascades by integrating topology-aware state representations. We evaluate TAMARL on a production-grade cloud-native stack leveraging Apache Flink and Kubernetes. Compared to state-of-the-art baselines across six demanding stress-test scenarios, experimental evaluations demonstrate that TAMARL improves Service Level Objective (SLO) attainment by 27% while reducing P99 tail latency by up to 68%. Additionally, TAMARL maintains stable, resilient performance under 90% cluster utilization while securing 95% network locality.
more »
« less
This content will become publicly available on May 24, 2027
Online Adaptive Resource Allocation for Dynamic Stream Processing in Cloud Environments using Deep Reinforcement Learning
Dynamic stream processing in cloud environments presents unique challenges due to fluctuating data rates, heterogeneous workloads, and constrained resource availability. Traditional static and heuristic-based resource allocation techniques often fail to adapt effectively to these real-time dynamics, leading to suboptimal throughput, excessive latency, and violation of service-level agreements (SLAs). In this paper, we propose a Hierarchical Multi-Agent Deep Reinforcement Learning (H-MADRL) framework to address these limitations through autonomous, data-driven adaptation. Our system integrates Proximal Policy Optimization (PPO) agents with Kubernetes orchestration and Graph Neural Networks (GNNs) for topology-aware scaling decisions. Experiments under simulated and live workloads demonstrate the PPO agent's ability to maintain stable performance during load surges up to 5,000 events/sec without over-provisioning. Comparative evaluation with heuristic rule-based methods shows that the proposed DRL-based framework improves elasticity and responsiveness by up to 35% lower latency and 22% higher throughput, particularly under bursty or deceptive low-CPU workloads where heuristics fail to detect overload. Furthermore, the DRL agent consistently optimizes resource utilization, reducing unnecessary replica scaling while maintaining SLA targets. The system's reward function jointly balances throughput maximization, latency reduction, and cost efficiency, providing a robust alternative to traditional methods.
more »
« less
- Award ID(s):
- 2400459
- PAR ID:
- 10712804
- Publisher / Repository:
- IEEE
- Date Published:
- ISBN:
- 979-8-3195-4209-0
- Page Range / eLocation ID:
- 1 to 6
- Subject(s) / Keyword(s):
- adaptive scheduling dynamic stream processing resource allocation deep reinforcement learning cloud computing
- Format(s):
- Medium: X
- Location:
- Glasgow, United Kingdom
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Serverless Function-As-A-Service (FaaS) is an emerging cloud computing paradigm that frees application developers from infrastructure management tasks such as resource provisioning and scaling. To reduce the tail latency of functions and improve resource utilization, recent research has been focused on applying online learning algorithms such as reinforcement learning (RL) to manage resources. Compared to existing heuristics-based resource management approaches, RL-based approaches eliminate humans in the loop and avoid the painstaking generation of heuristics. In this paper, we show that the state-of-The-Art single-Agent RL algorithm (S-RL) suffers up to 4.6x higher function tail latency degradation on multi-Tenant serverless FaaS platforms and is unable to converge during training. We then propose and implement a customized multi-Agent RL algorithm based on Proximal Policy Optimization, i.e., multi-Agent PPO (MA-PPO). We show that in multi-Tenant environments, MA-PPO enables each agent to be trained until convergence and provides online performance comparable to S-RL in single-Tenant cases with less than 10% degradation. Besides, MA-PPO provides a 4.4x improvement in S-RL performance (in terms of function tail latency) in multi-Tenant cases.more » « less
-
Modern cloud datacenters increasingly execute heterogeneous workloads composed of latency-sensitive short jobs and resource-intensive long jobs, often exhibiting complex task dependency structures and significant runtime uncertainty. Existing cluster schedulers rely primarily on static heuristics and handcrafted policies, which struggle to simultaneously achieve low tail latency, high resource utilization, and robustness to execution-time mis-estimation at scale. In this paper, we propose an online uncertainty-aware hierarchical scheduling framework that integrates Deep Reinforcement Learning (DRL) into a hybrid cloud scheduler. By formulating the scheduling problem as a Markov Decision Process (MDP), the scheduler dynamically learns to select scheduling targets, probing aggressiveness, and priority modes based on observed system states. To validate our approach, we implemented a custom discrete-event simulation environment using CloudSim, generating dynamic workloads inspired by production cluster traces. Experimental results demonstrate that our learning-based scheduler prevents queue saturation entirely, effectively mitigating head-of-line blocking and providing near-instantaneous recovery from transient workload spikes compared to traditional static sampling techniques.more » « less
-
Large deep neural networks (DNNs), especially transformer-based and multimodal architectures, are computationally demanding and challenging to deploy on resource-constrained edge platforms like field robots. These challenges intensify in mission-critical scenarios (e.g., disaster response), where robots must collaborate under tight constraints on bandwidth, latency, and battery life, often without infrastructure or server support. To address these limitations, we present COHORT, a collaborative DNN inference and task-execution framework for multi-robot systems built on the Robotic Operating System (ROS). COHORT employs a hybrid offline-online reinforcement learning (RL) strategy to dynamically schedule and distribute DNN module execution across robots. Our key contributions are threefold: (a) Offline RL policy learning combined with Advantage-Weighted Regression (AWR), trained on auction-based task allocation data from heterogeneous DNN workloads across distributed robots, (b) Online policy adaptation via Multi-Agent PPO (MAPPO), initialized from the offline policy and fine-tuned in real time, and (c) comprehensive evaluation of COHORT on vision-language model (VLM) inference tasks such as CLIP and SAM, analyzing scalability with increasing robot/workload and robustness under. We benchmark COHORT against genetic algorithms and multiple RL baselines. Experimental results demonstrate that COHORT reduces battery consumption by 15.4%, and increases GPU utilization by 51.67%, while satisfying frame-rate and deadline constraints 2.55x of the time.more » « less
-
Dynamic spectrum slicing is a critical enabler for 6G Radio Access Networks (RANs), allowing the coexistence of heterogeneous services. However, optimizing resource allocation in dense, interference-limited deployments remains challenging due to non-stationary channel dynamics, strict Quality-of-Service (QoS) requirements, and the need for data privacy. In this paper, we propose SliceFed, a novel Federated Constrained Multi-Agent Deep Reinforcement Learning (F-MADRL) framework. SliceFed formulates the slicing problem as a Constrained Markov Decision Process (CMDP) where autonomous gNB agents maximize spectral efficiency while explicitly satisfying inter-cell interference budgets and hard ultra-reliable low-latency communication (URLLC) latency deadlines. We employ a Lagrangian primal-dual approach integrated with Proximal Policy Optimization (PPO) to enforce constraints, while Federated Averaging enables collaborative learning without exchanging raw local data. Extensive simulations in a dense multi-cell environment demonstrate that SliceFed converges to a stable, safety-aware policy. Unlike heuristic and unconstrained baselines, SliceFed achieves nearly 100% satisfaction of 1 ms URLLC latency deadlines and exhibits superior robustness to traffic load variations, verifying its potential for reliable and scalable 6G spectrum management.more » « less
