More Like this

1. PIMCloud: QoS-Aware Resource Management of Latency-Critical Applications in Clouds with Processing-in-Memory

   Shuang Chen, Yi Jiang (April 2022, 28th IEEE International Symposium on High-Performance Computer Architecture (HPCA-28))

   The slowdown of Moore’s Law, combined with advances in 3D stacking of logic and memory, have pushed architects to revisit the concept of processing-in-memory (PIM) to overcome the memory wall bottleneck. This PIM renaissance finds itself in a very different computing landscape from the one twenty years ago, as more and more computation shifts to the cloud. Most PIM architecture papers still focus on best-effort applications, while PIM’s impact on latency-critical cloud applications is not well understood. This paper explores how datacenters can exploit PIM architectures in the context of latency-critical applications. We adopt a general-purpose cloud server with HBM-based, 3D-stacked logic+memory modules, and study the impact of PIM on six diverse interactive cloud applications. We reveal the previously neglected opportunity that PIM presents to these services, and show the importance of properly managing PIM-related resources to meet the QoS targets of interactive services and maximize resource efficiency. Then, we present PIMCloud, a QoS-aware resource manager designed for cloud systems with PIM allowing colocation of multiple latency-critical and best-effort applications. We show that PIMCloud efficiently manages PIM resources: it (1) improves effective machine utilization by up to 70% and 85% (average 24% and 33%) under 2-app and 3-app mixes, compared to the best state-of-the-art manager; (2) helps latency-critical applications meet QoS; and (3) adapts to varying load patterns. 

   more » « less
2. Fast and Efficient Scaling for Microservices with SurgeGuard

   https://doi.org/10.1109/SC41406.2024.00103  

   Ghosh, Anyesha; Yadwadkar, Neeraja J; Erez, Mattan (November 2024, IEEE)

   The microservice architecture is increasingly popular for flexible, large-scale online applications. However, existing resource management mechanisms incur high latency in detecting Quality of Service (QoS) violations, and hence, fail to allocate resources effectively under commonly-observed varying load conditions. This results in over-allocation coupled with a late response that increase both the total cost of ownership and the magnitude of each QoS violation event. We present SurgeGuard, a decentralized resource controller for microservice applications specifically designed to guard application QoS during surges in load and network latency. SurgeGuard uses the key insight that for rapid detection and effective management of QoS violations, the controller must be aware of any available slack in latency and communication patterns between microservices within a task-graph. Our experiments show that for the workloads in DeathStarBench, SurgeGuard on average reduces the combined violation magnitude and duration by 61.1% and 93.7%, respectively, compared to the well-known Parties and Caladan algorithms, and requires 8% fewer resources than Parties 

   more » « less
3. Cilantro: Performance-Aware Resource Allocation for General Objectives via Online Feedback

   Bhardwaj, Romil; Kandasamy, Kirthevasan; Biswal, Asim; Guo, Wenshuo; Hindman, Benjamin; Gonzalez, Joseph; Jordan, Michael I; Stoica, Ion (July 2023, USENIX Association)

   Traditional systems for allocating finite cluster resources among competing jobs have either aimed at providing fairness, relied on users to specify their resource requirements, or have estimated these requirements via surrogate metrics (e.g. CPU utilization). These approaches do not account for a job’s real world performance (e.g. P95 latency). Existing performance-aware systems use offline profiled data and/or are designed for specific allocation objectives. In this work, we argue that resource allocation systems should directly account for real-world performance and the varied allocation objectives of users. In this pursuit, we build Cilantro. At the core of Cilantro is an online learning mechanism which forms feedback loops with the jobs to estimate the resource to performance mappings and load shifts. This relieves users from the onerous task of job profiling and collects reliable real-time feedback. This is then used to achieve a variety of user-specified scheduling objectives. Cilantro handles the uncertainty in the learned models by adapting the underlying policy to work with confidence bounds. We demonstrate this in two settings. First, in a multi-tenant 1000 CPU cluster with 20 independent jobs, three of Cilantro’s policies outperform 9 other baselines on three different performance-aware scheduling objectives, improving user utilities by up to 1.2 − 3.7x. Second, in a microservices setting, where 160 CPUs must be distributed between 19 inter-dependent microservices, Cilantro outperforms 3 other baselines, reducing the end-to-end P99 latency to x0.57 the next best baseline. 

   more » « less
4. QaSAL: QoS-aware State-Augmented Learnable Algorithms for Coexistence of 5G NR-U/Wi-Fi

   https://doi.org/10.1109/CISS64860.2025.10944641  

   Fasihi, Mohammad Reza; Mark, Brian L (March 2025, IEEE)

   With the increasing demand for wireless connectivity, ensuring the efficient coexistence of multiple radio access technologies in shared unlicensed spectrum has become an important issue. This paper focuses on optimizing Medium Access Control (MAC) parameters to enhance the coexistence of 5G New Radio in Unlicensed Spectrum (NR-U) and Wi-Fi networks operating in unlicensed spectrum with multiple priority classes of traffic that may have varying quality-of-service (QoS) requirements. In this context, we tackle the coexistence parameter management problem by introducing a QoS-aware State-Augmented Learnable (QaSAL) framework, designed to improve network performance under various traffic conditions. Our approach augments the state representation with constraint information, enabling dynamic policy adjustments to enforce QoS requirements effectively. Simulation results validate the effectiveness of QaSAL in managing NR-U and Wi-Fi coexistence, demonstrating improved channel access fairness while satisfying a latency constraint for high-priority traffic. 

   more » « less
5. Dělen: Enabling Flexible and Adaptive Model-serving for Multi-tenant Edge AI

   https://doi.org/10.1145/3576842.3582375  

   Liang, Qianlin; Hanafy, Walid A.; Bashir, Noman; Ali-Eldin, Ahmed; Irwin, David; Shenoy, Prashant (May 2023, ACM/IEEE Conference on Internet of Things Design and Implementation)

   Model-serving systems expose machine learning (ML) models to applications programmatically via a high-level API. Cloud plat- forms use these systems to mask the complexities of optimally managing resources and servicing inference requests across multi- ple applications. Model serving at the edge is now also becoming increasingly important to support inference workloads with tight latency requirements. However, edge model serving differs substan- tially from cloud model serving in its latency, energy, and accuracy constraints: these systems must support multiple applications with widely different latency and accuracy requirements on embedded edge accelerators with limited computational and energy resources. To address the problem, this paper presents Dělen,1 a flexible and adaptive model-serving system for multi-tenant edge AI. Dělen exposes a high-level API that enables individual edge applications to specify a bound at runtime on the latency, accuracy, or energy of their inference requests. We efficiently implement Dělen using conditional execution in multi-exit deep neural networks (DNNs), which enables granular control over inference requests, and evalu- ate it on a resource-constrained Jetson Nano edge accelerator. We evaluate Dělen flexibility by implementing state-of-the-art adapta- tion policies using Dělen’s API, and evaluate its adaptability under different workload dynamics and goals when running single and multiple applications. 

   more » « less