Search for: All records

Creators/Authors contains: "Skarlatos, Dimitrios"

Note: When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).
What is a DOI Number?

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

  1. Encrypted AI using fully homomorphic encryption (FHE)enables inference directly over encrypted queries, provid-ing strong privacy guarantees. However, its computationaland memory overheads have limited practical deployment.Custom FHE accelerators improve performance, but rely onadvanced manufacturing technologies that limit their acces-sibility. GPUs offer a more widely available alternative, yetachieving ASIC-class performance on GPUs is challenging.Large models such as LLMs compound these challenges byrequiring optimized kernels, terabyte-scale memory man-agement, and efficient execution across multiple devices.We present Cerium, a multi-GPU framework for large-scale FHE inference. Cerium combines a domain-specificlanguage, an optimizing compiler, and a runtime system toautomatically generate optimized GPU kernels, compressplaintext data, manage terabyte-scale memory footprints,and optimize multi-GPU communication.Implemented on NVIDIA GPUs, Cerium outperforms expert-written GPU libraries by up to 2.25× and achieves perfor-mance competitive with state-of-the-art FHE ASICs, match-ing the CraterLake ASIC. It is the first GPU system to executeCKKS bootstrapping in under 10 milliseconds, achieving 7.5milliseconds, and is the first to demonstrate encrypted infer-ence for BERT-Base and Llama3-8B prefill in 8 seconds and 43seconds, respectively. 
    more » « less
    Free, publicly-accessible full text available September 28, 2027
  2. Free, publicly-accessible full text available July 7, 2027
  3. GPU power consumption has become a central systems challenge as modern AI datacenters increasingly rely on accelerators whose power footprints reach unprecedented scales. A single NVIDIA B200 already draws around 1000 W, making large multi-GPU nodes among the most power-hungry computing platforms ever deployed. Despite this, GPUs still expose only a single, device-wide DVFS domain. This coarse control is increasingly mismatched to modern AI inference, where power demand is highly non-uniform across models, phases, and workloads. LLMs alternate between compute-bound prefill and memory-bound decode phases, models exhibit widely varying kernel behavior, and emerging agentic pipelines chain together models with sharply different computational profiles. As GPUs shift toward multi-die designs and multi-model stacking becomes essential for utilization, a single global frequency increasingly leads to unnecessary and wasteful high-power operation. We introduce PowerWeave, the first spatial DVFS mechanism for GPUs, built around an OS-level power-management control plane. PowerWeave dynamically learns per-stream frequencyscaling behavior from kernel latencies and uses a global governor to react to request-rate changes, tail-latency behavior, and SLO slack. This design enables adaptive, fine-grained, kernel-aware power management that preserves SLOs while improving energy efficiency across diverse ML workloads. We build PowerWeave in Rust as a fully transparent powergovernor layer atop the GPU driver and evaluate it across diverse LLM scenarios and agentic workloads on B200 GPUs. PowerWeave reduces energy consumption by 28% on average in both disaggregated-prefill and multitenant settings, achieving up to an 8× improvement over device-wide DVFS in disaggregated prefill. For agentic workflows, PowerWeave reduces energy consumption by 19% on average without compromising throughput. PowerWeave eliminates thermal throttling across all evaluated workloads while maintaining zero SLO violations. Finally, our hardware analysis shows that spatial DVFS is practical: even perSM clock domains add less than 0.5% GPU die area overhead. 
    more » « less
    Free, publicly-accessible full text available June 1, 2027
  4. The unrelenting growth in the memory demands of datacenter applications, combined with DRAM’s volatile prices and ever-increasing costs, has made DRAM a major infrastructure expense. Alternative technologies, such as NVMe SSDs and emerging NVM devices, offer higher capacity than DRAM at a fraction of the cost and power. A promising approach is to transparently offload colder memory to cheaper memory technologies via kernel or hypervisor techniques. The key challenge, however, is to develop a datacenter-scale solution that is robust in dealing with diverse workloads and large performance variance of different offload devices, such as compressed memory, SSD, and NVM. This paper presents TMO, Meta’s transparent memory offloading solution for heterogeneous datacenter environments. TMO introduces a new Linux kernel mechanism that directly measures, in realtime, process stalls due to resource shortages across CPU, memory, and I/O. Guided by this information and without any prior application knowledge, TMO automatically adjusts how much memory to offload to heterogeneous devices (e.g., compressed memory or SSD) according to the device’s performance characteristics and the application’s sensitivity to memory-access slowdown. To maximize memory savings, TMO targets both anonymous memory and file cache, balancing the swap-in rate of anonymous memory and the reload rate of file pages that were recently evicted from the file cache. Moreover, it identifies offloading opportunities not only from the application containers but also from the sidecar containers that provide infrastructure-level functions. TMO has been in production since 2021, saving 20–32% of total memory across millions of servers in our hyperscale datacenter fleet. We have successfully upstreamed TMO into the Linux kernel. 
    more » « less
  5. The rise in memory demands of emerging datacenter applicationshas placed virtual memory translation in the spotlight, exposing itas a significant performance bottleneck. To address this problem,this paper introduces Learned Virtual Memory (LVM), a page ta-ble structure that effectively provides optimal single-access addresstranslation. LVM is founded on a novel learned index model thatdynamically adapts the address translation procedure based on thecharacteristics of an application’s virtual address space. Further-more, LVM’s learned index requires minimal memory space, doesnot impose stringent physical contiguity requirements, enjoys highcacheability in the MMU, efficiently supports insertions, and relieson simple fixed-point arithmetic. Finally, LVM supports all featuresof virtual memory, including multiple page sizes. We evaluate LVMwith a set of operating system (OS) extensions in Linux, RTL synthe-sis, and full-system simulations across a wide range of workloads.LVM reduces the address translation overhead by an average of44% over radix page tables, while reducing the page walk cachearea required by 1.5×. Overall, LVM achieves a 2-27% speedup inapplication execution time and is within 1% of an ideal page table. 
    more » « less
  6. The rapid growth of machine learning (ML) has made GPUs indispensable in datacenters and underscores the urgency of improving their efficiency. However, balancing diverse model demands with high utilization remains a fundamental chal- lenge. Transparent, fine-grained GPU resource management that maximizes utilization, energy efficiency, and isolation requires an OS approach. This paper introduces LithOS, a first step towards a GPU OS. LithOS includes the following new abstractions and mech- anisms for efficient GPU management: (i) a novel TPC Sched- uler that supports spatial scheduling at the granularity of individual TPCs, unlocking efficient TPC stealing between workloads; (ii) a transparent kernel atomizer to reduce head- of-line blocking and allow dynamic resource reallocation mid-execution; (iii) a lightweight hardware right-sizing mech- anism that dynamically determines the minimal TPC re- sources needed per atom; and (iv) a transparent power man- agement mechanism that reduces power consumption based upon in-flight work characteristics. We build LithOS in Rust and evaluate its performance across a broad set of deep learning environments, comparing it to state-of-the-art solutions from NVIDIA and prior re- search. For inference stacking, LithOS reduces tail latencies by 13×compared to MPS; compared to the best-performing SotA, it reduces tail latencies by 4×while improving ag- gregate goodput by 1.3×. Furthermore, in hybrid inference- training stacking, LithOS reduces tail latencies by 4.7×com- pared to MPS; compared to the best-performing SotA, it reduces tail latencies by 1.18×while improving aggregate throughput by 1.35×. Finally, for a modest performance hit under 4%, LithOS’s hardware right-sizing provides a quar- ter of GPU capacity savings on average, while for a 7% hit, LithOS’s transparent power management delivers a quarter of GPU total energy savings on average. Overall, LithOS transparently increases GPU efficiency, establishing a foun- dation for future OS research on GPUs. 
    more » « less