More Like this

1. CodedBulk: Inter-Datacenter Bulk Transfers using Network Coding

   Tseng, Shih-Hao; Agarwal, Saksham; Agarwal, Rachit; Ballani, Hitesh; Tang, Ao (April 2021, USENIX Symposium on Networked Systems Design and Implementation)

   null (Ed.)

   This paper presents CodedBulk, a system for high-throughput inter-datacenter bulk transfers. At its core, CodedBulk uses network coding, a technique from the coding theory community, that guarantees optimal throughput for individual bulk transfers. Prior attempts to using network coding in wired networks have faced several pragmatic and fundamental barriers. CodedBulk resolves these barriers by exploiting the unique properties of inter-datacenter networks, and by using a custom-designed hop-by-hop flow control mechanism that enables efficient realization of network coding atop existing transport protocols. An end-to end CodedBulk implementation running on a geo-distributed inter-datacenter network improves bulk transfer throughput by 1.2−2.5× compared to state-of-the-art mechanisms that do not use network coding. 

   more » « less
2. Toward Carbon-Aware Data Transfers

   https://doi.org/10.1109/MIC.2025.3565187  

   Goldverg, Jacob; Jamil, Hasibul; Rodrigues, Elvis; Kosar, Tevfik (March 2025, IEEE Internet Computing)

   The growing adoption of cloud, edge, and distributed computing, as well as the rise in the use of AI/ML workloads, have created a significant need to measure, monitor, and reduce the carbon emissions associated with these resource-intensive tasks. One significant but often overlooked source of emissions is data transfers over wide-area networks (WANs), primarily due to the challenges in accurately measuring the carbon footprint of end-to-end network paths. We introduce a novel mechanism to measure network carbon footprints and propose strategies for optimizing the scheduling of network-intensive tasks. We show that users can achieve significant carbon savings by shifting data transfer tasks across time and geographic regions based on local carbon intensity. 

   more » « less
3. Adapting Datacenter Capacity for Greener Datacenters and Grid

   https://doi.org/10.1145/3575813.3595197  

   Lin, Liuzixuan; Chien, Andrew A (June 2023, ACM Symposium on Future Energy Systems (E-Energy 2023))

   Cloud providers are adapting datacenter (DC) capacity to reduce carbon emissions. With hyperscale datacenters exceeding 100 MW individually, and in some grids exceeding 15% of power load, DC adaptation is large enough to harm power grid dynamics, increasing carbon emissions, power prices, or reduce grid reliability. To avoid harm, we explore coordination of DC capacity change varying scope in space and time. In space, coordination scope spans a single datacenter, a group of datacenters, and datacenters with the grid. In time, scope ranges from online to day-ahead. We also consider what DC and grid information is used (e.g. real-time and day-ahead average carbon, power price, and compute backlog). For example, in our proposed PlanShare scheme, each datacenter uses day-ahead information to create a capacity plan and shares it, allowing global grid optimization (over all loads, over entire day). We evaluate DC carbon emissions reduction. Results show that local coordination scope fails to reduce carbon emissions significantly (3.2%–5.4% reduction). Expanding coordination scope to a set of datacenters improves slightly (4.9%–7.3%). PlanShare, with grid-wide coordination and full-day capacity planning, performs the best. PlanShare reduces DC emissions by 11.6%–12.6%, 1.56x–1.26x better than the best local, online approach’s results. PlanShare also achieves lower cost. We expect these advantages to increase as renewable generation in power grids increases. Further, a known full-day DC capacity plan provides a stable target for DC resource management. 

   more » « less
4. OneDataShare - A Vision for Cloud-hosted Data Transfer Scheduling and Optimization as a Service [OneDataShare - A Vision for Cloud-hosted Data Transfer Scheduling and Optimization as a Service]

   https://doi.org/10.5220/0006793506160625  

   Imran, Asif; Nine, Md S.; Guner, Kemal; Kosar, Tevfik (January 2018, Proceedings of the 8th International Conference on Cloud Computing and Services Science)

   Fast, reliable, and efficient data transfer across wide-area networks is a predominant bottleneck for dataintensive cloud applications. This paper introduces OneDataShare, which is designed to eliminate the issues plaguing effective cloud-based data transfers of varying file sizes and across incompatible transfer end-points. The vision of OneDataShare is to achieve high-speed data transfer, interoperability between multiple transfer protocols, and accurate estimation of delivery time for advance planning, thereby maximizing user-profit through improved and faster data analysis for business intelligence. The paper elaborates on the desirable features of OneDataShare as a cloud-hosted data transfer scheduling and optimization service, and how it is aligned with the vision of harnessing the power of the cloud and distributed computing. Experimental evaluation and comparison with existing real-life file transfer services show that the transfer throughout achieved by OneDataShare is up to 6.5 times greater compared to other approaches. 

   more » « less
5. WattScope: Non-intrusive application-level power disaggregation in datacenters

   https://doi.org/10.1016/j.peva.2023.102369  

   Guan, Xiaoding; Bashir, Noman; Irwin, David; Shenoy, Prashant (November 2023, Performance Evaluation)

   Datacenter capacity is growing exponentially to satisfy the increasing demand for many emerging computationally-intensive applications, such as deep learning. This trend has led to concerns over datacenters’ increasing energy consumption and carbon footprint. The most basic prerequisite for optimizing a datacenter’s energy- and carbon-efficiency is accurately monitoring and attributing energy consumption to specific users and applications. Since datacenter servers tend to be multi-tenant, i.e., they host many applications, server- and rack-level power monitoring alone does not provide insight into the energy usage and carbon emissions of their resident applications. At the same time, current application-level energy monitoring and attribution techniques are intrusive: they require privileged access to servers and necessitate coordinated support in hardware and software, neither of which is always possible in cloud environments. To address the problem, we design WattScope, a system for non-intrusively estimating the power consumption of individual applications using external measurements of a server’s aggregate power usage and without requiring direct access to the server’s operating system or applications. Our key insight is that, based on an analysis of production traces, the power characteristics of datacenter workloads, e.g., low variability, low magnitude, and high periodicity, are highly amenable to disaggregation of a server’s total power consumption into application-specific values. WattScope adapts and extends a machine learning-based technique for disaggregating building power and applies it to server- and rack-level power meter measurements that are already available in data centers. We evaluate WattScope’s accuracy on a production workload and show that it yields high accuracy, e.g., often 10% normalized mean absolute error, and is thus a potentially useful tool for datacenters in externally monitoring application-level power usage. 

   more » « less