An official website of the United States government
In typical data centers, the servers and IT equipment are cooled by air and almost half of total IT power is dedicated to cooling. Hybrid cooling is a combined cooling technology with both air and water, where the main heat generating components are cooled by water or water-based coolants and rest of the components are cooled by air supplied by CRAC or CRAH. Retrofitting the air-cooled servers with cold plates and pumps has the advantage over thermal management of CPUs and other high heat generating components. In a typical 1U server, the CPUs were retrofitted with cold plates and the server tested with raised coolant inlet conditions. The study showed the server can operate with maximum utilization for CPUs, DIMMs, and PCH for inlet coolant temperature from 25–45 °C following the ASHRAE guidelines. The server was also tested for failure scenarios of the pumps and fans with reducing numbers of fans and pumps. To reduce cooling power consumption at the facility level and increase air-side economizer hours, the hybrid cooled server can be operated at raised inlet air temperatures. The trade-off in energy savings at the facility level due to raising the inlet air temperatures versus the possible increase in server fan power and component temperatures is investigated. A detailed CFD analysis with a minimum number of server fans can provide a way to find an operating range of inlet air temperature for a hybrid cooled server. Changes in the model are carried out in 6SigmaET for an individual server and compared to the experimental data to validate the model. The results from this study can be helpful in determining the room level operating set points for data centers housing hybrid cooled server racks.
more »
« less
Energy Analysis of Rear Door Heat Exchangers in Data Centers With Spatial Workload Distribution
Abstract Data centers have complex environments that undergo constant changes due to fluctuations in IT load, commissioning and decommissioning of IT equipment, heterogeneous rack architectures and varying environmental conditions. These dynamic factors often pose challenges in effectively provisioning cooling systems, resulting in higher energy consumption. To address this issue, it is crucial to consider data center thermal heterogeneity when allocating workloads and controlling cooling, as it can impact operational efficiency. Computational Fluid Dynamics (CFD) models are used to simulate data center heterogeneity and analyze the impact of two different cooling mechanisms on operational efficiency. This research focuses on comparing the cooling based on facility water for Rear Door Heat Exchanger (RDHx) and conventional Computer Room Air Conditioning (CRAH) systems in two different data center configurations. Efficiency is measured in terms of ΔT across facility water. Higher ΔT will result in efficient operation of chillers. The actual chiller efficiency is not calculated as it would depend on local ambient conditions in which the chiller is operated. The first data center model represents a typical enterpriselevel configuration where all servers and racks have homogeneous IT power. The second model represents a colocation facility where server/rack power configurations are randomly distributed. These models predict temperature variations at different locations based on IT workload and cooling parameters. Traditionally, CRAH configurations are selected based on total IT power consumption, rack power density, and required cooling capacity for the entire data center space. On the other hand, RDHx can be scaled based on individual rack power density, offering localized cooling advantages. Multiple workload distribution scenarios were simulated for both CRAH and RDHx-based data center models. The results showed that RDHx provides a uniform thermal profile across the data center, irrespective of server/rack power density or workload distribution. This characteristic reduces the risk of over- or under-provisioning racks when using RDHx. Operational efficiency is compared in terms of difference in supply and return temperature of facility water for CRAH and RDHx units based on spatial heat dissipation and workload distribution. RDHx demonstrated excellent cooling capabilities while maintaining a higher ΔT, resulting in reduced cooling energy consumption, operational carbon footprint (?), and water usage.
more »
« less
- Award ID(s):
- 2209751
- PAR ID:
- 10536111
- Publisher / Repository:
- American Society of Mechanical Engineers
- Date Published:
- ISBN:
- 978-0-7918-8751-6
- Format(s):
- Medium: X
- Location:
- San Diego, California, USA
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Effective heat removal has become a critical issue in today’s data centers. With the rapid expansion of computational power and the growing demands of artificial intelligence and machine learning, key components such as GPUs, CPUs, and network switches are pushed to their performance limits. This surge in workload causes IT servers to generate extremely high-power densities that can overwhelm traditional air-cooling systems. As a result, data centers are increasingly adopting innovative thermal management strategies. One promising solution is direct-to-chip liquid cooling, which employs cold plates mounted directly on critical components to efficiently dissipate heat. This method not only addresses the immediate problem of overheating but also allows data centers to operate at higher performance levels while reducing energy consumption and operational costs. In this work, we study the thermohydraulic performance of liquid-to-liquid cooling systems that use Cooling Distribution Units (CDU). In these systems, CDU delivers a cold secondary coolant (25% propylene glycol) to the server rack cooling loops, allowing for effective liquid-to-liquid heat exchange between the primary facility water and the coolant in the cold plates. To simulate high heat density servers, a set of thermal emulators was developed to replicate the extreme thermal loads experienced in modern computational platforms. Each emulator was equipped with heaters of two different footprints (A and B), resulting in varying heat fluxes and providing diverse thermal profiles that mimic real operating conditions. The emulators were evaluated under both natural and forced convection conditions, ensuring a comprehensive understanding of performance under different cooling scenarios. This study aims to experimentally characterize three different cooling loops with parallel and series configurations and conduct detailed analyses to evaluate the cooling characteristics of the complete liquid-to-liquid system, including the CDU. A systematic approach will be developed to identify and analyze discrepancies between surface temperatures and variations among thermocouple readings. Finally, a comprehensive comparative analysis of the cooling loops in terms of thermal, hydraulic performance and overall heat dissipation efficiency will be introduced, providing valuable insights into optimizing liquid-to-liquid cooling strategies for high heat density data centers. Additionally, further analysis assessed the impact of coolant flow variations and transient thermal events on system performance. These evaluations deepened our understanding of the dynamic interactions within the cooling loops and enhanced our experimental insights, ensuring that the developed strategies not only boost efficiency but also improve reliability in high-demand data center environments.more » « less
-
Air-cooled data centers continue to dominate the data center landscape. Despite advances made in recent years, current air cooling solutions are limited in their ability to adapt to configuration changes and scalability to different installation sizes. Further, these solutions are unable to handle the rapidly changing demands in cooling and inevitable temporal workload variations. Relatively longer control loop delays are also introduced by the longer response times that result from reactive control. Both of these limitations related to demand dynamics are addressed by over-provisioning cooling, leading to poor energy efficiency and operational sustainability. Our vision is to develop a predictive air cooling system that has none of these limitations. Towards this, we present a novel Dynamic Cooling System Control (DCSC) mechanism, which utilizes machine learning-enabled models that are needed for fast cooling system response time for a predictive controller. These hierarchical models predict the thermodynamic conditions and the degree of cooling needed, responsive to the predicted workload from utilization trends. The machine learning-based thermal model leverages experimental data collected from the Data Center at the State University of New York at Binghamton to estimate CPU temperatures based on varying CPU utilization levels and cooling conditions. It examines the temperature dynamics within the aisles, which are influenced by server workloads and the cooling control setting from the Computer Room Air Handling (CRAH) units, by simulating different cases. The model’s predictive performance is evaluated using various test datasets, demonstrating its capability for fast, real-time temperature predictions. This functionality serves as the foundation for developing an accurate and responsive cooling system controller aimed at reducing the overprovisioning of cooling resources. To enhance predictive accuracy and real-time adaptability, a digital twin of the data center can be developed, allowing for continuous simulation and optimization of cooling strategies based on real-time operational data. Future work will focus on refining the predictive models and implementing localized cooling strategies to further enhance the adaptability and scalability of the cooling system. This research contributes to the ongoing efforts to develop sustainable and efficient data center operations, ultimately supporting the growing demands of the digital economy while minimizing environmental impact.more » « less
-
In response to the exponential growth of online platforms and the rise of web-based Artificial Intelligence (AI), the demand for computational power and the expansion of data centers have surged significantly. This trend necessitates advanced cooling strategies and heightened energy efficiency to address the increasing power densities of Information Technology (IT) equipment and the consequent rise in energy consumption. Consequently, there is a significant pivot towards efficient cooling mechanisms that emphasize thermal management and energy efficiency. Against this backdrop, our study thoroughly evaluates a two-phase direct-to-chip liquid cooling system's ability to effectively manage and dissipate heat in high-density rack environments. Central to our research is the deployment of a highly efficient Refrigerant-to-Liquid (R2L) Coolant Distribution Unit (CDU) across multi-racks, which face high thermal demands. This innovative system, featuring an in-row pumped two-phase CDU with a cooling capacity of 160 kW, is intricately integrated with row and rack manifolds and server cooling loops to ensure optimal cooling performance. To accurately simulate the thermal loads encountered in real-world data center operations, the study employs Thermal Testing Vehicles (TTVs). These 3U TTVs are equipped with 2.5 kW heaters, covering an extensive area of 2500 mm², thereby effectively replicating server thermal loads up to 10 kW. The investigation starts with a detailed description of the system's design and continues with the commissioning process. This process includes extensive hydraulic and thermal testing, along with a comprehensive assessment of the impact of pressure drops across the system, focusing on supply manifolds, cooling loops, dry breaks, and return manifolds, utilizing Cooling Loops (CLs) each containing four Cold Plates (CPs). The study culminates in the analysis of experimental data from heating the TTVs, focusing on the efficiency of two-phase cooling in transferring heat from the TTVs to chilled water using R134a refrigerant as the performance benchmark. Future directions include exploring eco-friendly cooling practices by investigating alternative green refrigerants with low Global Warming Potential (GWP) to replace R134a, aligning with global sustainability goals and the imperative to reduce greenhouse gas emissions. The observed maximum values were calculated at a specific volumetric flow rate of 0.48 LPM/kW and a Tcase as low as 56.4 °C was achieved. These results demonstrate the system's capability to significantly enhance thermal management in data centers, tackle the challenges presented by high-power density chips, and encourage broader adoption of two-phase cooling technologies as a sustainable strategy for thermal regulation in the face of increasing computational demands.more » « less
-
About 40% of the energy utilized in data centers is used for cooling systems, and this percentage has increased significantly in recent years. Data center server racks receive indirect cooling from computer room air conditioning units (CRACs) or computer room air handler units (CRAHs), and chilled air is sent to the racks through a raised floor plenum to cool the server room. This approach is inefficient because the server room has excessive cooling while the IT equipment has inadequate cooling. The data center industry has begun to use thermally and energy-efficient single-phase liquid cooling solutions as a result of the tremendous increase in IT power density and energy usage. One of the most popular liquid cooling systems will be examined in the current study, which is based on numerous circumstances. Under various secondary coolant conditions, the hydro-thermal performance of different liquid-to-air and liquid-to-liquid coolant distribution units (CDUs) will be assessed experimentally using multi-racks loaded with different numbers of thermal testing vehicles (TTVs), and the system response to any change in the flow will be examined by disconnecting and reconnecting the TTVs cooling loops in the multi-racks through different sequences of transient events. Moreover, a set of rules and guidelines will be established to commission these units.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Conference Paper:
- https://doi.org/10.1115/IPACK2023-112078
