An official website of the United States government
Abstract Direct Liquid Cooling (DLC) has emerged as a promising technology for thermal management of high-performance computing servers, enabling efficient heat dissipation and reliable operation. Thermal performance is governed by several factors, including the coolant physical properties and flow parameters such as coolant inlet temperature and flow rate. The design and development of the coolant distribution manifold to the Information Technology Equipment (ITE) can significantly impact the overall performance of the computing system. This paper aims to investigate the hydraulic characterization and design validation of a rack-level coolant distribution manifold or rack manifold. To achieve this goal, a custom-built high power-density liquid-cooled ITE rack was assembled, and various cooling loops were plugged into the rack manifold to validate its thermal performance. The rack manifold is responsible for distributing the coolant to each of these cooling loops, which is pumped by a CDU (Coolant Distribution Unit). In this study, pressure drop characteristics of the rack manifold were obtained for flow rates that effectively dissipate the heat loads from the ITE. The pressure drop is a critical parameter in the design of the coolant distribution manifold since it influences the flow rate and ultimately the thermal performance of the system. By measuring the pressure drop at various flow rates, the researchers can accurately determine the optimum flow rate for efficient heat dissipation. Furthermore, 1D flow network and CFD models of the rack-level coolant loop, including the rack manifold, were developed, and validated against experimental test data. The validated models provide a useful tool for the design of facility-level modeling of a liquid-cooled data center. The CFD models enable the researchers to simulate the fluid flow and heat transfer within the cooling system accurately. These models can help to design the coolant distribution manifold at facility level. The results of this study demonstrate the importance of the design and development of the coolant distribution manifold in the thermal performance of a liquid-cooled data center. The study also highlights the usefulness of 1D flow network and CFD models for designing and validating liquid-cooled data center cooling systems. In conclusion, the hydraulic characterization and design validation of a rack-level coolant distribution manifold is critical in achieving efficient thermal management of high-performance computing servers. This study presents a comprehensive approach for hydraulic characterization of the coolant distribution manifold, which can significantly impact the overall thermal performance and reliability of the system. The validated models also provide a useful tool for the design of facility-level modeling of a liquid-cooled data center.
more »
« less
AN EXPERIMENTAL INVESTIGATION ON THE FLUID DISTRIBUTION IN A TWO-PHASE COOLED RACK UNDER STEADY AND TRANSIENT IT LOAD
Increasing power densities in data centers due to the rise of Artificial Intelligence (AI), high-performance computing (HPC) and machine learning compel engineers to develop new cooling strategies and designs for high-density data centers. Two-phase cooling is one of the promising technologies which exploits the latent heat of the fluid. This technology is much more effective in removing high heat fluxes than when using the sensible heat of fluid and requires lower coolant flow rates. The latent heat also implies more uniformity in the temperature of a heated surface. Despite the benefits of two-phase cooling, the phase change adds complexities to a system when multiple evaporators (exposed to different heat fluxes potentially) are connected to one coolant distribution unit (CDU). In this paper, a commercial pumped two-phase cooling system is investigated in a rack level. Seventeen 2-rack unit (RU) servers from two distinct models are retrofitted and deployed in the rack. The flow rate and pressure distribution across the rack are studied in various filling ratios. Also, investigated is the transient behavior of the cooling system due to a step change in the information technology (IT) load.
more »
« less
- Award ID(s):
- 1738793
- PAR ID:
- 10157106
- Date Published:
- Journal Name:
- ASME 2019 International Electronic Packaging Technical Conference and Exhibition on Packaging and Integration of Electronic and Photonic Microsystems
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
As the demand for faster and more reliable data processing is increasing in our daily lives, the power consumption of electronics and, correspondingly, Data Centers (DCs), also increases. It has been estimated that about 40% of this DCs power consumption is merely consumed by the cooling systems. A responsive and efficient cooling system would not only save energy and space but would also protect electronic devices and help enhance their performance. Although air cooling offers a simple and convenient solution for Electronic Thermal Management (ETM), it lacks the capacity to overcome higher heat flux rates. Liquid cooling techniques, on the other hand, have gained high attention due to their potential in overcoming higher thermal loads generated by small chip sizes. In the present work, one of the most commonly used liquid cooling techniques is investigated based on various conditions. The performance of liquid-to-liquid heat exchange is studied under multi-leveled thermal loads. Coolant Supply Temperature (CST) stability and case temperature uniformity on the Thermal Test Vehicles (TTVs) are the target indicators of the system performance in this study. This study was carried out experimentally using a rack-mount Coolant Distribution Unit (CDU) attached to primary and secondary cooling loops in a multi-server rack. The effect of various selected control settings on the aforementioned indicators is presented. Results show that the most impactful PID parameter when it comes to fluctuation reduction is the integral (reset) coefficient (IC). It is also concluded that fluctuation with amplitudes lower than 1 ᵒC is converged into higher amplitudesmore » « less
-
In response to the exponential growth of online platforms and the rise of web-based Artificial Intelligence (AI), the demand for computational power and the expansion of data centers have surged significantly. This trend necessitates advanced cooling strategies and heightened energy efficiency to address the increasing power densities of Information Technology (IT) equipment and the consequent rise in energy consumption. Consequently, there is a significant pivot towards efficient cooling mechanisms that emphasize thermal management and energy efficiency. Against this backdrop, our study thoroughly evaluates a two-phase direct-to-chip liquid cooling system's ability to effectively manage and dissipate heat in high-density rack environments. Central to our research is the deployment of a highly efficient Refrigerant-to-Liquid (R2L) Coolant Distribution Unit (CDU) across multi-racks, which face high thermal demands. This innovative system, featuring an in-row pumped two-phase CDU with a cooling capacity of 160 kW, is intricately integrated with row and rack manifolds and server cooling loops to ensure optimal cooling performance. To accurately simulate the thermal loads encountered in real-world data center operations, the study employs Thermal Testing Vehicles (TTVs). These 3U TTVs are equipped with 2.5 kW heaters, covering an extensive area of 2500 mm², thereby effectively replicating server thermal loads up to 10 kW. The investigation starts with a detailed description of the system's design and continues with the commissioning process. This process includes extensive hydraulic and thermal testing, along with a comprehensive assessment of the impact of pressure drops across the system, focusing on supply manifolds, cooling loops, dry breaks, and return manifolds, utilizing Cooling Loops (CLs) each containing four Cold Plates (CPs). The study culminates in the analysis of experimental data from heating the TTVs, focusing on the efficiency of two-phase cooling in transferring heat from the TTVs to chilled water using R134a refrigerant as the performance benchmark. Future directions include exploring eco-friendly cooling practices by investigating alternative green refrigerants with low Global Warming Potential (GWP) to replace R134a, aligning with global sustainability goals and the imperative to reduce greenhouse gas emissions. The observed maximum values were calculated at a specific volumetric flow rate of 0.48 LPM/kW and a Tcase as low as 56.4 °C was achieved. These results demonstrate the system's capability to significantly enhance thermal management in data centers, tackle the challenges presented by high-power density chips, and encourage broader adoption of two-phase cooling technologies as a sustainable strategy for thermal regulation in the face of increasing computational demands.more » « less
-
Abstract Demand is growing for the dense and high-performing IT computing capacity to support artificial intelligence, deep learning, machine learning, autonomous cars, the Internet of Things, etc. This led to an unprecedented growth in transistor density for high-end CPUs and GPUs, creating thermal design power (TDP) of even more than 700 watts for some of the NVIDIA existing GPUs. Cooling these high TDP chips with air cooling comes with a cost of the higher form factor of servers and noise produced by server fans close to the permissible limit. Direct-to-chip cold plate-based liquid cooling is highly efficient and becoming more reliable as the advancement in technology is taking place. Several components are used in the liquid-cooled data centers for the deployment of cold plate-based direct-to-chip liquid cooling like cooling loops, rack manifolds, CDUs, row manifolds, quick disconnects, flow control valves, etc. Row manifolds used in liquid cooling are used to distribute secondary coolant to the rack manifolds. Characterizing these row manifolds to understand the pressure drops and flow distribution for better data center design and energy efficiency is important. In this paper, the methodology is developed to characterize the row manifolds. Water-based coolant Propylene glycol 25% was used as the coolant for the experiments and experiments were conducted at 21 °C coolant supply temperature. Two, six-port row manifolds' P-Q curves were generated, and the value of supply pressure and the flowrate were measured at each port. The results obtained from the experiments were validated by a technique called flow network modeling (FNM). FNM technique uses the overall flow and thermal characteristics to represent the behavior of individual components.more » « less
-
Abstract Data centers are witnessing an unprecedented increase in processing and data storage, resulting in an exponential increase in the servers’ power density and heat generation. Data center operators are looking for green energy efficient cooling technologies with low power consumption and high thermal performance. Typical air-cooled data centers must maintain safe operating temperatures to accommodate cooling for high power consuming server components such as CPUs and GPUs. Thus, making air-cooling inefficient with regards to heat transfer and energy consumption for applications such as high-performance computing, AI, cryptocurrency, and cloud computing, thereby forcing the data centers to switch to liquid cooling. Additionally, air-cooling has a higher OPEX to account for higher server fan power. Liquid Immersion Cooling (LIC) is an affordable and sustainable cooling technology that addresses many of the challenges that come with air cooling technology. LIC is becoming a viable and reliable cooling technology for many high-power demanding applications, leading to reduced maintenance costs, lower water utilization, and lower power consumption. In terms of environmental effect, single-phase immersion cooling outperforms two-phase immersion cooling. There are two types of single-phase immersion cooling methods namely, forced and natural convection. Here, forced convection has a higher overall heat transfer coefficient which makes it advantageous for cooling high-powered electronic devices. Obviously, with natural convection, it is possible to simplify cooling components including elimination of pump. There is, however, some advantages to forced convection and especially low velocity flow where the pumping power is relatively negligible. This study provides a comparison between a baseline forced convection single phase immersion cooled server run for three different inlet temperatures and four different natural convection configurations that utilize different server powers and cold plates. Since the buoyancy effect of the hot fluid is leveraged to generate a natural flow in natural convection, cold plates are designed to remove heat from the server. For performance comparison, a natural convection model with cold plates is designed where water is the flowing fluid in the cold plate. A high-density server is modeled on the Ansys Icepak, with a total server heat load of 3.76 kW. The server is made up of two CPUs and eight GPUs with each chip having its own thermal design power (TDPs). For both heat transfer conditions, the fluid used in the investigation is EC-110, and it is operated at input temperatures of 30°C, 40°C, and 50°C. The coolant flow rate in forced convection is 5 GPM, whereas the flow rate in natural convection cold plates is varied. CFD simulations are used to reduce chip case temperatures through the utilization of both forced and natural convection. Pressure drop and pumping power of operation are also evaluated on the server for the given intake temperature range, and the best-operating parameters are established. The numerical study shows that forced convection systems can maintain much lower component temperatures in comparison to natural convection systems even when the natural convection systems are modeled with enhanced cooling characteristics.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
