An official website of the United States government
This study presents an experimental and numerical characterization of pressure drop in a commercially available direct liquid cooled (DLC) rack. It is important to investigate the pressure drop in the DLC system as it determines the required pumping power for the DLC system, which affects the energy efficiency of the data center. The main objective of this research is to assess the flow rate and pressure distributions in a DLC system to enhance the reliability and the cooling system efficiency. Other objectives of this research are to evaluate the accuracy of flow network modeling (FNM) in predicting the flow distribution in a DLC rack and identify manufacturing limitations in a commercial system that could impact the cooling system reliability. The main components of the investigated DLC system are: coolant distribution module (CDM), supply/return manifold module, and server module which contains a cold plate. Extensive experimental measurements were performed to study the flow distribution and to determine the pressure characteristic curves for the server modules and the coolant distribution module (CDM). Also, a methodology was described to develop an experimentally validated flow network model (FNM) of the DLC system to obtain high accuracy. The measurements revealed a flow maldistribution among the server modules, which is attributed to the manufacturing process of the micro-channel cold plate. The average errors in predicting the flow rate of the server module and the CDM using FNM are 2.5% and 3.8%, respectively. The accuracy and the short run time make FNM a good tool for design, analysis, and optimization for DLC systems. The pressure drop in the server module is found to account for 56% of the total pressure drop in the DLC rack. Further analysis showed that 69% of the pressure drop in the server module is associated with the module's plumbing (corrugated hoses, disconnects, fittings). The server cooling modules are designed to provide secured connections and flexibility, which come with a high pressure drop cost.
more »
« less
Experimental and CFD Analysis of a Rack Manifold for High Power Density Liquid-Cooled Rack
Abstract Direct Liquid Cooling (DLC) has emerged as a promising technology for thermal management of high-performance computing servers, enabling efficient heat dissipation and reliable operation. Thermal performance is governed by several factors, including the coolant physical properties and flow parameters such as coolant inlet temperature and flow rate. The design and development of the coolant distribution manifold to the Information Technology Equipment (ITE) can significantly impact the overall performance of the computing system. This paper aims to investigate the hydraulic characterization and design validation of a rack-level coolant distribution manifold or rack manifold. To achieve this goal, a custom-built high power-density liquid-cooled ITE rack was assembled, and various cooling loops were plugged into the rack manifold to validate its thermal performance. The rack manifold is responsible for distributing the coolant to each of these cooling loops, which is pumped by a CDU (Coolant Distribution Unit). In this study, pressure drop characteristics of the rack manifold were obtained for flow rates that effectively dissipate the heat loads from the ITE. The pressure drop is a critical parameter in the design of the coolant distribution manifold since it influences the flow rate and ultimately the thermal performance of the system. By measuring the pressure drop at various flow rates, the researchers can accurately determine the optimum flow rate for efficient heat dissipation. Furthermore, 1D flow network and CFD models of the rack-level coolant loop, including the rack manifold, were developed, and validated against experimental test data. The validated models provide a useful tool for the design of facility-level modeling of a liquid-cooled data center. The CFD models enable the researchers to simulate the fluid flow and heat transfer within the cooling system accurately. These models can help to design the coolant distribution manifold at facility level. The results of this study demonstrate the importance of the design and development of the coolant distribution manifold in the thermal performance of a liquid-cooled data center. The study also highlights the usefulness of 1D flow network and CFD models for designing and validating liquid-cooled data center cooling systems. In conclusion, the hydraulic characterization and design validation of a rack-level coolant distribution manifold is critical in achieving efficient thermal management of high-performance computing servers. This study presents a comprehensive approach for hydraulic characterization of the coolant distribution manifold, which can significantly impact the overall thermal performance and reliability of the system. The validated models also provide a useful tool for the design of facility-level modeling of a liquid-cooled data center.
more »
« less
- Award ID(s):
- 2209751
- PAR ID:
- 10537453
- Publisher / Repository:
- American Society of Mechanical Engineers
- Date Published:
- ISBN:
- 978-0-7918-8751-6
- Format(s):
- Medium: X
- Location:
- San Diego, California, USA
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Effective cooling is crucial for high-power liquid-cooled servers to ensure optimal performance and reliability ofcomponents. Thermal characterization is necessary to ensure that the cooling system functions as intended, is energy efficient, and minimizes downtime. In this study, a proposed methodology for thermal characterization of a high-powerliquid-cooled server/TTV [server and TTVs (thermal test vehicle) are used interchangeably] is presented. The server layout includes multiple thermal test vehicle setups equipped with direct-to-chip cold plates, with two or more connected in series to form a TTV cooling loop. These cooling loops are connected in parallel to the supply and return plenums of the cooling loop manifold, which includes a chassis-level flow distribution manifold. To obtain accurate measurements, two identical server/TTV prototypes are instrumented with sensors for coolant flow rate and temperature measurements for every TTV cooling loop. Four ultrasonic flow sensors are installed in the flow verification server/TTV to measure the coolant flow rate to each TTV cooling loop. In the thermal verification server, thermistors are installed at the outlet of each GPU heater of TTV cooling loop to log temperature measurements. The amount of heat captured by the coolant in each TTV cooling loop is subsequently estimated based on the flow rates determined from the flow verification server.This methodology enables precise characterization of the thermal performance of high-power liquid-cooled servers,ensuring optimal functionality, energy efficiency, and minimized downtime.more » « less
-
Miniaturization of microelectronic components comes at a price of high heat flux density. By adopting liquid cooling, the rising demand of high heat flux devices can be met while the reliability of the microelectronic devices can also be improved to a greater extent. Liquid cooled cold plates are largely replacing air based heat sinks for electronics in data center applications, thanks to its large heat carrying capacity. A bench level study was carried out to characterize the thermohydraulic performance of two microchannel cold plates which uses warm DI water for cooling Multi Chip Server Modules (MCM). A laboratory built mock package housing mock dies and a heat spreader was employed while assessing the thermal performance of two different cold plate designs at varying coolant flow rate and temperature. The case temperature measured at the heat spreader for varying flow rates and input power were essential in identifying the convective resistance. The flow performance was evaluated by measuring the pressure drop across cold plate module at varying flow rates. Cold plate with the enhanced microchannel design yielded better results compared to a traditional parallel microchannel design. The study conducted at higher coolant temperatures yielded lower pressure drop values with no apparent change in the thermal behavior using different cold plates. The tests conducted after reversing the flow direction in microchannels provide an insight at the effect of neighboring dies on each other and reveal the importance of package specific cold plate designs for top performance. The experimental results were validated using a numerical model which are further optimized for improved geometric designs.more » « less
-
Abstract Demand is growing for the dense and high-performing IT computing capacity to support artificial intelligence, deep learning, machine learning, autonomous cars, the Internet of Things, etc. This led to an unprecedented growth in transistor density for high-end CPUs and GPUs, creating thermal design power (TDP) of even more than 700 watts for some of the NVIDIA existing GPUs. Cooling these high TDP chips with air cooling comes with a cost of the higher form factor of servers and noise produced by server fans close to the permissible limit. Direct-to-chip cold plate-based liquid cooling is highly efficient and becoming more reliable as the advancement in technology is taking place. Several components are used in the liquid-cooled data centers for the deployment of cold plate-based direct-to-chip liquid cooling like cooling loops, rack manifolds, CDUs, row manifolds, quick disconnects, flow control valves, etc. Row manifolds used in liquid cooling are used to distribute secondary coolant to the rack manifolds. Characterizing these row manifolds to understand the pressure drops and flow distribution for better data center design and energy efficiency is important. In this paper, the methodology is developed to characterize the row manifolds. Water-based coolant Propylene glycol 25% was used as the coolant for the experiments and experiments were conducted at 21 °C coolant supply temperature. Two, six-port row manifolds' P-Q curves were generated, and the value of supply pressure and the flowrate were measured at each port. The results obtained from the experiments were validated by a technique called flow network modeling (FNM). FNM technique uses the overall flow and thermal characteristics to represent the behavior of individual components.more » « less
-
Increasing power densities in data centers due to the rise of Artificial Intelligence (AI), high-performance computing (HPC) and machine learning compel engineers to develop new cooling strategies and designs for high-density data centers. Two-phase cooling is one of the promising technologies which exploits the latent heat of the fluid. This technology is much more effective in removing high heat fluxes than when using the sensible heat of fluid and requires lower coolant flow rates. The latent heat also implies more uniformity in the temperature of a heated surface. Despite the benefits of two-phase cooling, the phase change adds complexities to a system when multiple evaporators (exposed to different heat fluxes potentially) are connected to one coolant distribution unit (CDU). In this paper, a commercial pumped two-phase cooling system is investigated in a rack level. Seventeen 2-rack unit (RU) servers from two distinct models are retrofitted and deployed in the rack. The flow rate and pressure distribution across the rack are studied in various filling ratios. Also, investigated is the transient behavior of the cooling system due to a step change in the information technology (IT) load.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1115/IPACK2023-114351
