Search for: All records

Abstract Demand is growing for the dense and high-performing IT computing capacity to support artificial intelligence, deep learning, machine learning, autonomous cars, the Internet of Things, etc. This led to an unprecedented growth in transistor density for high-end CPUs and GPUs, creating thermal design power (TDP) of even more than 700 watts for some of the NVIDIA existing GPUs. Cooling these high TDP chips with air cooling comes with a cost of the higher form factor of servers and noise produced by server fans close to the permissible limit. Direct-to-chip cold plate-based liquid cooling is highly efficient and becoming more reliable as the advancement in technology is taking place. Several components are used in the liquid-cooled data centers for the deployment of cold plate-based direct-to-chip liquid cooling like cooling loops, rack manifolds, CDUs, row manifolds, quick disconnects, flow control valves, etc. Row manifolds used in liquid cooling are used to distribute secondary coolant to the rack manifolds. Characterizing these row manifolds to understand the pressure drops and flow distribution for better data center design and energy efficiency is important. In this paper, the methodology is developed to characterize the row manifolds. Water-based coolant Propylene glycol 25% was used as the coolant for the experiments and experiments were conducted at 21 °C coolant supply temperature. Two, six-port row manifolds' P-Q curves were generated, and the value of supply pressure and the flowrate were measured at each port. The results obtained from the experiments were validated by a technique called flow network modeling (FNM). FNM technique uses the overall flow and thermal characteristics to represent the behavior of individual components. 

Abstract This paper proposes a computational fluid dynamics (CFD) simulation methodology for the multi-design variable optimization of heat sinks for natural convection single-phase immersion cooling of high power-density Data Center server electronics. Immersion cooling provides the capability to cool higher power-densities than air cooling. Due to this, retrofitting Data Center servers initially designed for air-cooling for immersion cooling is of interest. A common area of improvement is in optimizing the air-cooled component heat sinks for the fluid and thermal properties of liquid cooling dielectric fluids. Current heat sink optimization methodologies for immersion cooling demonstrated within the literature rely on a server-level optimization approach. This paper proposes a server-agnostic approach to immersion cooling heat sink optimization by developing a heat sink-level CFD to generate a dataset of optimized heat sinks for a range of variable input parameters: inlet fluid temperature, power dissipation, fin thickness, and number of fins. The objective function of optimization is minimizing heat sink thermal resistance. This research demonstrates an effective modeling and optimization approach for heat sinks. The optimized heat sink designs exhibit improved cooling performance and reduced pressure drop compared to traditional heat sink designs. This study also shows the importance of considering multiple design variables in the heat sink optimization process and extends immersion heat sink optimization beyond server-dependent solutions. The proposed approach can also be extended to other cooling techniques and applications, where optimizing the design variables of heat sinks can improve cooling performance and reduce energy consumption. 

Abstract To fulfill the increasing demands of data storage and data processing within modern data centers, a corresponding increase in server performance is necessary. This leads to a subsequent increase in power consumption and heat generation in the servers due to high performance processing units. Currently, air cooling is the most widely used thermal management technique in data centers, but it has started to reach its limitations in cooling of high-power density packaging. Therefore, industries utilizing data centers are looking to singlephase immersion cooling using various dielectric fluids to reduce the operational and cooling costs by enhancing the thermal management of servers. In this study, heat sinks with TPMS lattice structures were designed for application in singlephase immersion cooling of data center servers. These designs are made possible by Electrochemical Additive Manufacturing (ECAM) technology due to their complex topologies. The ECAM process allows for generation of complex heat sink geometries never before possible using traditional manufacturing processes. Geometric complexities including amorphous and porous structures with high surface area to volume ratio enable ECAM heat sinks to have superior heat transfer properties. Our objective is to compare various heat sink geometries by minimizing chip junction temperature in a single-phase immersion cooling setup for natural convection flow regimes. Computational fluid dynamics in ANSYS Fluent is utilized to compare the ECAM heat sink designs. The additively manufactured heat sink designs are evaluated by comparing their thermal performance under natural convection conditions. This study presents a novel approach to heat sink design and bolsters the capability of ECAM-produced heat sinks. 

Abstract Data centers have complex environments that undergo constant changes due to fluctuations in IT load, commissioning and decommissioning of IT equipment, heterogeneous rack architectures and varying environmental conditions. These dynamic factors often pose challenges in effectively provisioning cooling systems, resulting in higher energy consumption. To address this issue, it is crucial to consider data center thermal heterogeneity when allocating workloads and controlling cooling, as it can impact operational efficiency. Computational Fluid Dynamics (CFD) models are used to simulate data center heterogeneity and analyze the impact of two different cooling mechanisms on operational efficiency. This research focuses on comparing the cooling based on facility water for Rear Door Heat Exchanger (RDHx) and conventional Computer Room Air Conditioning (CRAH) systems in two different data center configurations. Efficiency is measured in terms of ΔT across facility water. Higher ΔT will result in efficient operation of chillers. The actual chiller efficiency is not calculated as it would depend on local ambient conditions in which the chiller is operated. The first data center model represents a typical enterpriselevel configuration where all servers and racks have homogeneous IT power. The second model represents a colocation facility where server/rack power configurations are randomly distributed. These models predict temperature variations at different locations based on IT workload and cooling parameters. Traditionally, CRAH configurations are selected based on total IT power consumption, rack power density, and required cooling capacity for the entire data center space. On the other hand, RDHx can be scaled based on individual rack power density, offering localized cooling advantages. Multiple workload distribution scenarios were simulated for both CRAH and RDHx-based data center models. The results showed that RDHx provides a uniform thermal profile across the data center, irrespective of server/rack power density or workload distribution. This characteristic reduces the risk of over- or under-provisioning racks when using RDHx. Operational efficiency is compared in terms of difference in supply and return temperature of facility water for CRAH and RDHx units based on spatial heat dissipation and workload distribution. RDHx demonstrated excellent cooling capabilities while maintaining a higher ΔT, resulting in reduced cooling energy consumption, operational carbon footprint (?), and water usage. 

Abstract The increasing demand for high-performance computing in applications such as the Internet of Things, Deep Learning, Big data for crypto-mining, virtual reality, healthcare research on genomic sequencing, cancer treatment, etc. have led to the growth of hyperscale data centers. To meet the cooling energy demands of HPC datacenters efficient cooling technologies must be adopted. Traditional air cooling, direct-to-chip liquid cooling, and immersion are some of those methods. Among all, Liquid cooling is superior compared to various air-cooling methods in terms of energy consumption. Direct on-chip cooling using cold plate technology is one such method used in removing heat from high-power electronic components such as CPUs and GPUs in a broader sense. Over the years Thermal Design Power (TDP) is rapidly increasing and will continue to increase in the coming years for not only CPUs and GPUs but also associated electronic components like DRAMs, Platform Control Hub (PCH), and other I/O chipsets on a typical server board. Therefore, unlike air hybrid cooling which uses liquid for cold plates and air as the secondary medium of cooling the associated electronics, we foresee using immersion-based fluids to cool the rest of the electronics in the server. The broader focus of this research is to study the effects of adopting immersion cooling, with integrated cold plates for high-performance systems. Although there are several other factors involved in the study, the focus of this paper will be the optimization of cold plate microchannels for immersion-based fluids in an immersion-cooled environment. Since immersion fluids are dielectric and the fluids used in cold plates are conductive, it exposes us to a major risk of leakage into the tank and short-circuiting the electronics. Therefore, we propose using the immersed fluid to pump into the cold plate. However, it leads to a suspicion of poor thermal performance and associated pumping power due to the difference in viscosity and other fluid properties. To address the thermal and flow performance, the objective is to optimize the cold plate microchannel fin parameters based on thermal and flow performance by evaluating thermal resistance and pressure drop across the cold plate. The detailed CFD model and optimization of the cold plate were done using Ansys Icepak and Ansys OptiSLang respectively. 

Abstract Data centers are critical to the functioning of modern society as they host digital infrastructure. However, data centers can consume significant amounts of energy, and a substantial amount of this energy goes to cooling systems. Efficient thermal management of information technology equipment is therefore essential and allows the user to obtain peak performance from a system and enables higher equipment reliability. Thermal management of data center electronics is becoming more challenging due to rising power densities at the chip level. Cooling technologies like single-phase immersion cooling allow overcoming many such challenges owing to their higher thermal mass, lower fluid pumping powers, and potential component reliability enhancements. It is known that immersion cooling deployments require extremely low coolant flow rates, and, in many cases, natural convection can also be used to sufficiently dissipate the heat from the hot server components. It, therefore, becomes difficult to ascertain whether the rate of heat transfer is being dominated by forced or natural convection. This may lead to ambiguity in choosing an optimal heat sink solution and a suitable system mechanical design due to unknown flow regimes, further leading to sub-optimal system performance. Mixed convection can be used to enhance heat transfer in immersion cooling systems. The present investigation quantifies the contribution of mixed convection using numerical methods in an immersion-cooled server. An open compute server with dual CPU sockets is modeled on Ansys Icepak with varying power loads of 115W, 160W and 200W. The chosen dielectric fluid for this single-phase immersion-cooled setup is EC-100. Steady-state Computational Fluid Dynamics (CFD) simulations are conducted for forced, natural, and mixed convection heat transfer in a thermally shadowed server configuration at varying inlet flow rates. A baseline heat sink and an optimized heat sink with an increased fin thickness and reduced fin count are utilized for performance comparison. The effect of varying Reynolds number and Richardson number on the heat transfer rate from the heat sink is discussed to assess the flow regime, stability of the flow around the submerged components which depends on the geometry, orientation, fluid properties, flow rate and direction of the flow. The dimensionless numbers’ influence on heat transfer rate from a conventional air-cooled heat sink in immersion versus an immersion-optimized heat sink is also compared. The impact of server orientation on heat transfer behavior for the immersion optimized heat sink is also studied on heat transfer behavior for the immersion optimized heat sink. 

Abstract In recent years there has been a phenomenal development in cloud computing, networking, virtualization, and storage, which has increased the demand for high performance data centers. The demand for higher CPU (Central Processing Unit) performance and increasing Thermal Design Power (TDP) trends in the industry needs advanced methods of cooling systems that offer high heat transfer capabilities. Maintaining the CPU temperature within the specified limitation with air-cooled servers becomes a challenge after a certain TDP threshold. Among the equipments used in data centers, energy consumption of a cooling system is significantly large and is typically estimated to be over 40% of the total energy consumed. Advancements in Dual In-line Memory Modules (DIMMs) and the CPU compatibility led to overall higher server power consumption. Recent trends show DIMMs consume up to or above 20W each and each CPU can support up to 12 DIMM channels. Therefore, in a data center where high-power dense compute systems are packed together, it demands efficient cooling for the overall server components. In single-phase immersion cooling technology, electronic components or servers are typically submerged in a thermally conductive dielectric fluid allowing it to dissipate heat from all the electronics. The broader focus of this research is to investigate the heat transfer and flow behavior in a 1U air cooled spread core configuration server with heat sinks compared to cold plates attached in series in an immersion environment. Cold plates have extremely low thermal resistance compared to standard air cooled heatsinks. Generally, immersion fluids are dielectric, and fluids used in cold plates are electrically conductive which exposes several problems. In this study, we focus only on understanding the thermal and flow behavior, but it is important to address the challenges associated with it. The coolant used for cold plate is 25% Propylene Glycol water mixture and the fluid used in the tank is a commercially available synthetic dielectric fluid EC-100. A Computational Fluid Dynamics (CFD) model is built in such a way that only the CPUs are cooled using cold plates and the auxiliary electronic components are cooled by the immersion fluid. A baseline CFD model using an air-cooled server with heat sinks is compared to the immersion cold server with cold plates attached to the CPU. The server model has a compact model for cold plate representing thermal resistance and pressure drop. Results of the study discuss the impact on CPU temperatures for various fluid inlet conditions and predict the cooling capability of the integrated cold plate in immersion environment. 

Abstract The data center’s server power density and heat generation have increased exponentially because of the recent, unparalleled rise in the processing and storing of massive amounts of data on a regular basis. One-third of the overall energy used in conventional air-cooled data centers is directed toward cooling information technology equipment (ITE). The traditional air-cooled data centers must have low air supply temperatures and high air flow rates to support high-performance servers, rendering air cooling inefficient and compelling data center operators to use alternative cooling technology. Due to the direct interaction of dielectric fluids with all the components in the server, single-phase liquid immersion cooling (Sp-LIC) addresses mentioned problems by offering a significantly greater thermal mass and a high percentage of heat dissipation. Sp-LIC is a viable option for hyper-scale, edge, and modular data center applications because, unlike direct-to-chip liquid cooling, it does not call for a complex liquid distribution system configuration and the dielectric liquid can make direct contact with all server components. Immersion cooling is superior to conventional air-cooling technology in terms of thermal energy management however, there have been very few studies on the reliability of such cooling technology. A detailed assessment of the material compatibility of different electronic packaging materials for immersion cooling was required to comprehend their failure modes and reliability. For the mechanical design of electronics, the modulus, and thermal expansion are essential material characteristics. The substrate is a crucial element of an electronic package that has a significant impact on the reliability and failure mechanisms of electronics at both the package and the board level. As per Open Compute Project (OCP) design guidelines for immersion-cooled IT equipment, the traditional material compatibility tests from standards like ASTM 3455 can be used with certain appropriate adjustments. The primary focus of this research is to address two challenges: The first part is to understand the impact of thermal aging on the thermo-mechanical properties of the halogen-free substrate core in the single-phase immersion cooling. Another goal of the study is to comprehend how thermal aging affects the thermo-mechanical characteristics of the substrate core in the air. In this research the substrate core is aged in synthetic hydrocarbon fluid (EC100), Polyalphaolefin 6 (PAO 6), and ambient air for 720 hours each at two different temperatures: 85°C and 125°C and the complex modulus before and after aging are calculated and compared. 

Abstract Direct Liquid Cooling (DLC) has emerged as a promising technology for thermal management of high-performance computing servers, enabling efficient heat dissipation and reliable operation. Thermal performance is governed by several factors, including the coolant physical properties and flow parameters such as coolant inlet temperature and flow rate. The design and development of the coolant distribution manifold to the Information Technology Equipment (ITE) can significantly impact the overall performance of the computing system. This paper aims to investigate the hydraulic characterization and design validation of a rack-level coolant distribution manifold or rack manifold. To achieve this goal, a custom-built high power-density liquid-cooled ITE rack was assembled, and various cooling loops were plugged into the rack manifold to validate its thermal performance. The rack manifold is responsible for distributing the coolant to each of these cooling loops, which is pumped by a CDU (Coolant Distribution Unit). In this study, pressure drop characteristics of the rack manifold were obtained for flow rates that effectively dissipate the heat loads from the ITE. The pressure drop is a critical parameter in the design of the coolant distribution manifold since it influences the flow rate and ultimately the thermal performance of the system. By measuring the pressure drop at various flow rates, the researchers can accurately determine the optimum flow rate for efficient heat dissipation. Furthermore, 1D flow network and CFD models of the rack-level coolant loop, including the rack manifold, were developed, and validated against experimental test data. The validated models provide a useful tool for the design of facility-level modeling of a liquid-cooled data center. The CFD models enable the researchers to simulate the fluid flow and heat transfer within the cooling system accurately. These models can help to design the coolant distribution manifold at facility level. The results of this study demonstrate the importance of the design and development of the coolant distribution manifold in the thermal performance of a liquid-cooled data center. The study also highlights the usefulness of 1D flow network and CFD models for designing and validating liquid-cooled data center cooling systems. In conclusion, the hydraulic characterization and design validation of a rack-level coolant distribution manifold is critical in achieving efficient thermal management of high-performance computing servers. This study presents a comprehensive approach for hydraulic characterization of the coolant distribution manifold, which can significantly impact the overall thermal performance and reliability of the system. The validated models also provide a useful tool for the design of facility-level modeling of a liquid-cooled data center. 

Abstract Data centers have started to adopt immersion cooling for more than just mainframes and supercomputers. Due to the inability of air cooling to cool down recent high-configured servers with higher Thermal Design Power, current thermal requirements in machine learning, AI, blockchain, 5G, edge computing, and high-frequency trading have resulted in a larger deployment of immersion cooling. Dielectric fluids are far more efficient at transferring heat than air. Immersion cooling promises to help address many of the challenges that come with air cooling systems, especially as computing densities increase. Immersion-cooled data centers are more expandable, quicker installation, more energy-efficient, allows for the cooling of almost all server components, save more money for enterprises, and are more robust overall. By eliminating active cooling components such as fans, immersion cooling enables a significantly higher density of computing capabilities. When utilizing immersion cooling for server hardware that is intended to be air-cooled, immersion-specific optimized heat sinks should be used. A heat sink is an important component for server cooling efficacy. This research conducts an optimization of heatsink for immersion-cooled servers to achieve the minimum case temperature possible utilizing multi-objective and multidesign variable optimization with pumping power as the constraint. A high-density server of 3.76 kW was modeled on Ansys Icepak that consists of 2 CPUs and 8 GPUs with heatsink assemblies at their Thermal Design Power along with 32 Dual In-line Memory Modules. The optimization is conducted for Aluminum heat sinks by minimizing the pressure drop and thermal resistance as the objective functions whereas fin count, fin thickness, and heat sink height are chosen as the design variables in all CPUs, and GPUs heatsink assemblies. Optimization for the CPU and the GPU heatsink was done separately and then the optimized heatsinks were tested in an actual test setup of the server in ANSYS Icepak. The dielectric fluid for this numerical study is EC-110 and the cooling is carried out using forced convection. A Design of Experiment (DOE) is created based on the input range of design variables using a full-factorial approach to generate multiple design points. The effect of the design variables is analyzed on the objective functions to establish the parameters that have a greater impact on the performance of the optimized heatsink. The optimization study is done using Ansys OptiSLang where AMOP (Adaptive Metamodel of Optimal Prognosis) as the sampling method for design exploration. The results show total effect values of heat sinks geometric parameters to choose the best design point with the help of a Response Surface 2D and 3D plot for the individual heat sink assembly.