INTRODUCTION

Thermal management of data center electronics has become increasingly difficult over the last decade owing to an increase in the utilization of data-intensive technologies such as machine learning, the Internet of things, and cryptocurrency mining. Traditional data center cooling methods that utilize air for heat dissipation of server platforms consume significantly large energy and also water [1]. Liquid-based cooling technologies offer higher heat transfer coefficients and are being increasingly used to dissipate high power densities of the new generation central processing units (CPUs) and graphics processing (GPUs) [2]. Some of the popular liquid cooling technologies include cold plate-based single and two-phase cooling and direct contact single and two-phase immersion cooling. Additional benefits in these cooling technologies have also been proposed using optimized geometrical shape cold plates [3], and the addition of nano-particles to base fluids to improve thermal properties of the base fluids specific to server platform cooling [4,5]. Cooling optimization has also been proposed by using real-time coolant transport modulation techniques either by flow control devices [6,7], chassis modifications [8], or using machine learning algorithms [9] at the server and data center level.

Advanced liquid cooling techniques such as single-phase immersion cooling allow data centers to not only dissipate high power densities but also enable high energy efficiency by offering very low PUE values [10,30]. The primary advantage of single-phase immersion (SPI) cooling lies in the fact that it allows complete removal of fan-based cooling from the servers and any of the synthetic dielectric fluids used are biodegradable with zero global warming potential (GWP) [11,31]. Also, in recent times significant attention has been given to addressing the material compatibility and reliability issues in this cooling technique that had been a major bottleneck in its widespread adoption [12][13][14]. With the reliability concerns being addressed, the data center industry is now seeing an increase in SPI cooling deployments, especially due to its ease of deployment even in harsh environments where air and hybrid cooling systems cannot be used [12,26].

Coolant flow rates in typical SPI cooled servers usually fall under the laminar flow regime [10,15]. This means if the server is designed with an air-cooled heat sink, it needs to be optimized for lower flow rates and changes in convective heat transfer properties. It also means that under such low Reynolds number flow, it should be carefully analyzed whether the dominant heat transfer mode is forced or natural convection or a combination of both these modes known as mixed convection. Mixed convection is commonly found in cooling systems where the coolant flow rates or flow velocities are very low. In such flows, buoyancy forces have a significant impact on flow behavior and heat transfer rate [16]. Mixed convection is characterized by a dimensionless number known as the Richardson number (Ri) defined by the ratio Gr/Re 2 which is a quantitative indicator of the balance between natural and forced convection in a flow. A value of Ri<1 depicts forced convection-dominated flow and a value of Ri>1 depicts natural convection-dominated flow.

An in-depth review of various convective heat transfer cooling options for electronic packages has been carried out by Incropera [17]. Papanicolaou and Jaluria [18] studied mixed convection in a horizontally oriented square enclosure for a single heat source. They varied the location of the outlet and size of the isothermal heat source assuming laminar 2-D flow with a Re range between 50-2000. It was observed that for Ri values between 0-10, the flow remained in the laminar regime and became oscillatory at higher values. Acharya and Ptankar [19] investigated the effect of buoyancy on laminar mixed convection for a fin-array placed in an enclosure using analytical methods. They observed that heat transfer is significantly affected by the buoyancy forces in laminar mixed convection for such an arrangement of a finned array. Also, an enhancement in heat transfer rate is obtained due to buoyancy-driven secondary flows. Maughan and Incropera [20] experimentally investigated the effects of laminar mixed convection airflow on longitudinal fins between parallel plates where the bottom plate is isothermally heated and the top plate is isothermally cooled. A considerable enhancement in heat transfer was observed for low values of the Rayleigh number (Ra). It was also observed that closely spaced fins had a higher heat transfer rate due to enhanced surface area but this delayed the formation of secondary flows that also enhance heat transfer. Dogan and Sivrioglu [21,22] studied the influence of fin spacing, fin height, and heat flux on laminar mixed convection for longitudinal fins in a rectangular channel. Conclusions were made on optimum fin spacing that yields maximum heat transfer and it was concluded that optimum fin spacing depends on the value of modified Ra.

While there are a lot of studies that look into the impact of forced, natural, or mixed convection separately for finned-arrays in angular ducts and enclosures, there is no discussion about the influence of mixed convection in servers using SPI cooling. Characterization of mixed convection is significant for SPI as a server typically contains multiple heat-dissipating components that may also have an impact on convective heat transfer from the heat sinks. The present investigation aims at characterizing the influence of mixed convection heat transfer in a single server for SPI cooling. A commercially available computational fluid dynamics (CFD) tool, ANSYS Icepak [22], was used to develop the server model and carry out the conjugate heat transfer simulations. The baseline version of the server design is a 2U open rack unit air-cooled chassis design with an air-cooled heat sink. The server was subjected to varying inlet flow rate boundary conditions from fully natural convection to highly forced flow (0<Re<2000). The analysis of mixed convection flow was done by identifying important dimensionless parameters such as Nusselt number, Richardson number, and Rayleigh number. The variation in the influence of mixed convection was observed by varying the geometry of the heat sink to make it more optimized for low flow rate viscous flows. The orientation of the server was also varied from vertical (same direction of flow and natural convection) to horizontal (transverse flow and natural convection directions) to quantify the variation in mixed convection heat transfer. For a constant heat source area (representing the CPU), the source power was also varied at three values of 115W, 160W and 200 W where 115W is the thermal design power of the current CPU design in the server.

NUMERICAL MODEL AND ANALYTICAL RELATIONS

This section describes the details of the numerical modeling, its boundary conditions, and the required mathematical background on mixed convection.

Numerical Modeling and Methodology FIGURE 1: OVERVIEW OF THE CFD MODEL SHOWING THE SERVER COMPONENTS AND BOUNDARY CONDITIONS

Figure 1 shows the overview of the numerical model and physical layout of the server used in this study. The server is a 3 rd generation Intel-based 3 rd generation open compute server with dual CPU sockets, each with maximum thermal design power (TDP) of 115W, in a thermally shadowed configuration [23]. The simplified numerical model of the server was developed with only the CPU as a 2-D heat source under the heat sink base accompanied by the memory modules with no heat dissipation given to the memory modules. Thermo-physical properties of pure aluminum are imparted to the heat sink which was also modeled without the cutouts and the embedded heat pipe. The heat transfer fluid used for the CFD modeling is a synthetic dielectric fluid EC-100. Table 1 shows the overview of the known temperature-dependent properties of the fluid used for the simulation study. A constant value of density for the inlet temperature of 30°C is used during the study due to the limitation of the CFD tool. The thermal performance of this server in SPI with the baseline air-cooled heat sink was experimentally characterized by McWilliams with mineral oil [24]. The results of this experimental study were used to benchmark the baseline CFD model. A grid independence study was first carried out for the baseline CFD model developed as shown in Figure 2. The value of thermal resistance of the heat sink was used to ascertain the grid independence results. The thermal resistance value was observed to be independent of the global mesh size outside of the heat sink and the base value of mesh elements was used for the simulations. For benchmarking the CFD model, the source temperature value was compared with the junction temperature of the server for different inlet flow rates. As can be seen in Figure 3, a close agreement is observed between the experimental data and the CFD modeling results. The maximum Outlet Inlet DIMMs Heat Sinks difference between the experimental data and the numerical model was observed to be less than 3% for all the simulation cases. The slight variation could be attributed to different fluids being used in both the experimental and numerical study (mineral oil for experimental and thermally enhanced synthetic fluid for CFD). The thermal performance was also reduced in the CFD modeling as the no heat pipes were modeled.

FIGURE 3: COMPARISON OF THE EXPERIMENTAL RESULTS AND RESULTS FROM THE NUMERICAL MODELED

To vary the Re values, the inlet flow rate was varied by defining a velocity inlet boundary condition. The source power is varied for three values of 115 W, 160 W and 200 W. To quantify the participation of mixed convection for low flow rate viscous flows where natural convection may be assumed to be the dominating heat transfer mode, the baseline air-cooled heat sink design was assumed to be not beneficial. A few studies in the current literature also point to this fact [25,27]. As these studies conclude, immersion optimized parallel plate-fin heat sinks result in thicker fins with reduced fin count [28]. Taking this as the base assumption, two different fin count and fin thickness heat sinks were modeled, a baseline heat sink and an optimized heat sink with an increased fin thickness and reduced fin count. The fincount for the baseline heatsink is taken as 35 and the fin thickness is 0.23 mm. On the other hand, the fin count for the optimized heatsink is taken as 25 and the fin thickness is 0.59mm. The heatsink height remained the same as 41 mm in both the cases.

Analytical Relations and Governing Equations FIGURE 4: OVERVIEW OF THE BOUNDARY CONDITIONS USED FLOW DIRECTION AND DIRECTION OF THE GRAVITY

ANSYS Icepak [22] solves Navier-Stokes equations of mass, momentum, species, and energy to calculate heat transfer in laminar flow conditions. Additional transport equations of turbulence and radiation can be used if the flow and heat transfer involves these phenomena, which was not the case for the current study. These equations are written as follows:

Mass conservation:

The above equation reduces to ∇. (ÿ ⃗) = 0 for incompressible fluids.

Momentum equation:

Energy Equation:

Here, the fluid energy equation is written in terms of sensible enthalpy, h. k is the molecular conductivity and kt is the turbulence transport conductivity. The source term Sh represents user-defined volumetric heat sources. For the solid regions, the energy equation due to conduction within the solid looks as follows:

Here, k is the thermal conductivity of the solid, is the density, T is the temperature and Sh is the source term for volumetric heat sources.

For buoyancy-driven flows in mixed or fully natural convection-driven flows are simulated using the Bousinessq model for natural convection. This model treats density as constant in all solved equations except the buoyancy term in the

Gravity

Experimental Numerical

Numerical Experimental momentum equation:

In the above equation, 0 is the constant density of the fluid, ÿ 0 is the operating temperature and is the volume expansion coefficient of the fluid.

The impact of various inlet boundary conditions on the heat transfer behavior using the baseline air-cooled heat sink as compared to an immersion optimized heat sink was calculated using an analytical model for parallel plate heat sinks [29]. This model predicts the average heat transfer rate using a composite solution for fully developed and developing flows in a single channel of an 8n9 channel parallel plate heat sink. The model assumes that the fins are isothermal and have the temperature of the base, assuming the material has a high thermal conductivity. The model also includes the effect of fin efficiency and combines the impact of ideal heat transfer rate (Nui) and average heat transfer rate (Nub). The model uses a modified Reynolds number value which is a dimensionless number combining the basic definition of the Reynolds number with dimensionless channel width as,

Here, L is the channel length and b is the channel width. Reb is a dimensionless Reynolds number calculate using channel width as the characteristic length. The average heat transfer coefficient is related to the ideal value of the Nusselt number (for a fin with η=1) by,

where kf is the fluid conductivity. The final composite model is defined as in equation 8,

Where Nui

Here k is the thermal conductivity of the solid, t is the fin thickness and H is fin height.

Also, Richardson Number (Ri) is defined as the ratio of the buoyancy-induced flow to the forced flow within a fluid system. It helps assess the relative significance of natural convection (caused by buoyancy forces) compared to forced convection (caused by external means such as fans or pumps).

The Richardson Number is expressed as:

where, g is the acceleration due to gravity, β is the coefficient of thermal expansion, ΔT is the temperature difference across the fluid, L is a characteristic length or height scale, ν is the kinematic viscosity of the fluid, and U is the characteristic velocity of the fluid flow.

RESULTS AND DISCUSSION

FIGURE 5: HEAT TRANSFER BEHAVIOR OF THE BASELINE AIR-COOLED HEAT SINK AT 115W, 160W AND 200W FOR VARIOUS MODIFIED REYNOLDS NUMBER VALUES

It was also observed that the heat transfer behavior saturates after a certain flow rate or Reynolds number value. This Reynolds number value occurs at a very high flow rate which will be redundant for the current server's total heat dissipation value. It should also be noted that an optimized heat sink for 160W and 200 W may give a better thermal performance as opposed to the current trend. Figure 5 shows the heat transfer behavior for the baseline heat sink at 115W, 160 W and 200W. It is seen that the value of the modified Nusselt number reduces to almost half of the value in the immersion optimized heat sink case implying that the heat transfer behavior is significantly impacted if an unoptimized heat sink is used, especially for low flow rate values. Figure 6 shows the variation of the modified Nusselt number with varying modified Reynolds numbers for the immersion optimized heat sink when the server is immersed vertically. The modified Nusselt and Reynolds numbers are calculated using the channel width as the hydraulic diameter instead of the wetted channel cross-section and perimeter. It was observed that the heat transfer behavior at low Reynolds number values is initially better for the 160 W and 200 W CPU power for the same heat sink than 115 W CPU power as seen in Table 2. This could be because, at very low flow rates, higher CPU power induces a greater amount of natural convection behavior that assists in better heat transfer. After that, as the heat sink has been optimized for a power value of 115 W, the heat sink produces better heat transfer for that power value thereon. Figure 8 shows the influence of server orientation on heat transfer behavior for the immersion optimized heat sink for 115W CPU power based off the CFD simulations. It can be seen that for the optimized heat sink in the vertical orientation, the heat transfer rate keeps increasing even though it appears to be saturating for high flow rate values. When the server is kept in horizontal orientation, depicting that the forced flow direction is orthogonal to the direction of the gravity vector. For the horizontal immersion case, the heat transfer behavior saturates after increasing rapidly and then starts reducing as the flow rate is increased further. This means, that for horizontal orientation if the impact of natural convection is reduced, the overall heat transfer will reduce after a certain flow rate. This flow rate must be quantified for the servers if a horizontal immersion case so that the thermal performance of the server is not compromised in operational conditions.

CONCLUSION

This study numerically investigated the influence of mixed convection heat transfer in a single server SPI cooling. With increasing, edge deployments and expansion of 5G, natural convection-based immersion tanks operating at elevated ambient temperatures will be useful owing to simple cooling infrastructure and high heat capture capabilities. The following conclusions were drawn from this investigation:

• For the current server design and power values analyzed, natural convection is concluded to be the dominant heat transfer mode • Mixed convection conditions are observed at very high liquid flow rate conditions that are operationally not feasible in single-phase immersion cooling • Immersion optimized heat sink performs better in both vertical and horizontal server orientations • Immersion optimized heat sink also outperformed baseline heat sink outside the optimized power range • The point at which natural and forced convection are balanced occurs at a very large flow rate value for the current server • For server horizontally immersed server, the peak heat transfer point should be characterized as a sharp decrease in heat transfer that occurs thereafter