Over the last several decades, the world has witnessed a substantial increase in the demand for electricity [1]. Consequently, there has been a significant rise in carbon emissions from power generation systems [2], which has raised concerns over their environmental impact and contribution to the climate change crisis [3]. These facts encouraged the research to investigate the potentials for improving systems energy efficiency [4]. Data centers, which support information-driven societies, account for nearly 1.3% of the world s electricity consumption [5]. The energy consumption of these bulk energy consumers was forecasted to increase by approximately 140 billion kWh by 2020, in the US only [6].
It is reported that one to two thirds of a data center s energy consumption is allocated to cooling infrastructure [7,8]. In the electronics industry, increasing heat flux dissipation density in chips has become a bottleneck challenge that restricts further technological development. Thus, data center cooling systems have become the focal point of many researchers who want to investigate energy-saving opportunities while considering reliability concerns.
Increasingly, many emerging cooling technologies like liquid cooling [9] and two-phase cooling [10] are being widely adopted. However, air cooling remains the most reliable and widely used cooling approach to date [11]. R. Gupta, S. Asgari, H. Moazamigoodarzi, S. Pal, and I. Puri [12] compared four cooling architectures for air-cooled data centers from exergy point of view. H. Moazamigoodarzi, P. Tsai, S. Pal, S. Ghosh, and I. Puri [13] compared room, row, and rack cooling architectures in terms of data center power consumption. Their results revealed that adopting cooling architectures at the row and rack levels decreased the overall cooling system power consumption by up to 29% when compared with the room level cooling architecture. Z. Song [14] examined using fans to assess the cooling system performance in data centers with and without Cold Aisle Containment (CAC). H. Lu and Z. Zhang [15] analyzed the data center thermal performance variation while changing its geometrical configurations and the computer room s air conditioning arrangement. X. Meng, J. Zhou, X. Zhang, Z. Luo, H. Gong, and T. Gan [16] optimized the thermal performance of a relatively small data center in China. Y. Zhang, K. Zhang, J. Liu, R. Kosonen, and X. Yuan [17] constructed T-shaped underfloor air ducts in a modular data center to improve the airflow uniformity. A. Almoli, A. Thompson, N. Kapur, J. Summers, H. Thompson, and G. Hannah [18] developed a new computational fluid dynamics (CFD) methodology for modeling data centers that adopted rear door heat exchangers. M. T-Evans, N. Kapur, J. Summers, H. Thompson, and D. Oldham [19] investigated the effect of air bypass and recirculation in data centers on the cooling infrastructure power consumption. J. Cho and J. Woo [20] introduced an in-row coolers for improving the thermal environment of a data center by providing a lesser air distribution path. L. S-Llanca, A. Ortega, K. Fouladi, M. Valle, and V. Sundaralingam [21] tested direct and indirect exergy destruction approaches against simplified and actual data center flow. They found that using the direct method results in a better when calculating the exergy destruction for the airside in data center. C. Zhou, C. Yang, C. Wang, and X Zhang [22] conducted a numerical analyses on a cooling system designed for small-scale data centers. J. Athavale, M. Yoda, and Y. Joshi [23] compared different data-driven modeling approaches according to their ability to predict temperatures in data centers. A. Khalaj, T. Scherer, J. Siriwardana, and S. Halgamuge [24] tested the impact of distributing the workload on a data center s cooling system s efficiency. It is important to note that all these studies employed CFD models whilst conducting their analyses at the data center facility level.
Over the years, CFD techniques have been recognized as the most convenient tool for designing, characterizing, and diagnosing data centers [17,25]. Using a CFD digital twin model of a data center can accurately replicate the data center s performance at different operating conditions. compared to other data center modeling techniques, such as proper orthogonal decomposition [26 29], console-based simulation tools [30], thermodynamic flow network modeling [31], and machine learning models [32 35], CFD simulations are the most adoptable [36]. However, CFD models are also the most computationally intensive among all modeling approaches. As a result, compact models have been employed in CFD tools as a method for avoiding excess computational cost. A compact model assumes that the complexities of physics inside an object can be represented by an approximate model of the object s impact on its surroundings at the locations where the object and the surroundings interact. This assumption is less computationally intensive than those made by traditional CFD simulations because it requires less level of detail. In facility level analyses of white space, compact models are considered necessary for modeling the complexities of IT equipment Additionally, server compact models are not limited to CFD analyses, but can also be used to conduct thermal and energy management functions in a data center, identify the potential for power management, predict power consumption, and balance the needs of the data center, and so on [37]. Other approaches to developing compact models for IT equipment and power consumption have been investigated by researchers for use in CFD modeling at the data center facility level. H. Cheung, S. Wang, C. Zhuang, J. Gu [38] presented IT equipment simplified model for calculating power consumption in a data centers. This model was developed to be utilized in simulating the dynamic IT equipment energy performance in data centers under various operation conditions. J. VanGilder, Z. Pardey, X. Zhang, and C. Healey [39] introduced a compact server model that captured the effects of the server s thermal mass. H. Moazamigoodarzi, R. Gupta, S. Pal, P. Tsai, S. Ghosh, and I. Puri [40] developed IT server enclosures compact model which can capture the temperature distribution and power consumption inside the enclosures. S. Erden, E. Khalifa, and R. Schmidt [41] proposed a new method to derive some server s properties such as thermal capacitance and conductance from external measurements. A review of existing power consumption models for IT equipment was conducted by C. Jin, X. Bai, C. Yang, W. Mao, and X. Xu [37]. All these studies focused mainly on predicting the power consumed by the IT equipment or on developing a simplified equipment model without considering the IT equipment s different operating conditions. Furthermore, these studies ignored the physics-based variations between different IT equipment, which significantly affected their model s accuracy, especially at the offdesign conditions. Lastly, none of the previous studies were able to provide an accurate model for calculating the IT equipment s airflow requirements.
In a real-world data center, IT equipment operating conditions, such as the data center s architecture, spatial location, inlet pressure, inlet air temperature, and CPU utilization, can vary significantly. This requires a physics-based compact model that can predict the IT equipment s behavior under off-design conditions. Since error can be significantly magnified when implementing these compact models in data center facility level CFD models, the compact model s degree of accuracy at different operating conditions is a mandatory factor that must be considered. Moreover, capturing the amount of flow required by each IT equipment is vital for power and capacity planning in data centers. Generally, data center operators tend to overprovide IT equipment with cool air to ensure their reliability. Adopting this practice results in a huge amount of wasted energy by the Computer Room Air Handler (CRAH) unit, which is responsible for a considerable share of the cooling system s total power consumption. On the other hand, underprovisioning leads to an increase in the amount of hot air recirculation and the formation of hotspots, consequently decreasing the cooling system s efficiency and exposing the equipment to reliability issues. Therefore, accurately predicting the amount of air flow required by IT equipment is as critical as predicting its heat dissipation, since it is used to calculate the cooling units air delivery.
This study presents a novel extended method of building an IT equipment compact model that employs a combination of machine learning (regression analysis) and CFD simulation. By integrating machine learning with the detailed CFD IT equipment model, the CFD model transforms from a rigid model that can predict the IT equipment performance at the design point conditions to a dynamic model which is capable of predicting the overall performance at various environmental conditions. As this model simulates the overall performance of the IT equipment, it can be used to derive a compact model that predicts the required amount of flowrate and the air temperature leaving the IT equipment, besides the power dissipated by the equipment. The key novelty of this approach that distinguish it from existing methods for building compact models of IT equipment, and they can be summarized as follows:
This approach captures the IT equipment s behavior under different environmental conditions in the data center, which is essential for predicting the data center s energy load demand, hence regulating the data center s power consumption [42]. This model adapts a full physics-based model, hence it is expected to provide very accurate predictions at both design and off-design conditions. All the data and tools used in this approach are readily available to almost everyone involved in the thermal management of a DC (CFD software, technical manuals, Intelligent Platform Management Interface (IPMI) data, etc.). This approach can also compute the amount of airflow required by the IT equipment at various operating conditions with a high degree of accuracy.
Combined, these distinguishing characteristics provide the foundation for a novel method of building an IT equipment compact model. Beside introducing the novel approach, the integration of machine learning algorithms with the IT equipment CFD model, where the machine learning model was used to generate data that is used as input for IT equipment CFD model, is completely new.
For many years, specifying the accurate flowrate demand in data centers at different operating conditions is identified as a key restriction for further improvement in the data center s cooling system s energy efficiency. Without providing an accurate practical solution, the incapability of calculating the airflow demand leaves a remarkable knowledge gap in the available literature. Hence, this study is to provide a significant contribution to the new body of knowledge as it introduces a reliable and fast tool for calculating the flowrate demand in data centers. The following are the major advantages of utilizing this tool:
By integrating this tool into the thermal and energy management of a DC, its cooling system capacity planning can potentially be expanded to cover calculating the amount of airflow required by the equipment, along with the IT equipment power consumption and heat dissipation. Since this extended method considers the amount of airflow required by the IT equipment, with the appropriate adjustments to the airflow supply from the cooling unit it introduces the potential for substantial energy savings in currently operating data centers. It was proven that adopting effective airflow management in data centers could result in a substantial amount of energy savings [43]. According to a previous study, adopting proper flow management showed a possibility for significant energy saving opportunities on both the CRAH unit and the chiller, where the CRAH power consumption was reduced by 75% and the chiller power consumption was reduced by 16% [44]. Using this tool can fill in the lack of information on IT equipment s flow requirements, which can potentially prevent cooling system oversizing or overprovisioning in future data centers. When it comes to CFD models, implementing this approach could remarkably improve their accuracy and reduce their computational costs, especially during transient simulations in which a data center experiences various operating conditions. This tool eliminates the need to calibrate CFD models because the IT equipment compact model can replicate accurate performances at different environmental conditions. The remainder of this paper is organized as follows: the method used to develop the IT equipment compact model is described, including the details of data collection, deriving the power consumption model, the fan control algorithm, the numerical model, and the regression function. Then, the experimental setup that was used to validate the results is introduced. Thereafter, the results from each stage of development for the IT equipment compact model are discussed and compared with the experimental results. Finally, further conclusions are drawn in the last section.
s p e ci fi c s er v er.
W
[ 6 8 -7 0 ] :
T his is f e asi bl e si n c e y + li es wit hi n t h e r a n g e 3 0 < y + l ess t h a n 3 0 0, Fi g. 6. C A D dr a wi n g of t h e I T e q ui p m e nt i n v esti g at e d i n t his st u d y.
T a bl e 1 I T e q ui p m e nt g e o m etri c al p ar a m et ers.
c o n d u ct all a n al ysis i n t his w or k is 1 5. 4 milli o n.
o ns e q u e nti al I T e q ui p m e nt fi o wr at e r e q uir e m e nt.
I T e q ui p m e nt P o w er = 0 .0 0 0 2 Utiliz ati o n 3 -0 .0 4 6 2 Utiliz ati o n 2 + 3 .7 0 9 1 Utiliz ati o n + 3 1 .0 1 3 ( 1 0)
Notably, fan speed is critical for calculating the IT equipment airflow demand. Yet, fans themselves can have different control algorithms. Therefore, machine learning techniques were used to develop models that can predict the speed of fans individually. The machine learning algorithm is integrated with the CFD model, where the machine learning prediction is used as an input to the CFD model to develop a dynamic CFD model that can predict the performance of the IT equipment at different operating conditions. After that, a compact model (regression model) was extracted from this dynamic CFD model.
As mentioned earlier, RF provides a way to rank features based on their importance, because of this, RF s are also used as a feature selection method. The features ranked by the RF algorithm that impact the fan speed are illustrated in Fig. 11. The RF hyperparameters used throughout this study are as following: each RF consisted of 50 DTs, the maximum number of features considered when looking for the best split was six, the minimum number of samples required to split an internal node was set to two, and the maximum depth of each tree was set to none so that nodes are expanded until all leaves contain less than the minimum number of samples set, which is two.
Based on Fig. 11, the feature selection results showed that the most relevant contributor to all fans was the SAT. According to the similarity in the feature weight order, it was observed that there were two main fan control algorithms within the IT equipment. One was for fans 1, 4, 5, and 6; the second was for fans 2 and 3. The similarity was not only in the feature weight order, but also in the weight contribution. Fans 1, 4, 5, and 6 were mainly controlled by the SAT, while the rest of the features had minimal contribution. Meanwhile, fans 2 and 3 have a second major contributor beside the SAT, which was the total IT equipment power. This conclusion that multiple fans within the same IT equipment could have multiple fan control algorithm was confirmed by a server vendor company.
Some of the features used for developing the prediction models are not reported by all IT equipment. Additionally, some of the highly correlated or redundant features must be excluded from the model. For highly correlated features, having two or more independent variables (features) that are highly correlated with each other and share almost the same information, this is a phenomenon in statistics called multicollinearity. When present, multicollinearity can cause the estimate of one variable s impact on the dependent variable while controlling for the others tends to be less precise than if the independent variables were uncorrelated with one another. A commonly used way to address this issue is that only one of the highly correlated features needs to remain, while the rest are removed. Regarding redundant features, some datasets may include features that have little to no significance impact on the output and removing them may decrease the model s complexity and increase its performance and accuracy.
For these reasons, the number of features used in the fan speed prediction models were reduced. Feature reduction was carried out while considering the CFD boundary conditions as these data should be available to run the model. Initially, the prediction model was modified to predict the fan speed using the SAT, CPU utilization, and CPU power. After that, the model was customized to predict the fan speed based on just the SAT and CPU utilization. A total of three different models were developed, Model 1 considered all the features that remained after removing highly correlated and redundant features; Model 2 used the features SAT, CPU utilization, and CPU power; and Model 3 used the features SAT and CPU utilization. To evaluate their reliabilities, the R 2 performance measure was used, and 10-fold cross validation was done across all runs, where, as previously mentioned, training is carried out on nine folds and tested on the remaining fold, and this is repeated ten times, so that each fold is trained nine times and tested once. The average performance across all folds is taken. In addition, a validation set that was not used in the training phase of the models was used to validate the performances of feature selection and predictions. In regression problems, the most common performance metric for evaluating a machine learning model is the R 2 coefficient. The R 2 coefficient, also called the coefficient of determination, indicates the proportion of variance in the dependent variable that is predictable from the independent variables. It is a statistical measure of the goodness of fit of a model and is calculated using the following equation: (11) Where y i is the true output value of observation i, y i is the predicted value of observation i, and y is the average value of the dependent variable y. Generally, the closer R 2 value to unity, the better the model accuracy, and when R 2 1, it indicates that the regression model fits perfectly and explains all variance. For model 1, the calculated R 2 coefficients revealed that the fan speed predictions fit almost perfectly to the experimental data. The lowest R 2 was found to be 0.957, belonging to fan 4, while the highest was 0.985, belonging to fans 2 and 3. The lowest calculated R 2 value drops down to 0.77 when only two features were considered. R 2 values for the six fans that were obtained using the three prediction models are summarized in Table 3.
Table 3 shows that reducing the number of inputs for the fan speed prediction model decreased its prediction ability. Nonetheless, Model 2 showed an impressive performance since its lowest calculated R 2 value of all the fans was 0.84. In Model 3, when the CPU power was disregarded, the model s predictive ability deteriorated considerably and the lowest calculated R 2 value dropped to 0.77. Table 3 also shows that fans reacted differently when the number of inputs was reduced. This agrees with what was concluded from Fig. 11 that the fans have various control algorithms.
To further demonstrate RF s capability, its performance was compared with linear regression, lasso regression, support vector machine (SVM) with a linear kernel, and a SVM with a radial basis function (RBF) kernel. The R 2 for the different algorithms on Model 1 is shown in Table 4, where the RF achieved the highest performance for all fans.
As mentioned earlier, not all IT equipment reports the same features. Hence, in order to generalize the proposed approach and to facilitate adopting in building compact models for various IT equipment, a machine learning prediction model that predicts the fan speed using common features which are reported by different IT equipment should be considered. Accordingly, and since that Model 2 provided high R 2 values while considering only three features (SAT, CPU utilization, and CPU power), it was chosen to conduct investigations in the remainder of this study.
After the power dissipation and fan speed prediction models were developed, their results were incorporated into the CFD model. By doing so, it was expected that the CFD model would replicate the IT equipment s performance at off-design conditions. For validation the CFD model was tested against the experimental results considering different cases. These cases were selected to cover a wide range of the IT equipment s operating conditions, as shown in Table 5. The corresponding predicted and experimental values of the fans speeds for these cases are shown in Table 6.
The fan speed model showed an impressive prediction ability, seeing as the maximum absolute error was found to be less than 12%, as provided in Table 6. The impact of the 11.9% discrepancy between the experimental and predicted fan speeds on the overall mode accuracy is expected to be minimal. This is attributed to the fact that each fan contributed with almost 1 Totalnumberoffans when calculating the total IT equipment flowrate. Thus, even if one or two of the fans has a relatively large error, the overall system will continue to show great prediction ability.
A comparison of the numerical results and the field measurements, in terms of outlet air velocity and temperature, is provided in Table 7. It can be inferred from this Table 7 that there was great consistency between the results in all cases. For the outlet air velocity, the maximum error was found to be 5.4% and the minimum was 1.8%. For the outlet air temperature, the maximum error was found to be 4.6% and the minimum was 1.1%. This confirms what were discussed earlier in this section, where case 3 exhibited 11.9% absolute relative error in predicting one of the fans speeds, however, CFD results showed only 2.4% and 1.6% error in the overall IT equipment outlet air velocity and temperature, respectively. Based on this, it was found that the reliability and accuracy of the CFD model applied in this work is comprehensively validated.
Thereafter, the model was utilized to calculate the IT equipment s flowrate and outlet air temperature at various operating conditions as shown in Fig. 12. As expected, Fig. 12 (a) reveals that the air flowrate increased when the SAT and the CPU utilization increased, which was due to the increase in fan speed. At some points when the flowrate increased the OAT exhibited a slight variation or even stayed constant, even when the SAT increased as more flowrate was delivered. Furthermore, can be inferred from Fig. 12 (b) that the OAT was almost identical for different CPU utilizations, which meant that the fans speeds were being adjusted to maintain a certain thermal field inside the IT equipment at different SAT regardless of CPU utilization.
It can be also noted from Fig. 12 (b) that the air temperature difference between the server inlet and outlet is rather low. This highlights that this specific IT equipment is adopting excessively conservative fan control algorithms by running the fans at unnecessary high speed. Adopting such algorithms result in wasting a significant amount of energy by the IT equipment fans, especially when hundreds of such IT equipment are operating in different data centers. This leaves a huge spot for improving and optimizing IT equipment fan control algorithms. For example, the work conducted by J. Sarkinen, R. Brannvall, J. Gustafsson, and J. Summers [82] in which they specified an optimal operating condition considering the components power losses and the fan power consumption. In addition, this relatively low T across the IT equipment could cause energy losses by the data centers cooling equipment in most cases. This attributed to fact that more air is required by the IT equipment to maintain this T across the IT equipment, and hence more air consumed by the CRAH units blowers. More in-depth discussion can be found in [82]. ul at e wit hi n t h e t est e d I T e q ui p m e nt 's c h assis, as s h o w n i n Fi g. 1 3 . As a r es ult, t h e air fi o w p assi n g t hr o u g h t h e I T e q ui p m e nt w as l ess t h a n t h e t ot al fl o w dri v e n b y t h e f a ns, w hi c h m e a nt t h at t h e air fl o w l e a vi n g t h e I T e q ui p m e nt w as at a hi g h er t e m p er at ur e. For the latter model, both the R 2 and adjusted R 2 values were very close to unity, which verified the quality of fit and the model s ability to predict new operating points. Moreover, the R 2 and adjusted R 2 values were high and similar, which indicated that the model was not overfitted. Finally, the function accurately predicted the IT equipment s flowrate and OAT, with an MAE of 5.1 and 1.7, respectively.
After conducting a diagnostic of the regression model and validating it statistically, its accuracy was assessed comparatively using the experimental data. The experimental results used for comparison were those obtained in the four cases discussed in the previous section. Table 9 and Table 10 present a comparison of the results obtained through the experiment and those predicted by the regression model for the IT equipment s flowrate and the OAT, respectively. Unfortunately, to the best of the authors knowledge, no article has created a compact model which predicts the flowrate required for the IT equipment and the air temperature leaving the equipment. Therefore, to further assessing the developed compact model, these two Tables compared the developed regression function with state-of-the-art compact for the same IT equipment. This state-of-the-art model is currently used in some CFD software, which adopts generalized and linear correlation for the various IT equipment.
For the regression model, the maximum error was found to be 11.4% and 5.7% for the IT equipment s flowrate and OAT, respectively. This confirms its accuracy and ability to predict the IT equipment s thermal performance at off-design operating conditions. For the state-of-the-art compact model, the discrepancy between its results and the experimental results was substantial. In some cases, the flowrate calculated by this model was more than double that of the experimental results. For the OAT, slightly higher than an 8 C difference was observed. The regression model developed in this work reduced the error for the IT equipment s flowrate to 5.2% that of the state-of-the-art IT equipment compact model. For the OAT, it reduced error to 9.3 % that of the stateof-the-art compact model.
With hundreds of IT equipment components installed at the data center facility level, the impact of any error introduced by the state-of the art compact model is certain to grow considerably. As a result, calibration experiments are seen as necessary for improving a given model s reliability. However, the method used for developing an IT equipment compact model in this study can potentially eliminate the need for calibration. It has been shown that this methodology can tremendously improve the CFD model s accuracy at the data center level and its ability to predict the IT equipment s thermal performance even at off-design conditions.
Proper thermal and energy management of data centers requires a thorough understanding of the basic unit for power and heat flow in data centers, which is the IT equipment. This study establishes a comprehensive methodology that builds an accurate compact model for the IT equipment. Initially, a power consumption model and fan speed prediction models were derived from the IT equipment data. Merging these models with the CFD code results in an IT equipment CFD model that can predict the IT equipment at off-design conditions. To verify the reliability of this model it was tested against the experimental measurements under four different operating conditions. The maximum mismatch between the CFD model and the experiments was found to be 5.4% and 4.6% in terms of air velocity and air temperature, respectively. Finally, a compact model for the IT equipment was created using the data generated by the CFD model. Compared to the experiment, the developed model showed a maximum inconsistency in the IT equipment flowrate and outlet air temperature of 11.4% and 5.7%, respectively. This model is superior to existing IT equipment compact models for the following reasons:
It can be easily embedded with any CFD code or any other simulation program used in designing energy systems for data centers. This approach can be adapted to build a compact model for any IT equipment. The model can simulate the IT equipment performance at different supply air temperature and CPU utilization. This model does not only predict the power consumption of the IT equipment, but also the amount of flow required by this equipment and its OAT. This model can be derived using tools and data that are easily accessible. As this compact model is developed from a full physics based CFD model, it is expected to be the most accurate IT equipment compact model. Compared with a state-of-the-art IT equipment compact model, the maximum error in the flow rate was reduced from 155.4% to 11.4%. For the outlet air temperature, the maximum error was reduced from 30.2% to 5.7%.