<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Use of Real-Time Information to Predict Future Arrivals in the Emergency Department</title></titleStmt>
			<publicationStmt>
				<publisher></publisher>
				<date>01/01/2023</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10401413</idno>
					<idno type="doi">10.1016/j.annemergmed.2022.11.005</idno>
					<title level='j'>Annals of Emergency Medicine</title>
<idno>0196-0644</idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>Yue Hu</author><author>Kenrick D. Cato</author><author>Carri W. Chan</author><author>Jing Dong</author><author>Nicholas Gavin</author><author>Sarah C. Rossetti</author><author>Bernard P. Chang</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Study objective:We aimed to build prediction models for shift-level Emergency Department (ED) patient volume that could be used to facilitate prediction-driven staffing. We sought to evaluate the predictive power of rich real-time information, and understand 1) which real-time information had predictive power, and 2) what prediction techniques were appropriate for forecasting ED demand.
Methods:We conducted a retrospective study in an ED site in a large academic hospital in New York City. We examined various prediction techniques including linear regression, regression tree, extreme gradient boosting, and time series models. By comparing models with and without real-time predictors, we assessed the potential gain in prediction accuracy from real-time information.Results: Real-time predictors improved prediction accuracy upon models without contemporary information by 5%-11%. Among extensive real-time predictors examined, recent patient arrival counts, weather, Google trends, and concurrent patient comorbidity information had significant predictive power. Out of all the forecasting techniques explored, SARIMAX (Seasonal Auto-Regressive Integrated Moving Average with eXogenous factors) achieved the smallest out-ofsample RMSE (Root-Mean-Square Error) of 14.656 and MAPE (Mean Absolute Percentage Error) of 8.703%. Linear regression was the second best with out-of-sample RMSE and MAPE equal to 15.366 and 9.109%, respectively. Conclusion: Real-time information was effective in improving prediction accuracy of ED demand. Practice and policy implications for designing staffing paradigms with real-time demand forecasts to reduce ED congestion were discussed. * ) : 𝛽 * + (1 + 𝜙 , * )𝑦 2&, + (𝜙 . * -𝜙 , * )𝑦 2&. + (𝜙 / * -𝜙 . * )𝑦 2&/ -𝜙 / * 𝑦 2&0 + 𝜖 2 * + 𝜃 , * 𝜖 2&, * + 𝜃 . * 𝜖 2&. * + 𝜃 / * 𝜖 2&/ * + 𝜃 0 * 𝜖 2&0 * = (𝑥 2 * ) : 𝛽 * + 0.693𝑦 2&, + 0.328𝑦 2&. -0.648𝑦 2&/ + 0.627𝑦 2&0 + 𝜖 2 * + 𝜃 , * 𝜖 2&, * + 𝜃 . * 𝜖 2&. * + 𝜃 / * 𝜖 2&/ *where 𝑥 2 * is the vector of covariates in the embedded linear regression model, and 𝛽 * is the associated vector of estimated coefficients, whose value, together with the other estimated parameters denoted with an asterisk in the superscript, is provided in Table 3. In addition, the estimated value for 𝜎 is 14.032. Note that 𝑦 2&, is the arrival count during the previous shift, 𝑦 2&.is the arrival count during the shift before the previous shift, and 𝑦 2&0 is the arrival count during the same day/night shift on the day before previous day. The estimated coefficients suggest that 𝑦 2&, , 𝑦 2&. , and 𝑦 2&0 are positively correlated with 𝑦 2 , the arrival count during the focal shift.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head>INTRODUCTION</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Background and Importance</head><p>Across the United States, there has been a growing focus within Emergency Medicine on developing computational/machine learning models to predict demand for patient care in the emergency departments (ED). Over the years, a variety of prediction techniques have been examined. Early studies have utilized time-series models to forecast future arrivals based on recent arrival count information. <ref type="bibr">[1]</ref><ref type="bibr">[2]</ref><ref type="bibr">[3]</ref><ref type="bibr">[4]</ref><ref type="bibr">[5]</ref><ref type="bibr">[6]</ref> Additional studies have utilized other prediction models with exogenous predictors, such as linear regression, regression tree, etc. <ref type="bibr">[7]</ref><ref type="bibr">[8]</ref><ref type="bibr">[9]</ref><ref type="bibr">[10]</ref><ref type="bibr">[11]</ref> There have also been recent efforts that explored techniques to combine time-series models with exogenous features. <ref type="bibr">12-</ref> Most of the existing literature has utilized classic predictors such as seasonality, holidays, weather, and previous arrival counts. A few other studies have examined limited real-time information beyond weather and previous arrival counts, including ambulance diversion status and physician workload. <ref type="bibr">[14]</ref><ref type="bibr">[15]</ref><ref type="bibr">[16]</ref> However, to the best of our knowledge, little research has explored the comprehensive patient-level and regional data that are now more readily available. Such data could provide novel additional information and improve ED demand prognostication.</p><p>An important motivation behind these developments is that predictive information about ED demand can be used to improve operational efficiency in resource allocation and better meet patients' needs. <ref type="bibr">17</ref> Such proactive planning is particularly relevant for nurse staffing, as nurses provide a substantial portion of patient care and are an increasingly limited resource in the ED (e.g., due to nursing shortages exacerbated by burnout and quitting during the coronavirus (COVID-19) pandemic <ref type="bibr">[18]</ref><ref type="bibr">[19]</ref><ref type="bibr">[20]</ref> ). Inefficient and inadequate staffing is often associated with ED crowding, reduced quality of care, clinician burnout, and reduced hospital revenue. <ref type="bibr">[21]</ref><ref type="bibr">[22]</ref><ref type="bibr">[23]</ref><ref type="bibr">[24]</ref><ref type="bibr">[25]</ref><ref type="bibr">[26]</ref> In the current nurse staffing practice, EDs typically divide a day into multiple shifts. The ED manager staffs most of the nurses for a shift weeks to months in advance. A few hours before the nursing shift, the ED manager could call in extra nurses with incentive pays if s/he senses a higher patient volume that renders the planned staffing level insufficient (after taking into account staffing fluctuations due to sick calls and personal emergencies). We refer to the former as basestaffing and the later as surge-staffing. ED demand forecasts synchronized with these two staffing decision epochs can greatly facilitate these decisions. Since overtime/surge staff are more expensive and less convenient for nurses, it is important to understand how much we can improve the prediction accuracy at the surge stage (when we can utilize more real-time information) compared to the base stage (when limited information about the shift is available).</p><p>A recent study shows that even a small accuracy improvement at the surge stage can lead to effective prediction-driven two-stage (base and surge) nurse staffing policies. <ref type="bibr">27</ref> However, little is known about whether (and if so, by how much) real-time information improves prediction accuracy in practice.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Goals of This Investigation</head><p>The goal of this study was to explore and evaluate rich real-time information (including previous arrival counts, temporal and seasonal variations, holidays, weather, electronic health records, and Google trends), and a variety of prediction techniques. By comparing prediction models with and without real-time predictors, we assessed the gain in prediction accuracy from real-time information. Lastly, we described how these two types of prediction models (with and without real-time information) could both contribute to a prediction-driven staffing framework.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>METHODS</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Study Setting and Objective</head><p>We conducted a retrospective study using data obtained from the electronic health records for an adult ED in a large academic hospital in New York City. A total of 284,550 adult patients who arrived at the ED from 12:00 AM January 1, 2018, through 11:59 PM January 31, 2021, were included in the analysis.</p><p>At the hospital, each day was divided into two main 12-hour nursing shifts that start at 7:00 AM and 7:00 PM, respectively. To facilitate relevant operational decision making (e.g., nurse staffing decisions), the subject of prediction was the shift-level arrival count defined as the total number of patients who arrived at the ED during each shift. Many hospitals have more nursing shifts than the two listed above. In those cases, we can divide the day into non-overlapping intervals and predict the interval-level arrival count similarly.</p><p>Model fitting and selection was performed using one year of data from January 1, 2018 to January 31, 2019, which we hereafter refer to as the training set. Model performance was tested on the subsequent one-year data from February 1, 2019 to January 31, 2020, which we refer to as the test set. The remaining data from February 1, 2020 to January 31, 2021 contained the outbreak of the COVID-19 pandemic, and we thus refer to it as the COVID test set. Since patient volume was highly unpredictable during the pandemic and the pandemic is likely a unique generational event, we relegate the results and discussions regarding the COVID test set to Appendix A. The training, test, and COVID test sets were fixed across all prediction models. This study was approved by Columbia University Institutional Review Board: Protocol IRB-AAAT6452.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Data Source</head><p>We utilized three sources of data: patient electronic health records, weather data published by the National Centers for Environmental Information, <ref type="bibr">28</ref> and Google trends. <ref type="bibr">29</ref> These data sources were selected based on past work, extant models, and our own novel hypotheses. While the importance of weather information has been well established in the literature, <ref type="bibr">[14]</ref><ref type="bibr">[15]</ref><ref type="bibr">[16]</ref> the prediction power of real-time patient electronic health records and Google trends has been relatively underexplored.</p><p>The data extracted from the patient electronic health records specified for each patient: (i) the patient's clinical time stamps in the ED, including arrival time, first evaluation time, admission decision time, and departure time; (ii) the arrival source of the patient, e.g., walking in or by ambulance; (iii) the patient's chief complaint(s), i.e., reason of visit; (iv) the patient's Emergency Severity Index (ESI); (v) lab and imaging ordered: indicators for whether lab, CT, MRI, Ultrasonography, and X-ray were ordered; (vi) indicator for whether the patient was admitted into the hospital; (vii) the Charlson comorbidity index (CCI) based on a list of 17 comorbidities;</p><p>(viii) age; and (ix) indicator for whether the patient left without being seen.</p><p>In addition to the patient electronic health records, we obtained retrospective daily weather information, including the minimum temperature, precipitation, snow, wind, and a hot-weather indicator for whether the maximum temperature exceeds 86&#176;F (30&#176;C).</p><p>The last source of data came from Google trends, which specified, for each day, the relative Google search volume for the words "flu", "emergency room", "abdominal pain", "respiratory infection", "chest pain", "depression", "heart attack", "abuse", "disorder", "weather", and "hospital" in New York State. We came up with the list of keywords based on existing studies and our own novel hypotheses. Araz et al. (2014)  <ref type="bibr">30</ref> established that the Google trends for "flu" were able to forecast influenza-like-illness related ED visits. Tuominen et al. (2022) <ref type="bibr">31</ref> found that the Google trends for "ED" facilitated prediction. The other Google trends keywords were constructed based on our own hypotheses. Since the most frequent reasons for ED visits were abdominal pain, respiratory infection, and chest pain, <ref type="bibr">32</ref> we hypothesized that the Google search volumes for these keywords were positively correlated with ED visits. In addition, we hypothesized that the search volumes for "depression", "heart attack", "abuse", "disorder" signaled relevant illnesses in the neighborhood. Moreover, the Google search record for "weather" might reflect citizens' subjective perception of weather conditions which might influence their stay-at-home/travel plans. Lastly, similar to "ED", a higher Google search volume for "hospital" might indicate that more patients were seeking care.</p><p>When selecting the data sources, we tried to be comprehensive by including as much potentially relevant information as possible. Later in the Model Training and Feature Selection section, we discuss procedures to train different prediction models and identify relevant predictors.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Data Processing</head><p>We processed the data into shift-level predictors. The data regarding day vs. night, day of the week, month, season, near-holiday indicators, weather, and Google trends were readily available at the shift level. As for the data from electric health records, we constructed the following three categories of shift-level predictors.</p><p>The first category was the previous arrival counts, which specified for each shift, the arrival count 1 day ago and 7 days ago, as well as the moving average of the shift-level arrival count over the last 30 days. More precisely, the arrival count on the previous day was the total number of patients who arrived during the previous 24 hours. The arrival count on the previous nth day was the two shifts between the previous 24*(n-1)th and 24*nth hour.</p><p>The second category of predictors was the patient comorbidity information, which we processed into the following three sets. The first set specified for each comorbidity, the total number of patients with that comorbidity on the previous day, i.e., during the previous two shifts, and the sum and weighted sum of CCIs for all patients on the previous day. The second set contained similar information as the first set, but instead of considering the previous-day, calculated the average daily number of patients with each comorbidity over the last 3 days, as well as the average daily sum and weighted sum of CCIs for all patients over the last 3 days. The third set calculated for each comorbidity, the percentage of patients with that comorbidity over the last 3 days, as well as the average sum and weighted sum of CCIs per patient over the last 3 days. The difference between the second and third sets was that the third set considered average comorbidity measures on the individual level, and was not influenced by how many patients arrived over the last 3 days. The motivation to consider comorbidity information over the last 3 days was due to the existing findings that patients with certain comorbidities are more likely to be readmitted to the ED within 72 hours <ref type="bibr">[33]</ref><ref type="bibr">[34]</ref> . These three sets of information were likely to be correlated. Since it was a priori unclear which specification had the most predictive power, we left it to the model training and feature selection procedures to sift out redundant information and identify important features.</p><p>The third category of predictors was the recent ED volume and patient severity information on the previous day (i.e., during the 24 hours before the focal shift). This included the total number of patients who arrived by ambulance, the total number of patients with ESI from 1 to 5, the total number of labs, CT, MRI, US, and XR ordered, the total number of patients admitted to the hospital, the total number of patients whose age exceeds 65 years old, the total number of patients whose age exceeds 80 years old, the total number of patients who left without being seen, the average waiting time (from arrival time to first evaluation time), the average treatment time (from first evaluation time to discharge decision time), and the average boarding time (from discharge decision time to departure time) on the previous day. Intuitively, the waiting and boarding times captured how busy the ED was on the previous day.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Model Evaluation</head><p>We focused on two measures of forecast accuracy for shift-level arrival counts---the root mean square error (RMSE) and the mean absolute prediction error (MAPE). Let (y1, y2, &#8230;, yn) be the vector of observed arrival counts for a total of n shifts, and let (&#119910; "1, &#119910; "2, &#8230;, &#119910; "n) be the corresponding vector of predicted arrival counts given by the prediction model. The RMSE was the square root of the mean squared error between the predicted and observed values:</p><p>The MAPE was the average percentage error of the prediction:</p><p>Both RMSE and MAPE are standard measures of prediction accuracy. <ref type="bibr">[1]</ref><ref type="bibr">[2]</ref><ref type="bibr">[3]</ref><ref type="bibr">[4]</ref><ref type="bibr">[5]</ref><ref type="bibr">[6]</ref><ref type="bibr">[7]</ref><ref type="bibr">[8]</ref><ref type="bibr">[9]</ref><ref type="bibr">[10]</ref><ref type="bibr">[11]</ref><ref type="bibr">[12]</ref><ref type="bibr">[13]</ref> Hereafter, we refer to the RMSE (MAPE) calculated on the training set as the training RMSE (MAPE), and on the test set as the test RMSE (MAPE). In addition to the overall RMSE and MAPE, we also examined the over-estimation and under-estimation error separately.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Model Training and Feature Selection</head><p>Using the predictors developed in the Data Processing section, we examined various prediction models. For the baseline models without real-time information, as we had relatively few predictors, we trained linear regression and regression tree models, only. As we incorporated more real-time information, in addition to linear regression and regression tree, we trained more sophisticated models including extreme gradient boosting (XGBoost), seasonal autoregressive integrated moving average (SARIMA), and SARIMA embedded with linear regression (SARIMAX). Comparatively, linear regression and regression tree models are highly interpretable statistical models, but may be inadequate for nonlinear or autocorrelated data. SARIMA and SARIMAX models are time-series models that are effective of modeling seasonal trends and autocorrelation. XGBoost is a sophisticated black-box model for complex and nonlinear relationships, but is less interpretable than the other models. <ref type="bibr">10</ref> To select the relevant features, for linear regression, we used a modulated two-way stepwise model selection method based on the Akaike's information criterion (AIC). For regression tree and XGBoost, we used 10-fold cross-validation for hyperparameter tuning. For the time-series models, we used a variation of the Hyndman-Khandakar algorithm <ref type="bibr">39</ref> to determine the hyperparameters, Detailed training and feature selection procedures for each model are provided in Appendix B.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>RESULTS</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Models without Real-Time Information</head><p>We referred to the linear regression model without real-time information as LR1. The significant covariates in LR1 were day vs. night, day of the week, month, and holidays. On the test set, LR1 achieved an RMSE of 16.425 and an MAPE of 9.627%. Table <ref type="table">1</ref> lists the estimated coefficients for the covariates in LR1. We refer to the tree model without real-time information as TR1, which had hyperparameters cp = 0.01 and maxdepth = 7. Figure <ref type="figure">2</ref> illustrates the structure of TR1. TR1 performed similarly to LR1 on the test set and achieved test RMSE of 16.644 and test MAPE of 9.353%.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Models with Real-Time Information</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Linear Regression</head><p>We referred to the linear regression model with real-time information as LR2. It contained the following predictors: day vs. night, day of the week, season, holidays, weather, the total number of arrivals 1 and 7 days ago, the moving average of daily arrival count over the last 30 days, Google trends for "flu", "respiratory infection", "depression", "heart attack", "abuse", "weather", and "hospital" , and the average daily numbers of patients with comorbidity "HP" (hemiplegia or paraplegia), "CANC" (cancer), and "REND" (renal disease) over the last 3 days. LR2 achieved a test RMSE of 15.366 and a test MAPE of 9.109%. Table <ref type="table">1</ref> lists the estimated coefficients for the covariates in LR2.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Regression Tree</head><p>We referred to the tree model with real-time information as TR2, which had hyperparameters cp = 0.01 and maxdepth = 7. Note that the model trained without vs. with real-time predictors (TR1 vs. TR2 (Figure <ref type="figure">2</ref>)) were identical.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>XGBoost</head><p>The XGBoost model had the following hyperparameters: number of boosting rounds (num_round) equal to 180, (ii) maximum tree depth for base learners (max_depth) equal to 3, (iii) boosting learning rate (eta) equal to 0.1, (iv) L1 regularization term on weights (alpha) equal to 0.2, and (v) L2 regularization term on weights (lambda) equal to 0.8. Figure <ref type="figure">3</ref> illustrates the top 20 most informative predictors identified by the selected model, including day vs. night, day of the week, month, holidays, weather, Google trends for "respiratory infection", "disorder" and "weather", the daily average number of patients with comorbidity "AIDS" (acquired immunodeficiency syndrome) over the last 3 days, and the percentages of patients with comorbidity "CEVD" (cerebrovascular disease) over the last 3 days. The final model achieved a test RMSE of 16.315 and a test MAPE of 9.582%.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>SARIMA and SARIMAX</head><p>Among all SARIMA models, SARIMA(6,0,7)(7,1,3)14 was selected, achieving a test RMSE of 15.501 and a test MAPE of 8.817%. After incorporating the external regressors and setting the seasonal term to 0, the final ARIMAX(3,1,4) model achieved a test RMSE of 14.656 and a test MAPE of 8.703%. Table <ref type="table">1</ref> lists the estimated coefficients in the ARIMAX(3,1,4) model. As expected, the coefficients for the exogeneous covariates had the same signs (i.e., directional trends) as those for the final linear regression model (LR2). Moreover, as explicitly derived in Appendix B, the coefficients suggested a positive correlation between the arrival count during the current shift and the arrival counts during the previous two days.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Comparison of Different Prediction Models</head><p>For each prediction model examined, Table <ref type="table">2</ref> summarizes the RMSE and MAPE on the training and test sets, and Table <ref type="table">3</ref> lists the RMSE and MAPE associated with over-prediction and underprediction instances. Among models that did not utilize real-time information, the linear regression model (LR1) performed the best on the test set. After incorporating real-time information, the prediction accuracy on the test set can be improved. ARIMAX achieved the best performance among models that utilized real-time information, improving prediction accuracy from LR1 by 10.770% (in test RMSE) and 9.598% (in test MAPE). LR2 achieved the second best performance, with 6.630% reduction in test RMSE and 5.381% reduction in test MAPE compared to LR1.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>LIMITATIONS</head><p>Limitations of the study include the limited amount of training data. The training set only contained one year of data with 730 observations, which limited the performance of more sophisticated models that required substantial hyper-parameter tuning such as XGBoost. In addition, our study was performed for a single quaternary care facility in New York City. A meaningful extension is to apply our analysis to multiple ED sites and compare the prediction accuracy and trends. That said, the directional and structural insights (e.g., procedures to develop prediction models, and the value of real-time information) should be valid across facilities.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>DISCUSSION</head><p>Our work employed rich real-time information to build prediction models for ED demand which can be an integrated part of the two-stage nurse staffing framework. Existing studies have applied different prediction techniques to forecast ED arrivals, but have not explored as comprehensive real-time information as in our study. <ref type="bibr">35</ref> By exploring a novel large set of realtime predictors from the concurrent patient electronic health records, weather, and Google trends, we demonstrated that this real-time information was able to improve demand forecasts compared to base prediction models. The improvement in prediction accuracy can be used to develop prediction-driven two-stage staffing policies to improve operational efficiency.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Non-Inferiority of the "Tried-And-True" Prediction Models</head><p>As illustrated by Tables <ref type="table">2</ref> and<ref type="table">3</ref>, LR2 and ARIMAX achieved the best performance among all prediction models that utilized real-time information, improving prediction accuracy by 5%-11% in RMSE and MAPE compared to models without real-time information (LR1). The worse performance of the regression tree and SARIMA models was well expected due to their relatively simple structure, e.g., the SARIMA models only took previous arrival counts into account. On the other hand, the performance of the more advanced XGBoost model could be impeded by overfitting, e.g., the XGBoost model was trained with 128 features on 730 observations (shifts) only. The XGBoost model also had the disadvantage of lacking interpretability, which was especially concerning in healthcare settings due to the high-stakes decision making. Hence, by establishing the non-inferiority of the "tried-and-true" linear regression and time series models (embedded with exogeneous variables), we provided the foundation for ED managers to deploy more interpretable models.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Relevant Real-Time Information in Predicting ED Demand</head><p>Among the extensive amount of real-time information examined, only a few real-time predictors had predictive power and were coherently identified by different prediction models. According to the estimated coefficients by LR2 and ARIMAX (Table <ref type="table">1</ref>), ED arrivals were positively correlated with the patient volume 1 day and 7 days prior. Severe weather such as snow, precipitation, and extremely cold or hot temperature could reduce ED arrivals. Nevertheless, the ED tended to see more patients on days with strong wind. In addition, ED arrivals increased during the weeks when there were more Google search records for "flu". Intuitively, the search volume for "flu" could be seen as the concurrent flu trend information. Moreover, the total number of patients with a history of cancer (CANC) over the last 72 hours was positively correlated with ED arrivals. This trend could be corroborated by the findings that patients with higher weighted sum of CCIs were more likely to return to the ED within 72 hours. <ref type="bibr">[33]</ref><ref type="bibr">[34]</ref> The selected XGBoost model identified similar significant predictors (Figure <ref type="figure">3</ref>), with several new features such as the Google trends for "disorder", the percentages of patient with comorbidities of cerebrovascular disease (CEVD) and acquired immunodeficiency syndrome (AIDS) over the last 3 days.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Implication for Prediction-Driven Staffing</head><p>The development of accurate prediction models for ED demand was an integrated part of our efforts in utilizing predictive analytics to facilitate better medical resource planning. As mentioned before, ED staffing generally involves two stages: a base stage, which takes place weeks to months ahead of the actual shift, and surge stage, which happens days to hours before the shift starts. The base prediction model without real-time information can be used to guide the base staffing decision, while the more sophisticated prediction model with real-time information can be used to guide surge staffing decisions. At the base stage, the staffing cost is lower and more preferrable by nurses due to consistency and predictability of work hours. However, the accuracy of the prediction model may be low. On the other hand, at the surge stage, the staffing cost is higher, but more accurate prediction of patients' demand is available. How to optimally balance the tradeoff depends on how much real-time predictors improve prediction accuracy over the base prediction. Our results provide important quantification of this, which can be incorporated into the two-stage staffing framework developed in Hu et al. 2022 <ref type="bibr">27</ref> to reduce the staffing cost and ED waiting times. We note that even relatively small prediction accuracy improvement, i.e., 5%-11% as found in our study, can lead to significant cost savings, 11%-16% as demonstrated in Hu et al. 2022. <ref type="bibr">27</ref> Lastly, we remark that alternative prediction targets other than shift-level arrival counts could be used in the prediction-driven staffing framework. In Appendix C, we constructed logistic regression models to predict "outlier" shifts that would have demand surges, and obtained similar insights on the value of real-time information. That said, predicting shift-level arrival counts (compared to a binary indicator on whether there would be demand surge) led to more actionable staffing implications.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>CONCLUSION</head><p>We constructed and evaluated predictions models with rich real-time information to forecast ED patient volume. In alignment with the nursing shift structure in an ED site at a quaternary care facility in New York City, we aimed to predict the shift-level patient arrival count. Various prediction techniques were examined, including linear regression, regression tree, XGBoost, SARIMA, and (S)ARIMAX. Based on the data from our partner ED site, linear regression and ARIMAX when combined with real-time information achieved the highest prediction accuracy measured by RMSE and MAPE. Comparing to prediction models without real-time predictors, we found that contemporary information was able to improve prediction accuracy in near-real time. Among the extensive list of real-time predictors tested, recent patient arrival counts, weather, Google trends, and concurrent patient comorbidity information had the highest predictive power. The effectiveness of real-time information in improving demand forecast has policy implications for staffing. ED management can utilize real-time demand forecast to make timely adjustments to staffing levels, which in turn can effectively mitigate ED overcrowding.  we performed the standard stepwise selection procedure and identified the significant covariates in each subset. Covariates that were highly correlated (with correlation coefficient larger than 0.5) were sifted out by comparing the correlation matrix.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Regression Tree</head><p>Regression tree model was implemented via the rpart package in R. <ref type="bibr">37</ref> The following hyperparameters were tuned: (i) complexity parameter (cp) ranging from 0 to 0.08 in increment of 0.01, and (ii) maximum depth of any node of the final tree (maxdepth) ranging from 1 to 10 in increment of 1. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>SARIMA and SARIMAX</head><p>To express the SARIMA model explicitly, we let B be the backward shift operator, where &#119861; 1 &#119910; 2 = &#119910; 2&amp;1 , &#119895; = 0, &#177;1, &#8943;.</p><p>In the equation above and hereafter, the subscript t is a time index for each shift. We define the related operators</p><p>where &#120601;(&#119861;) is the non-seasonal AR polynomial, &#934;(&#119861;) is the seasonal AR polynomial, &#120579;(&#119861;) is the non-seasonal MA polynomial, &#920;(&#119861;) is the seasonal MA polynomial, &#916; is the non-seasonal difference operator, and &#916; 4 is the seasonal difference operator. A SARIMA(p,d,q)(P,D,Q)s model can be formally written as &#120601;(&#119861;)&#934;(&#119861;)&#916; 8 &#916; 4 9 &#119910; 2 = &#120579;(&#119861;)&#920;(&#119861;)&#120598; 2 &#120598; 2 a noise term that follows a normal distribution with mean 0 and standard deviation &#120590;.</p><p>The ARIMAX(p,d,q) model combines the model (where seasonal hyperparameters (P, D, Q, s) are set to 0) and a linear regression model with external regressors.</p><p>Let &#119909; 2 be the vector of covariates in the linear regression model, and &#119909; 2 : be its transpose. Let &#120573; be the vector of coefficients for the covariates. Then a ARIMAX(p,d,q) model can be formally represented as &#120601;(&#119861;)&#916; 8 &#119910; 2 = &#119909; 2 : &#120573; + &#120579;(&#119861;)&#120598; 2 .</p><p>To train the SARIMA model, we set the seasonal term to 14 (i.e., s = 14) to distinguish the day vs. night and day of the week effects. In addition, since the time series had a stationary increasing trend (Figure <ref type="figure">1</ref>), it was reasonable to conduct a difference for the original series.</p><p>However, whether to conduct the difference directly (i.e., setting d = 1, D = 0) or seasonally (i.e., setting d = 0, D = 1) needed to be determined. For each of these two options, we conducted the Dickey-Fuller test to check whether the differenced time series was stationary. The resulting pvalues were both 0.01, which suggested at 99% confidence level that the differenced time series under each option did not have a unit root and was therefore stationary. We then used a variation of the Hyndman-Khandakar algorithm <ref type="bibr">39</ref> to determine the hyperparameters. In particular, for each differencing method, we varied the AR term (p), the seasonal AR term (P), the MA term (q), and the seasonal MA term (Q) from 1 to 7 in increment of 1. We considered models where the highest-order AR and MA terms were statistically significant. The final model was then selected based on AIC on the training set and evaluated on the test As for the SARIMAX model, we used the same covariates in the selected linear regression model (LR2) as external regressors, except that we excluded the previous-day arrival count in the covariates to avoid double counting the recent arrival counts information. Since the embedded linear regression model already took into account the day vs. night and day of the week variations, as well as the arrival counts over the last 7 and 30 days, we set the seasonal hyperparameters (P, Q, s) in the SARIMAX model to 0 to avoid overfitting, which leads to an ARIMAX model.</p><p>For </p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="13" xml:id="foot_0"><p>In addition to utilizing appropriate prediction techniques, it is important to identify what information is most relevant in predicting ED demand, especially because vast amount of information is now made available by electronic health records and various other data sources.</p></note>
		</body>
		</text>
</TEI>
