In early 2020, the COVID-19 outbreak reached the level of a global health emergency [1]. In response, scientists worldwide developed epidemic models to project the trajectory of the disease, evaluate control measures, and guide policy decisions. These models varied in their approaches, assumptions, data sources, distribution of key parameters, and consequently in their predictions of disease progression and optimal policies [2][3][4]. They also differed in respect to how human behavior changes during the pandemic is modeled. Prior to the pandemic, several studies highlighted the importance of incorporating human behavior into epidemic models with realistic assumptions [5][6][7][8][9][10][11]. Yet, during the pandemic, many models did not fully incorporate human behavior change, often assuming human behavior change is exogenous to the model. Now, increasing attention is being paid toward capturing the feedback loops between disease spread and human behavior to provide more accurate predictions and effective policy suggestions [12][13][14].
A common approach to modeling feedback loops between human behavior and disease spread is to focus on a single aspect of behavior, typically human risk response, as the primary link between human behavior and disease spread. The feedback loop is often termed riskresponse or adaptation [12,15], and while it is mathematically formulated in different ways, the concept simplifies the complex interplay of behavioral factors. However, many other behavioral mechanisms influence disease spread, such as accumulated psychological and economic fatigue from prolonged adherence to non-pharmaceutical interventions (NPIs), among others. Previous studies have identified a range of behavioral factors influencing disease dynamics [8,9,11,13,[16][17][18] with possible implications for epidemiological modeling. Considering the extensive behavioral dynamics in the literature, important questions emerge: To what extent and in what ways should behavioral detail be incorporated into epidemic models to enhance predictive accuracy? Are there specific behavioral factors that explain greater variations in deaths and cases compared to others? Identifying such factors will be invaluable to modelers by improving model accuracy while maintaining parsimony, making models simple and explainable to stakeholders and the public, and advancing a systemic understanding of the influence of behavior during pandemics [19,20].
Additionally, despite the surge in real-time behavioral data during the pandemic, many models still rely solely on disease data for the purpose of validation [21]. Specifically, models mostly replicate disease data without replicating human behavior data (such as mobility), and model fidelity is often assessed only against variables such as cases or death. Simultaneously replicating disease and human behavior data in a single model is a challenging endeavor, but it is necessary to validate the behavioral mechanisms of coupled behavior-disease models. These limitations hinder our understanding of the feedback loop between behavior and disease spread and may potentially lead to less accurate forecasts.
To address these gaps, we propose a more comprehensive model that captures various behavioral phenomena that drive disease dynamics. The model allows for simultaneous replication and prediction of both disease and behavioral data (deaths, policy stringency, adherence fatigue, and mobility) across eight countries (Denmark, Germany, Hungary, Italy, Sweden, the United Kingdom, and the United States). Additionally, we estimate parameters for both disease and behavioral dynamics. Notably, all behavioral and policy changes arise endogenously, enabling the isolation of feedback loops driven by each behavioral factor. This allows us to test which feedback loops significantly improve the model's ability to replicate both overall data and mortality data compared to simpler models that lack these feedback mechanisms.
Before vaccine availability, controlling COVID-19 relied heavily on NPIs to curb transmission [21]. This involved increased mask-wearing and hygiene practices, but most significantly, reductions in mobility and social distancing to minimize infectious contacts and transmission. These actions, a combination of voluntary public efforts and mandated public health policies, significantly curbed disease spread until widespread vaccination became possible. The global scale of these responses was unprecedented. Mask mandates and hand sanitizer use became commonplace, often with consequences for non-compliance. Many countries implemented nationwide lockdowns, closing schools, workplaces, businesses, and travel, leading to economic and social disruptions [22]. While the public understood the severity of the pandemic and the necessity of NPIs, their adherence came at an excessive cost [23].
Therefore, studying the impact of voluntary responses or government policies in isolation is insufficient to capture the complex interplay of behavior during the pandemic. We need to analyze the interaction of these factors alongside other significant behavioral responses. From the vast literature, we focus on four key responses explained in detail subsequently.
Most modelers refer to risk-response dynamics as the most intuitive behavioral mechanism during a pandemic: Higher numbers of cases or deaths trigger protective actions like social distancing, which with a delay, decrease transmission rate [24]; this decline in transmission, and consequently in perceived risk, encourages riskier behaviors leading to a resurgence of the disease. The significance of this feedback loop has been extensively studied before COVID-19, with various approaches documented in literature [8][9][10]. Similarly, several COVID-19 studies have incorporated this feedback loop to make forecasts and estimate underreporting [18,[25][26][27], quantify the effects of human behavior to evaluate optimal control strategies [28,29], and understand the drivers of multiple epidemic waves [30][31][32]. Rahmandad and colleagues [33] provide evidence on the importance of formulating public risk response by studying the performance of >60 models used by the Centers for Disease Control and Prevention (CDC) as the COVID-19 forecast hub.
Beyond individual choices, government policies also shape behavior during pandemics by mandating public actions to contain disease spread. The effectiveness of these measures has been widely studied, with a consensus that effective implementation of containment policies can reduce disease spread by limiting infectious contacts [34][35][36]. The majority of models set a boundary that excludes factors that trigger government response. Typically, these studies treat government responses as external factors or data inputs, neglecting the dynamic nature of policymaking [21]. However, government responses are not static but adapt to perceived risk. Governments implement stricter measures during periods of high cases or deaths and relax policies in periods of low disease severity. While individuals focus on personal risk, governments must balance broader factors like economic impact, social disruption, and public health when making policies [21]. Thus, their perception of risk differs from the public's risk assessment. Rarely do modelers formulate changes in policies in epidemic models, though new studies show that endogenous representation of policy changes is critical in understanding variations in death rates across countries [37]. Therefore, we incorporate a separate feedback loop for government response in our model. This loop reflects how changes in death rates trigger adjustments in policy stringency, which in turn affects contact rates and disease dynamics.
During the COVID-19 pandemic, researchers observed a decline in public adherence to NPIs over time, often termed "adherence fatigue" or "pandemic fatigue" [23]. This stemmed from the high personal and societal costs associated with sustained adherence. A multi-country study [38] shows that pandemic fatigue gradually emerges over time, intensifying in prolonged periods of stringent interventions and waning with increasing deaths. Moreover, analysis of mobility and fatality data in the United States showed that mobility increased amid rising deaths, a pattern only possible in models incorporating both risk response and adherence fatigue [31]. This finding is further supported by Atkeson [39], who showed that seasonality and public response to death rates alone were insufficient to explain the second wave of deaths without considering "pandemic fatigue" in the United States and the United Kingdom. Additionally, Petherick et al. [40] showed that low-cost behaviors like mask wearing saw a steady increase in adherence, while high-cost behaviors like physical distancing declined in multiple countries. This evidence suggests that adherence fatigue was a dominant behavioral response during COVID-19, and its inclusion in epidemic models might be necessary for accurate long-term forecasts. However, existing models rarely consider its impact on government response. Governments face pressure to mitigate disruptions to daily life and economic decline caused by stringent measures. As a result, they become less willing to implement stricter policies over time for the same level of perceived risk. Therefore, our model incorporates the effects of adherence fatigue on both public compliance with NPIs and government response.
Early in the pandemic, researchers observed strong correlations between mobility restrictions and disease transmission, suggesting the effectiveness of strict lockdown policies [41]. However, subsequent analysis revealed a weakening of these correlations over time. This indicated that epidemic control is possible with smaller mobility restrictions if lower cost behaviors like mask wearing, and social distancing are widely adopted [42][43][44]. In this study, societal learning represents society's ability to learn to reduce disease transmission without relying solely on mobility restrictions [45,46]. This is crucial because strict policies contribute to adherence fatigue, hindering long-term compliance with public health interventions [38,40]. Hence, we incorporate this factor into our model to test its effect on model forecasts.
While some studies have incorporated multiple behavioral responses, none have addressed all four simultaneously, and none have attempted to test the fidelity of their models against behavior data. For instance, most models do not include adherence fatigue, and many others do not consider that societal learning affects mobility patterns over time. Even one of the most comprehensive models of the pandemic that includes a variety of behavioral feedback loops and is validated against disease data from 92 countries [26], leave uncertain the extent to which the model replicates behavioral data. This limitation should be addressed, as research shows that even small amounts of behavioral data can significantly improve accuracy of disease forecasts in behavioral epidemic models [47].
The limited effort to replicate behavioral data could previously be related to lack of such data. Historically, epidemic models relied primarily on disease and non-real-time mobility data, and scientists called for real-time behavioral data to improve disease forecasts [9]. The COVID-19 pandemic spurred collaborative efforts to develop such data. Technology companies provided openly accessible mobility data, and social scientists developed surveys measuring public risk perception and compliance [38,[48][49][50]. The Oxford Blavatnik School created valuable indexes measuring government policy stringency [22]. Despite this wealth of new behavioral data, many COVID-19 models failed to adequately integrate them. Many researchers used the data as an input to their models, i.e. to exogenously model the historical trend of the disease. Consequently, while we possess ample behavioral data, progress in validating theoretical frameworks for the disease-behavior feedback loop remains slow [21].
Motivated by these gaps, we develop a holistic model that not only captures the interaction of multiple behavioral factors but also replicates disease and behavioral data simultaneously. Our model adopts an endogenous approach to provide a more nuanced and realistic representation of the underlying mechanisms driving behavior. However, integrating additional feedback loops and data can make parameter estimation challenging and potentially lead to overfitting and unreliable predictions. Therefore, we prioritize model parsimony, aiming for the simplest model that captures these behavioral responses. By prioritizing parsimony while incorporating diverse behavioral factors and data, we aim to develop a model that offers a more accurate and robust understanding of disease dynamics during pandemics.
We construct an ensemble of datasets from multiple sources to include both disease and behavior related indicators. Specifically, we use time series data on confirmed COVID-19 deaths from Our World in Data (available at: https://ourworldindata.org/covid-deaths). Google Community Mobility Reports [48] track changes in mobility across various categories of locations compared to pre-pandemic baselines. The Oxford Covid-19 Government Response Tracker (OxCGRT) [22] reports government interventions through 24 policy indicators. We make use of the mean score of 9 containment and closure policy indicators that are more closely related to public interactions (school closures; workplace closures; cancellation of public events; restrictions on public gatherings; closures of public transport; stay-at-home requirements; public information campaigns; restrictions on internal movements; and international travel controls), where higher scores indicate stricter government response. Data on adherence fatigue is drawn from longitudinal and panel surveys conducted by Jørgensen et al. [38] between September 2020 and July 2021. The study defines and measures adherence fatigue (termed "pandemic fatigue" in their study) as individuals' willingness to sustain compliance with pandemic restrictions. The validity of this measure is thoroughly discussed within the study. Data on adherence fatigue is available for eight countries, which limits our study to these locations.
We rely on death data rather than case data to represent disease spread, as COVID-19 cases are significantly undercounted, and detection rates vary widely across and within countries over time [51][52][53]. This is due to substantial changes in testing capacity, and consequently in ascertainment rates, as the pandemic progressed. Therefore, death counts offer a more reliable measure. Many studies have utilized this approach [31,33,39].
Fig. 1 illustrates four specific trends for eight countries. This includes the progression of COVID-19 deaths, mobility, policy stringency, and adherence fatigue in each country. Some general patterns are observable. For example, during the first wave, mobility sharply decreases as deaths increase and later rebounds as deaths decline. In subsequent waves, these trends are less pronounced. Policy stringency similarly increases sharply with rising deaths but declines more slowly. Adherence fatigue gradually rises over time and with policy stringency but changes in adherence fatigue occur at a slower rate compared to other behavioral measures. These observations guide the mathematical formulations used in developing our model. We aim at developing a model that can replicate all data series for each country simultaneously.
The model comprises two sectors: the physical sector, simulating disease spread, and the behavioral sector, simulating how people adjust their contacts in response to disease severity. This behavioral response, in turn, influences disease spread, creating a feedback loop between disease transmission and behavioral changes.
A high-level representation of our model is depicted in Fig. 2. Arrows indicate causality, while boxes represent compartments (or stocks). Arrows with valves show flows into and out of these compartments. A positive feedback loop (reinforcing), marked by a positive loop sign, amplifies variable values, potentially leading to exponential growth. For example, everything else constant, more infected individuals increase exposure and infection rates, further increasing the infected population. A negative feedback loop (balancing) is a mechanism that regulates activities around a limit or equilibrium. For instance, more deaths can raise public risk perception, leading to a lower contact rate, which lowers transmission rates, and ultimately decreases infections and deaths. In the next sections, we review each sector of the model.
The physical sector of our model is a simple SEIR model. From the left, people move from the susceptible state (S) to the exposed state (E), staying in this state for an average period of τ 1 days. Individuals in the exposed state do not show any major symptoms and cannot transmit the disease. The 'Infection' positive loop creates the initial exponential growth in the number of cases as individuals move to the infected state (I) after the exposure period. They remain in this state until they are eventually removed (R) after an average period of τ 2 days.
Eqs. (1-4) depict the main structure of the model.
(3)
The infectivity rate, β, is defined as the product of three factors, as shown in Eq. ( 5). The baseline infectivity rate, β 0 , represents the initial rate when contact frequency remains at the pre-pandemic, normal level. Variations in contact rate over time are captured by the multiplier 0 < k ≤ 1, which is a function of behavioral responses during the pandemic. Finally, seasonal effects on disease transmission are represented by the multiplier w, included based on strong evidence of seasonality patterns in COVID-19 infection [54]. In alignment with other modeling studies [26,37], we use independent, published estimates for the magnitude of this seasonal effect [54].
As daily deaths change, public risk perception influences contacts. The government (or public health officials) also perceive the risks and react through the stringency of containment policies, which then influence public behavior. On the other hand, government response and compliance with NPIs decline as the public becomes tired of restrictions. The contact rate is modeled as the combination of the effect of public risk perception on contact k 1 , the effect of policy stringency on contact k 2 , and the effect of adherence fatigue on contact k 3 (Eq. ( 6)).
We obtain daily COVID-19 deaths (f) by estimating the infected fatality rate (i) for each country, assuming that the period of infectiousness is the same for recovery and death (Eq. ( 7)). Changes in deaths lead to changes in public risk perception (f ʹ p ) with an associated information delay representing the time it takes for information to reach the public and for the public to respond (λ p ). Psychological studies indicate that people respond differently to increasing versus decreasing loss or risk [55,56]. This can imply that perceiving risk and adjusting behavior may happen more quickly as death rates rise than when death falls. Accordingly, we apply two distinct delay parameters to estimate risk perception, λ H p and λ L p , allowing the model to vary delay lengths during periods of rising versus declining risk. We model this using a first-order, exponential smoothing function that assigns greater weight to recent deaths (Eqs. (8 and 9) [57]. The effect of public risk perception on contact (k 1 ) is a decrease as risk perception rises, with the rate of this decrease dependent on the public's sensitivity to death (α p ) (Eq. 10). The time for public to perceive risk (λ p ), and public sensitivity to death (α p ) parameters are estimated for each country.
We model government risk perception (f ʹ g ) similarly to public risk perception (f ʹ p ) with two delay periods (λ H g , λ L g ) for adjusting government perceived risk to reflect that governments respond differently to increasing and decreasing risk (Eqs. 11 and 12). The effect of government risk perception on policies (j 1 ) is modeled as a gradual increase in stringency as government risk perception rises (Eq. ( 13)). The rate of this increase is determined by the government's sensitivity to death (α g ). However, various countryspecific factors impose limits that prevent governments from attaining the strictest policies (where stringency = 1). Incorporating these factors endogenously would overly complicate the model. Instead, we define policy stringency (y) as the effect of government risk perception (j 1 ) constrained by a stringency limit (ŷ) (Eq. ( 14)). This approach creates a continuous measure of policy stringency. Although, in general, a single policy may not change gradually, the continuous measure for stringency reflects the aggregation of various policy initiatives that might be activated at different times and across regions within each country. We model the effect of government response on contact, (k 2 ), inversely related to policy stringency, with the rate of this decrease determined by an effectiveness parameter (ρ) (Eq. ( 15)). We estimate parameters for the delay periods in government risk perception (λ H g , λ L g ), government sensitivity to death (α g ), stringency limit (ŷ), and the effectiveness parameter (ρ) for each country.
3.2.2.3. Societal learning feedback loop. Societal learning represents individuals' tendency to adopt cost-effective strategies to deal with the pandemic. Specifically, over time, individuals recognize that compliance with public health measures-such as masking-enables greater mobility while mitigating risk. To represent the cumulative experience of the pandemic, we use the removed population (R) variable within the SEIR framework, which tracks the total number of cases since the pandemic's onset. We define a learning rate, (l), steadily increasing at a rate of gamma (γ) as the removed population (R) grows (Eq. 16). The goal is to reach a value of 1, signifying that all possible learning has been achieved and the reduction of infectious contacts is achieved without any restrictions on mobility. We estimate gamma (γ) for each country.
Consequently, we derive the equation for mobility (m) as a linear function of the learning rate and contact rate such that mobility equals contact rate when learning is 0 and is independent of contact rate (k) when learning is 1 (Eq. 17).
)
.2.2.4. Adherence fatigue feedback loop. Quantifying costs of adherence to containment measures endogenously is complex because it encompasses several areas such as economic decline, job losses, psychological tolls, and reductions in quality of life due to illnesses. Recognizing that stricter policies with mobility restrictions are more disruptive, we use reductions in mobility compared to pre-pandemic levels as a proxy for life disruptions. Therefore, adherence fatigue (a) is represented as a function of the normal mobility lost (1 -m) and a fatigue rate (ϕ) estimated for each country. The fatigue rate (ϕ) signifies how swiftly each country accumulates fatigue. Statistical analysis suggests that adherence fatigue accumulates gradually over an extended period [38]. This is attributed to delays in perceiving the adverse effects of reduced mobility and reassessing adherence to containment policies. As such, we model adherence fatigue as the outcome of a third-order delay process (Eq. 18), and we estimate the delay period for fatigue accumulation in each country, (λ a ).
The effect of adherence fatigue on contact (k 3 ) is a gradual decrease as adherence fatigue increases, with the rate of this decrease dependent on the public's sensitivity to fatigue (υ a ) (Eq. 19). Similarly, the effect of adherence fatigue on stringency (j 2 ) is a gradual decrease as adherence fatigue increases, with the rate of this decrease dependent on the government's sensitivity to fatigue (υ g ) (Eq. 20). Consequently, we redefine policy stringency (y) as the combined effects of death (j 1 ) and adherence fatigue (j 2 ) on stringency constrained by a limit ŷ (Eq. 21).
The model is calibrated using data from early 2020 through December 2020, as pivotal elements like variants and vaccination become more significant beyond this period. Two disease parameters (incubation period and infection period) are set based on prior literature, leaving 15 unknown parameters for each country. We implement weighted least squares calibration using Vensim DSS's optimization function, seeking parameter values that minimize the sum of squared errors between model output and data [58,59]. Details of the parameter estimation method are available in the Supplementary Information File.
Model validation includes in-sample and out-of-sample tests. It is performed by quantifying the errors between simulation outputs and data using the Root Mean Squared Error (RMSE) for both the in-sample period (early 2020 through December 2020) and the out-of-sample period (January 2021-April 2021).
We assess the relative importance of behavioral feedback loops on model performance by testing whether smaller models excluding some feedback loops can achieve comparable performance. We achieve this by removing the effects of specific behavioral responses, resulting in smaller models that still include all data variables.
The following nested models are considered:
1) No societal learning: In this model, the learning rate remains at 0 throughout the pandemic. 2) No adherence fatigue effect: In this model, adherence fatigue does not have any effect on contact rate and policy stringency. Thus, contact rate (k)= effect of public risk perception on contact (k 1 ) * effect of policy on contact (k 2 ).
have no effect on contact rate. Thus, contact rate (k)= effect of public risk perception on contact (k 1 ) * effect of adherence fatigue on contact (k 3 ). 4) Public risk perception only: This model assumes that public risk perception is the sole driver of behavioral change, excluding the effects of adherence fatigue, government response, and societal learning. Thus, the learning rate remains at 0 and contact rate (k)= effect of public risk perception on contact (k 1 ).
For each model, we repeat the estimation procedure using all available data, subsequently generating simulated outcomes for each based on the respective estimated parameter values.
In a separate set of analyses, we focus on the specific task of predicting only deaths as a baseline for comparison. We assess whether alternative models, which exclude some of the behavioral feedback loops, are sufficient for predicting only deaths. Model calibration is performed using only deaths data.
The subsequent alternative models are considered:
1) No societal learning feedback loop: In this model, the societal learning feedback loop is deemed unnecessary for accurately replicating deaths. Therefore, the learning rate is set to 0 throughout the pandemic. 2) No adherence fatigue feedback loop: In this model, the adherence fatigue feedback loop is deemed unnecessary for accurately replicating deaths. Thus, there is no need to incorporate mobility changes using the societal learning feedback loop. Therefore, the learning rate and fatigue rate are set to 0 throughout the pandemic and contact rate (k)= effect of public risk perception on contact (k 1 ) * effect of policy on contact (k 2 ).
In this model, having a separate feedback loop for government response is deemed unnecessary for accurately replicating deaths. Therefore, government sensitivity to death is set to 0 throughout the pandemic and contact rate (k)= effect of public risk perception on contact (k 1 ) * effect of adherence fatigue on contact (k 3 ). 4) Public risk perception only: This model deems only the public risk perception feedback loop as necessary for accurately replicating deaths. Therefore, the learning rate, fatigue rate, and government sensitivity to death are set to 0 throughout the pandemic and contact rate (k)= effect of public risk perception on contact (k 1 ).
There are no behavioral feedback loops in this model. The contact rate (k) has a constant value of 1 throughout the pandemic. 6) SEIR with no behavior and exogenous contact rate: There are no behavioral feedback loops in this model. However, values for the contact rate (k) are estimated for each month in the in-sample period using deaths data.
Model performance is evaluated using two metrics: the Akaike Information Criterion (AIC) for in-sample comparison and the Root Mean Squared Error (RMSE) for out-of-sample predictive accuracy.
The AIC balances how well the model fits the data (goodness of fit) with its complexity (number of parameters). Eq. (22) shows the AIC formula. The first term in the equation measures how well the model fits the data, while the second term penalizes the model for having more parameters. Therefore, a lower AIC score means the model is both a good fit and less complex, which is better for prediction [60,61].
where θMLE is the vector of maximum likelihood estimates of the model parameters and K is the number of parameters.
In ordinary differential equation (ODE) models, it is difficult to calculate the negative of the log-likelihood. Instead, we follow the approach described in Portet (2020) [62] where the negative of the log-likelihood is replaced by the residual sum of squares, as shown in Eq. (23).
where N is the number of observations, K is the number of parameters + 1, and RSS is the sum of squared differences between data and simulation.
The model with the minimum AIC value is the best model. We also calculate Akaike weights for each model (Eqs. 24 and 25). These weights tell us the probability of each model being the best given the data and the set of models tested. A good model to choose should have a weight above 0.9. Finally, we compare the change in AIC between different models and in relation to the comprehensive model to ascertain strength of evidence for individual models against others.
4. Results
This section describes the comprehensive model's performance under two calibration scenarios: using all available datasets (deaths, mobility, policy stringency, and adherence fatigue) and using only deaths data. Detailed reports on the estimated parameters, including those adopted from existing literature, are available in the Supplementary Information File. The performance metrics for all eight countries included in the analysis can also be found in the Supplementary Information File.
Fig. 3 compares simulated deaths from the comprehensive model for both calibration scenarios.
In-sample period. The first scenario (all datasets) effectively replicates the data for all countries, with an average RMSE of 1.52 (0.73). This is noteworthy considering the model's simplified physical sector, which does not account for factors like asymptomatic transmission, testing, and underreporting in deaths data.
The second scenario (deaths data only) achieves even better fit to data, with the average RMSE falling to 1.00 (0.63). This implies that for replicating historical deaths alone, calibrating solely using death data might be sufficient.
4.1.1.6. Out-of-sample period. Model performance weakens for both scenarios, with an average RMSE of 4.30 (3.13) for the first scenario and 4.40 (4.71) for the second scenario. However, the first scenario yielded better results, indicating that using both disease and behavior data for calibration leads to more accurate predictions of deaths. We note that the model performs poorly in specific countries. It failed to detect the resurgence of deaths in Hungary and underestimated the size of the second wave of the pandemic in the United Kingdom. This points to the limitations of the comprehensive model, which excludes several important factors influencing disease progression. Nonetheless, the results show that calibration with both deaths and behavioral data enhances prediction accuracy in the out-of-sample test compared to using deaths data alone.
Fig. 4 compares simulated adherence fatigue from the comprehensive model for both calibration scenarios.
In-sample period. The first scenario (all datasets) excels at replicating adherence fatigue, with a low average RMSE of 0.02 (0.01). This performance surpasses that for deaths, which is expected given the smaller variations in adherence fatigue compared to deaths.
The second scenario (deaths data only) yields a very poor performance, with simulated outcomes deviating significantly from data across all countries except Denmark. The average RMSE using only deaths data is 0.76 (0.64). This highlights the detrimental effect of omitting fatigue data from calibration. 4.1.3.9. In-sample period. Similar to adherence fatigue, the first scenario (all datasets) effectively replicates mobility data for all countries, with a low average RMSE of 0.09 (0.02).
The second scenario (deaths data only) shows a poorer performance compared to the first. The average RMSE is 0.25 (0.09), showing the detrimental effects of excluding mobility data from calibration.
In the out-of-sample period, the first scenario performs well in predicting mobility, achieving an RMSE of 0.11 (0.03). Conversely, the second scenario has a higher average RMSE of 0.28 (0.11), further reinforcing the importance of using behavior data, as the omission of mobility data leads to worse results.
Fig. 6 compares simulated policy stringency from the comprehensive model for both calibration scenarios.
4. 1.4.11. In-sample period. The first scenario (all datasets) replicates the data effectively for all countries, with a strong average RMSE of 0.07 (0.02). Conversely, the second scenario (deaths data only) fails for the majority of the countries and achieves a poor average RMSE of 0.40 (0.23).
Here, the first scenario maintains its strong performance with an average RMSE of 0.11 (0.04), and the second scenario maintains its poor performance with an average RMSE of 0.50 (0.26). The results are consistent with the findings for adherence fatigue and mobility that excluding behavior data from model calibration worsens model performance.
Overall, the comprehensive model consistently achieves strong performance, both in-sample and out-of-sample. Calibration with both disease and behavioral data consistently outperformed calibration with only deaths data. While calibrating with only deaths data yielded good performance in replicating deaths, it resulted in significantly poorer results for behavioral data. This indicates that models relying solely on death data may accurately fit historical death trends but fail to efficiently estimate behavior parameters. As a result, the projection of human behavior by such models is likely to be erroneous. This stresses the importance of testing the fidelity of behavioral epidemic models in replicating both behavior and disease data.
This section compares the performance of the comprehensive model with alternative models that exclude specific behavioral feedback loops. The provided tables summarize model performance based on AIC and RMSE in the prediction period. Detailed tables containing RMSE values for prediction for each country can be found in the Supplementary Information File.
Table 1 compares model performance for different models calibrated using all available datasets. The comprehensive model, incorporating all behavioral feedback loops, achieves the lowest AIC value. This indicates superior performance compared to the alternative models. Additionally, its Akaike weight of 1 signifies it as the only plausible model given the data. Therefore, it is the most effective choice despite its increased complexity with more parameters and feedback loops.
The Akaike weights of 0 for all the alternative models suggest that none of the behavioral responses are inconsequential enough to be excluded from the comprehensive model without significantly affecting model performance. Furthermore, the change in AIC from the comprehensive model is notably larger in the model capturing only the effect of public risk perception on contact compared to the other alternative models. This shows that the public risk perception feedback loop alone is insufficient to replicate all datasets and there is a need for models with more detailed behavioral dynamics.
In addition, the comprehensive model has the lowest average error across all countries during the out-of-sample period. These findings emphasize the need for epidemic models to integrate multiple aspects of human responses to enhance the performance of behavioral epidemic models. A. Osi and N. Ghaffarzadegan Mathematical Biosciences 380 (2025) 109368
Table 2 compares model performance for models calibrated using deaths data alone. The SEIR model with exogenous contact rates achieved the smallest AIC value and is the only plausible model given the insample data. However, the exogenous model's prediction performance was significantly worse than all other models (average RMSE of 16.24 (23.9)) compared to the comprehensive model (average RMSE of 4.4 (4.7)). This poor performance indicates that the exogenous model is overfitted and is unable to replicate future trends when new data is introduced. Therefore, despite its AIC, the comprehensive model is a better choice due to its superior out-of-sample performance.
Similar to previous findings, removing any behavioral feedback loop is shown to be detrimental to model performance. This finding suggests that even if the goal is to predict deaths alone, a comprehensive, endogenous model incorporating multiple human behavior dynamics outperforms simpler behavioral epidemic models and exogenous models lacking behavioral dynamics.
This study presented a novel behavioral epidemic model, calibrated and tested using data from the recent COVID-19 pandemic in eight countries. The model integrates endogenous human responses, specifically focusing on public risk perception, government response, adherence fatigue, and societal learning. It utilizes both epidemiological and behavioral data, including deaths, mobility, policy stringency, and adherence fatigue, for parameter estimation and model testing. The relative importance of each behavioral feedback loop is evaluated by systematically comparing models that exclude individual loops with the full model.
The model accurately replicates historical trends in mortality and behavioral responses across multiple waves. Additionally, it outperforms simpler models that exclude certain behavioral feedback loops. These findings highlight the importance of developing endogenous, feedback-rich models that incorporate a broad range of human behavioral dynamics to improve forecasting accuracy during pandemics. Models relying on exogenous inputs to represent human behavior often fail in policy analysis and projections, as they lack insights into how people will respond to interventions or to the future trajectory of the disease. In contrast, an endogenous model formulates human response based on underlying theories about how individuals react to changes in the state of the disease. By enriching the model with a variety of important behavioral phenomena, the resulting feedback-rich model can more accurately adjust human responses and yield more reliable simulation outcomes.
Moreover, our results underscore the importance of testing models against a wide range of data series, from epidemiological indicators to public behavior, over a long period of time and in multiple regions. When model parameterization solely relies on epidemiological data, simulated behavioral trends show significant deviation from real-life observations. This finding shows that exclusive dependence on epidemiological data prevents the model from accurately capturing human response trajectories, which can be important for effective policymaking. These findings corroborate existing research that integrating disease and behavioral data enhances forecasting accuracy in behavioral epidemic models [47]. Furthermore, conducting analysis over a relatively long period and testing the model's ability to replicate patterns across multiple regions are essential steps to properly validate the model. This study contributes to the field of epidemiological modeling, particularly the growing literature of behavioral epidemic models. Our approach resonates with the recent calls for including human responses in epidemic models [8,9,13]. We build on studies that have coupled disease dynamics with societal risk response [27,28,30,33,39], introducing several additional behavioral mechanisms. Although some studies assume that risk perception is driven by prevalence rather than deaths [7,16], our model uses the latter as the key risk indicator influencing risk perception. In simple SEIR models, where deaths are proportional to active cases, this modeling choice has minimal impact. However, it may have greater implications in more detailed models [63]. Specifically, as the infection fatality rate declines over time due to improved medical treatments or the emergence of less fatal variants, people may not respond as strongly to rising prevalence as they once did. This favors assuming deaths as the key risk indicator.
Our model is in line with a previous comprehensive study by Rahmandad et al. [26] but takes a major step forward by simultaneously simulating and testing behavioral constructs. The model presented here shows the benefits of comprehensive modeling of change in human behavior. While the risk-response feedback, representing changes in human contact rate as a function of the state of the disease, is critically important [12], it is not the only mechanism affecting human decisions [8,11]. Adherence fatigue in long-term simulations becomes important as people accumulate economic and psychological costs which eventually affect their adherence to NPIs. This is a commonly observed phenomenon in many countries, sometimes leading to social movements against government policies [38]. We have incorporated the mechanism in potentially its simplest format in the model, demonstrating its utility in long-term projections.
This study also contributes to mathematical biology literature by showing the benefits of broadening model boundaries to include policy decisions within epidemic models. While many modelers consider governments as external entities in a pandemic [34][35][36], we view governments as additional players with their own perceived risks, adjusting their policies accordingly. Thus, we expanded the model boundary to include policy changes within the model, an approach that has proven to be insightful in a few other cases before [37]. Finally, we included societal learning to represent how societies' responses to COVID-19 have evolved over time. Specifically, during the pandemic, strategies to decrease contact rate moved from solely focusing on mobility to adopting actions that have less socio-economic burden [45,46]. Our model for the first time formulates such a learning process, which proves helpful in replicating mobility data.
Furthermore, the study sheds light on the main difference between exogenous and endogenous modeling of human behavior [64]. We demonstrate that an exogenous representation of human behavior may provide an excellent fit to historical trends but fails in out-of-sample tests. This finding is critically important, as many studies rely on exogenously modeling human behavior, often by changing parameter values to capture change in human behavior. As our analysis shows, the exogenous approach can misleadingly provide a good fit to historical data, but since it does not model human behavior endogenously, it will fail in projection when people start changing their behavior. This finding resonates with similar points raised in other studies [47,64,65].
In addition to methodological contributions, the study has several practical and policy implications. The important role of data in developing proper behavioral epidemic models necessitates the development of real-time data collection methods for capturing human responses at scale during disease outbreaks. Importantly, the collected data should be quantitative or easy to transform to a quantitative measure for use in mathematical models. Furthermore, the study shows limitations of models missing behavioral mechanisms, possibly leading to misleading projections. This was the case during the COVID-19 pandemic too. Users of epidemic models should more carefully assess the reliability of the models, and whether they accurately reflect behavioral complexities. In addition, feedback-rich models are helpful tools for enriching community awareness during pandemics as well as enriching model structures to include a wider range of behavioral feedback [66,67]. Acknowledging model limitations is also important for making model-informed decisions. We invite more collaborations between mathematical modelers, policy researchers, and social scientists. Such collaboration is pivotal in developing theoretically sound models.
This study has several limitations. First, while this study introduces four major behavioral mechanisms, other relevant phenomena might be missing from our model. Specifically, many important behavioral factors can emerge after the introduction of new vaccines including the spread of (mis)information about the vaccines. More studies should be conducted to identify other potential psychological factors influencing people's behavior. Second, our model was tested against eight countries at the aggregate level. More comprehensive validation tests that include data from other countries are required to build more trust in these types of models. In addition, the model can be calibrated for different communities to better capture variation in psychological responses across communities. Altogether, this study presents a behavioral epidemic model that couples psychological constructs with disease dynamics, closing several feedback loops. We invite other researchers to build upon this study, use more data on behavioral factors, and test more comprehensive models.