Introduction

Popular support is an important aspect of the ability of any state to fight wars. The evidence from recent decades that have seen authoritarian states go to war without initial popular support-Putin's invasion of Ukraine, for example-and sustain expeditionary warfare with little popular engagement-the US's 'forever wars' in Afghanistan and Iraq-suggests that the relationship between public opinion and war is complex. The degree to which public opinion matters depends on the nature of the state and the war it is fighting. Authoritarian states can fight unpopular wars and use their control of communications and media to build jingoistic sentiment around wars that the regime chooses to fight. Democratic states can sustain faraway wars provided they are lowintensity and there is establishment consensus around their necessity.

Public support for war in response to invasion is a distinctive case within the study of war and public opinion. When faced with an existential threat, the power of the state to inspire loyalty and devotion is severely tested. Wars are often said to induce 'rally-around-the-flag' behaviour. Wars against invasion test the loyalty of citizens to their state like no other event. The Ukrainian population's dogged resistance to Russian military aggression is a testament to the power of faith in their country to inspire action and to that of Ukraine's leader, Volodymyr Zelensky, its military institutions, and its civil society to inspire and organize their country's resistance (e.g. Onuch and Hale 2022).

Sustaining resistance to military invasion can, however, be extremely challenging for a country's leadership. Research on political regimes and war highlights the importance that incumbents place on favourable public opinion and mass support (Reiter and Stam 2002). When faced with an existential threat, the public often rally in support of the war effort. However, research shows that public support tends to dwindle as wars drag on, there is no end in sight, and more people are affected by the direct costs of violence-loss of life, displacement, and economic misery (Gartner and Segura 2021;Vigers 2023). Therefore, political elites may attempt to foster support for the ongoing war effort. One way to do this is to show that public support is generally high, which can then encourage more support. Public support for the war, then, is endogenous to public support. These considerations can, in part, explain Russia's sham referenda in occupied parts of Donetsk, Luhansk, Kherson, and Zaporizhzhia in September 2022.

As noted by Rosenfeld (2023: 45), the fact that there are attempts to manipulate public opinion should not discredit the entire survey research enterprise; rather, it 'heightens the importance of academic survey research, and the standards of transparency it embraces'. War and violence make public opinion harder to study but, equally, elevate the importance of such efforts. Persistent questions with wartime polling, then, are: What effect does war itself have on the accuracy of polling? Are public opinion polls that show shifts in political attitudes reliable? We tackle these questions with an analysis of survey data collected in Ukraine during the ongoing war, working with the experienced pollsters at the Kyiv International Institute for Sociology. In doing so, we aim to highlight challenges to the study of Ukrainian public opinion and point to steps forward in these uncertain times.

Our analysis relies on a nationally representative survey fielded in Ukraine in December 2019 and a follow-up telephone survey in government-controlled territories in October 2022, six months after Russia's full-scale invasion of Ukraine. Part of the 2022 survey includes a re-surveyed sample from the 2019 survey (the 20 per cent that could be re-contacted, representing 25 per cent of those who agreed to be re-contacted at a later date). We draw on this unique longitudinal survey to highlight two methodological challenges that affect studies of public opinion in Ukraine and other war settings: (1) under-coverage and (2) response bias. Combined, these methodological challenges lead us to call for great caution in the analysis and interpretation of public opinion data.

We make two recommendations to those working with wartime polls. First, it is important that researchers tailor their interpretations to the areas and populations to which they have access. In a setting of war, researchers and polling companies are likely to be unable to access certain areas and populations due to the security risks posed to both enumerators and respondents. Second, due to significant social desirability and stigma effects, we recommend that researchers employ experimental survey designs to elicit more accurate preferences on politically sensitive questions.

The rest of the paper is structured as follows: Section 2 discusses the importance of public opinion during war; Section 3 outlines the research design: Section 4 investigates the two key challenges to such research; Section 5 looks at possible ways forward; Section 6 concludes.

Public opinion during war

Popular support for war is important because it is ultimately the people who bear the costs of war. This matters particularly for leaders in democracies, who rely on public support to remain in power and to initiate, continue, or conclude a war-indeed, public support (or lack thereof) for war is a pillar of the democratic peace theory (Kant 1992). High levels of public support can help unify a nation, while low support can lead to turmoil. Authoritarian leaders, although certainly not immune to the pressure of public opinion, have more leeway to repress dissenting voices, shift the human costs of war onto certain segments of the population, and, thus, ensure the regime's survival. This is evident today in wartime Russia, where new censorship laws threaten anti-war dissenters to long prison sentences (Human Rights Watch 2023) and the direct human costs of the war are felt most in the country's peripheral regions (Kovalev 2022).

Political elites know the importance of mass support for the war effort, and they take steps to foster it (Bueno de Mesquita et al. 2005). One technique is to spread information about support for the war. Mass support for war is, in a sense, endogenous to perceptions of support for war (Russett 2014). In related research, Buckley et al. (2023) show that perceptions of support for the authoritarian regime in Russia are endogenous to actual support, thus underscoring the importance for leaders of fostering positive images of their regime through the media, education, and ceremonial practices. The parallels for states in times of war are striking. Wartime polls are an important way to measure support for war and political elites, and for the potential for settlements (Dill et al. 2023). When favourable, they can also be part of the state's information campaign to mobilize the public.

Public opinion is central to Russia's war in Ukraine in at least two ways. First, it was evidently ignored by Russian President, Vladimir Putin, on the eve of the full-scale invasion in February 2022. While reports emerged that Kremlin-conducted 'opinion surveys' in Ukraine in January 2022 showed lack of support for Ukraine's president, Volodymyr Zelenskyy (Miller and Belton 2023), this did not equate to support for the invading forces. In fact, numerous polls in January 2022 showed that many Ukrainians were ready to oppose any Russian invasion (Reznik 2023). This has to date proven to be the case. (Reisinger et al. 2023), and limitations on surveying (Rosenfeld 2023) may affect the future of the field. A similar discussion might be warranted among those studying public opinion in Ukraine.

Indeed, researchers studying Russian and Ukrainian political opinion face similar challenges. In both countries respondents may be reluctant to share anti-war sentiment or support for peace. However, Ukraine differs substantially in two respects. First, large swathes of Ukrainian territory are not accessible because they are currently experiencing fighting or are under Russian occupation. The populations that reside in these areas are thus not included in surveys. Second, many have fled Ukraine. Some 12-18 per cent of our 2022 survey respondents are internally displaced, but a large number are likely in neighbouring countries.foot_0

In response to these challenges, we first consider how restricted geographical coverage may be affecting interpretations of public opinion polls in Ukraine. Second, we consider the effect of different forms of response bias in the polling that can be carried out.

Research design

We designed two waves of nationwide public opinion surveys in Ukraine, both fielded by the Kyiv international Institute for Sociology (KIIS), an experienced and highly reputable polling firm. The first wave was funded by the National Science Foundation in the US, and the second wave by the Norwegian Research Council. 2 The first wave was conducted in December 2019, at a time when there was an ongoing war between Russian-backed separatists in eastern Ukraine (including active participation of Russian troops) and Ukrainian forces under the command of the Kyiv government (Arel and Driscoll 2022). The survey asked respondents a range of questions aimed at capturing their geopolitical orientations, interest in politics (both domestic and international), political trust, and outlook for the future, as well as socio-demographic and background questions. Many of the topics in the survey remain central to the war.

The 2019 survey was conducted face to face on people's doorsteps. Respondents were assured that their answers were anonymous and confidential, and they could opt to end the survey at any point. 3 The sample (2,212 respondents) is nationally representative at the time (excluding the areas not controlled by the Ukrainian government in the east and Crimea). We asked respondents for consent to recontact them for a second wave. Over 75 per cent (N=1,712) agreed to take part in a future survey. We intended to field a second survey with the same respondents a year later, but this wave was put off first by the outbreak of the global COVID pandemic and then by Russia's full-scale invasion.

In October 2022, we conducted a follow-up survey by telephone. KIIS called all respondents surveyed during the first round. Just over 20 per cent of respondents from the first wave in 2019 took part in the second wave in 2022 (N=429), in the midst of war. This represents high levels of attrition, in large part because KIIS did not contact people living in areas with active combat or occupied by Russia and did not contact respondents who had fled abroad. KIIS also used random sampling (N=1,783) to 'top up' the second wave to ensure a sample large enough to be representative of government-controlled Ukraine-as much as it could be in a war setting. Combined, the data collection therefore contains two nationally representative samples of government-controlled areas and, embedded within them, a longitudinal sample-the same respondents surveyed in both 2019 and 2022.

Key challenges

In this section we explore two important methodological challenges that may complicate the interpretation of war-time surveys, particularly those of a non-panel format: under-coverage and response bias.

Under-coverage

Under-coverage occurs when a sample lacks representation from some groups in a population (Agresti 2018). For instance, nomadic communities and homeless people are often excluded from national household surveys. Equally, coverage in phone surveys may be a function of social class and income. For instance, some respondents may not own a phone. To overcome under-coverage, researchers may employ weighting adjustment measures, often by relying on additional sources of data (Wagner and Stoop 2018). For nationally representative surveys, population weights based on recent census data often ensure that the sample is representative of the population.foot_4

Under-coverage is a major challenge for anyone collecting and analysing data in Ukraine. There are at least four causes of under-coverage: migration abroad, migration within the country, inability to access areas under Russian occupation, and inability to access territories with active military operations.

For surveys assessing political questions related to the war-and wanting to present a nationally representative set of views-one key issue is the potential exclusion of individuals who were geopolitically oriented towards Russia prior to Russia's full-scale invasion in 2022. Russianoriented respondents may have either fled to Russia-controlled areas in the eastern part of Ukraine or sought refuge as displaced persons, either in neighbouring European countries or in Russia.

Although the exact figures are subject to debate, a considerable number of people fled to Russia willingly, out of necessity, or under duress.foot_5

Figure 1 maps the number of respondents per oblast surveyed in 2019 (top) and the number of respondents surveyed in 2022 (bottom). Areas where surveys were possible in 2019 but not so in October 2022 are clearly visible. It is important to note that the data collection method changed from face-to-face to telephone, although that change is unlikely to be driving such high levels of attrition alone and/or to explain its spatial variation.

Where and why did respondents drop out between the survey waves? To understand the causes of attrition between survey rounds, we analyse the spatial variation in survey attrition between surveys. Figure 2 shows the results of a simple logistic regression in which the dependent variable takes 1 if respondents drop out of the survey for the second wave and 0 otherwise. 6 As an independent variable, we include respondent oblast in 2019 as a categorical variable with the reference category as Kyiv city. The coefficient is shown on the x-axis and coloured red if it is statistically distinguishable from zero. All oblasts are shown in order of coefficient strength, i.e. the likelihood of attrition, along the y-axis. The figure shows that respondents in eastern oblasts such as Luhansk and Donetsk were less likely to be re-surveyed. This tendency is also visible in Kherson in the south. 7 Zaporizhzhya, too, is almost statistically significant at 95 per cent levels of confidence. These regions constitute a significant portion of wartime violence. Based on data from the Armed Conflict Location & Event Data Project (ACLED), approximately 52 per cent of fatalities recorded between February and October 2022 occurred in Luhansk and Donetsk. Additionally, about 15 per cent of the total fatalities took place in Kherson, jumping to 24 per cent if we also include Zaporizhzhya. While this analysis does not account for the common reasons for attrition-people are unreachable, lost phones, changed numbers, or lost interest-it is clear that the areas with the highest rates of attrition are those experiencing the highest levels of violence in the war and/or are under Russian occupation. For questions that are pivotal to public discourse and policy debates about the war, we must acknowledge that our 2022 sample is derived from regions less impacted by the violence of war and Russian occupation, which are also the areas that were more inclined towards NATO membership and to the West before the war (O'Loughlin et al. 2022). While the war has had a clear rallying effect in the areas that our (and other 2022) surveys cover, we cannot know whether there has been a similar effect in the most war-affected or Russia-controlled territories-or among the populations that sought refuge outside Ukraine.

This means that the conclusions derived from wartime polling are limited to the areas accessible to researchers and the current residents within those areas. While this may seem obvious, it carries implications beyond academic research. It is essential for researchers to clarify that surveys conducted in Ukraine during the war do not capture the perspectives of individuals who remain in the most war-affected areas and under Russian occupation within Ukraine.

Why might this matter? Decades of research have identified a persistent East-West divide in political opinions within Ukraine (Abdelal 2005;Bakke et al. 2023;Munro 2007), although more recent research indicates that these regional differences have been waning (Onuch and Arkwright 2021;Onuch andHale 2018, 2022). To illustrate the importance of carefully considering who is and is not surveyed, let us return to the question of NATO membership referred to by Boris Johnson. NATO membership was a key foreign policy issue in 2019-and a salient policy issue in Ukraine for several years, especially since Russia's military support to separatists in Eastern Ukraine in 2014-that is central to the ongoing war (Ishchenko 2023). Popular support for NATO membership is instrumental to politicians in the West calling for greater military support for the Kyiv government. Like other researchers, we asked respondents whether they 'agree that your country should be in NATO'. Comparing our two samples reveals why scholars must be cautious when interpreting such surveys in Ukraine. The 2019 sample re-surveyed in 2022 was generally more supportive of NATO membership than the full 2019 sample (comparing the top left and top right plots). These differences are statistically significant (p < 0.001) in 2019. 8 This difference all but disappears in 2022 (i.e. when comparing the re-surveyed sample and the full 2022 sample). The analysis raises concerns that the type of person surveyed is different between survey waves, regardless of personal changes in attitude. If this is the case, we may be over-estimating the rate of change for support to join NATO due to limited coverage.

To try to assess this possibility, we estimate support for NATO membership by imputing missing values. The imputation relies on strong assumptions but serves merely to illustrate the scale of the coverage challenge. Here, we focus on the 2019 sample (N=2,212), which is representative of government-controlled Ukraine in 2019 (therefore excluding areas controlled by pro-Russian separatists in Luhansk and Donetsk shown in red in Figure 1). For just over 20 per cent of the sample (N=429), we can estimate the effect of Russia's full-scale invasion in 2022 with the following ordinary least squares (OLS) specification:

In the equation, 𝛾𝛾 𝑖𝑖 is the level of support for NATO membership for respondent 𝑅𝑅. Our main variable of interest is 𝛽𝛽 1 𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅 𝑅𝑅𝑅𝑅𝑖𝑖𝑅𝑅𝑅𝑅𝑅𝑅𝑖𝑖𝑅𝑅 𝑖𝑖 , which captures the average effect of Russia's invasion on support for NATO membership. To isolate this effect, we also include a vector 𝑋𝑋 𝑖𝑖ℎ of demographic controls measured in 2019 (age, gender, income, education, and oblast fixed effects). On average, 𝛽𝛽 1 𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅 𝑅𝑅𝑅𝑅𝑖𝑖𝑅𝑅𝑅𝑅𝑅𝑅𝑖𝑖𝑅𝑅 𝑖𝑖 leads to a 22 percentage point increase in support for NATO membership. The effect of the war on support for NATO membership is likely to be heterogeneous dependent on respondent characteristics.

The extent to which the full-scale invasion shifted support for NATO membership may depend on whether respondents were more or less Russia-oriented on the eve of the invasion. As we discuss further below, Russian identification and ethnicity in Ukraine are debated concepts and difficult to measure. For this exercise, we consider whether respondents responded to the 2019 survey in the Russian or Ukrainian language, which we take to be a measure of individual language preference (Onuch and Hale 2018).foot_9 For the panel, the full-scale invasion led to a higher change among those who were interviewed in Russian in 2019 (27 percentage points) compared with those who were interviewed in Ukrainian (5.6 percentage points), although their average levels of support are considerably lower in 2019: 17 per cent of respondents who took the survey in Russian supported NATO membership compared with 59 per cent of respondents who took the survey in Ukrainian. We impute different rates of change depending on the 2019 interview language. 10 We also consider the magnitude of these effects. For each person for whom we do not have a value in 2022 due to attrition, we impute different values under the following assumptions:

1 Support for NATO has not changed. 2 The average change for these respondents was the same.

3 The average change for these respondents was larger by two orders of magnitude. 4 The average change for these respondents depends on whether they were more Russiaoriented before the conflict. 5 The average change for these respondents was larger by two orders of magnitude if they were more Russia-oriented before the conflict.

Table 1 shows average levels of support with panel data (column 1) and imputed data according to the five assumptions. The average levels of support vary significantly between 57 per cent (assumption 1) and 94 per cent (assumption 3). Assumption 5, which assumes heterogenous and much stronger effects, comes closest to the average levels of support measured in the panel sample.

Our purpose here is not to estimate actual levels of support for NATO membership in Ukraine today. Instead, we show how different assumptions about the effects of Russia's full-scale invasion of Ukraine lead to highly variable estimated levels of support for NATO membership. This exercise-which, for the sake of parsimony, ignores selection problems, response biases,foot_11 and statistical confidence-serves to illustrate how under-coverage may create difficulties in estimating the effects of the invasion on Ukrainian public opinion regarding a question that is central to the war. These considerations are generally ignored when comparing support after and before Russia's full-scale invasion, for instance, in the style of Boris Johnson.

Response bias

In addition to the challenge of under-coverage, there are two forms of response bias that may be affecting our results. It is possible that respondents who were more Russia-oriented in 2019 are now (1) less likely to take part in surveys (unit non-response) and (2) less likely to either answer politically sensitive questions (item non-response) or, relatedly, reveal their true preferences (preference falsification). The common thread linking these challenges is the issue of sensitivity (Tourangeau and Yan 2007) and the perceived risk of punishment (Reisinger et al. 2023). Given that surveys today can only be representative of areas controlled by the government, these issues pose extra challenges to the interpretation of public opinion polls. In a first instance, they may lead to surveys not being representative of the population of government-controlled Ukraine. Second, they may lead to inaccurate measures of public opinion. In this section, we statistically explore these possibilities.

Unit non-response arises when potential respondents can participate in a survey but decline to do so. If specific groups of respondents opt out of the study, it can introduce sampling bias, which has the potential to undermine the research project's conclusions. There are reasons for not taking part that may not lead to bias in the sample. However, in some instances, respondents will refuse to take part in a survey for reasons related to the topic of research. There are several reasons why potential participants refuse to take part in a survey, including personal reasons (for example, levels of altruism or concerns with privacy), survey-related reasons (for example, responses are generally lower to surveys on sensitive topics), and egoistic reasons (for example, respondents are motivated by money) (Groves et al. 2000).

War exacerbates these challenges. In wartime, individuals who are fearful of voicing unpopular opinions are less likely to make their voices heard or be willing to participate in surveys, thereby skewing the composition of the sample (Rosenfeld 2023). At the time of writing, most polling in Ukraine is conducted through computer-assisted telephone interviews (CATI) rather than face-toface interviews. While CATI surveys employ randomization, the act of answering an unfamiliar call and agreeing to participate in a survey of uncertain duration with a stranger is only appealing to certain types of person (Marken 2018). Some individuals may fear personal harm if they participate. Additionally, sensitive subjects like criminal activity or experiences of violence may lead to re-traumatization. Ethical research practices must take measures to prevent actual harm (Cronin-Furman and Lake 2018). Regardless of these precautions and efforts to communicate them to potential respondents, however, the perceived risk of harm remains high within the context of armed conflict. Therefore, unit non-response due to the sensitivity of the research topic is particularly pertinent for war polls.

Unit non-response can lead to bias if there is a relationship between the dependent variables of a study and the characteristics that lead to the non-response. This is likely to be common in war settings, especially if surveys are interested in the political views of respondents as opposed to their immediate needs. For example, in an armed conflict, people who hold positive views of an armed actor A may not respond to surveys in a territory controlled by a rival armed actor B. The analysis of this hypothetical survey-regardless of whether it employs experimental methods to overcome item non-response and preference falsification, as discussed in detail below-will most likely underestimate levels of support for armed actor A.

Research on unit non-response is limited due to ethical concerns around consent and the privacy of people who opt out of surveys. Instead, the analysis of non-response generally relies on paradata-data that are collected as part of the process of conducting the survey. During data collection, interviewers 'record the visits they make, the outcome of these visits, intermediate and final results, reasons for refusal, assessments of future cooperation, and neighbourhood observations' (Wagner and Stoop 2018: 37). Specifically, enumerators generally approach potential survey respondents, provide information about the survey, outline the associated benefits and risks, and, finally, ask whether they would like to participate (i.e. they seek informed consent). Enumerators then record whether respondents agree or decline. If they decline, enumerators may ask why, the interaction then ends, and no more information is gathered. Therefore, statistics on unit non-response tend to be summary statistics (for example, the percentage of respondents who decline to take part). Summary statistics may show little variation over time but contain different stratifications within the data. In our survey, for instance, it may be that many West-oriented respondents are slightly more likely to participate in a survey in 2022 but some Russia-oriented respondents are much less likely to participate in the same survey. If this were the case, summary statistics would show, on average, an insignificant change in unit non-response. This is an important limitation of analysing response rates over time, as noted by Rosenfeld (2023) in her analysis of non-response in face-to-face surveys conducted in Russia. While it is possible to examine different levels of attrition across geographic space (for example, at the level of primary sampling units, such as districts), surveys are often no longer representative due to limited sample sizes at smaller spatial units. In sum, it is difficult to assess the characteristics of people declining to participate due to important ethical limitations on gathering additional information about those respondents.

Panel data such as ours, therefore, provides an opportunity to analyse unit non-response, allowing us to base our conclusion 'on evidence rather than faith' (Rosenfeld 2023: 40). Data collected during a first wave can be used to provide additional information about unit non-response in the second wave. In our 2019 survey, we sought permission to contact respondents for a second wave.

In this section, we analyse the political opinions and personal characteristics of respondents in 2019 that may be related to non-response in 2022. We ask the following: what type of person did not respond to the survey in the second round? To answer this question, we explore three dimensions: (1) proximity to fighting, (2) demographic characteristics, and (3) salient group identifiers in 2019. In subsequent analyses, we run a series of logit and linear regression models.

To account for proximity to violence, we hold respondent location in 2019 constant by including oblast-fixed effects. This is important in the Ukrainian context because, as shown in the previous section, patterns of violence generally map onto public opinions on salient, and potentially sensitive, political issues.

For follow-up interviews in October 2022, enumerators coded different forms of attrition for respondents who agreed to be re-surveyed in 2019. First, if reachable, enumerators asked respondents the reason for not taking the survey. Enumerators also coded different ways in which respondents were not reachable. We recode this information for ease of analysis. The full coding is shown in Table 2. Overall, 24.7 per cent of respondents who agreed in 2019 to be re-surveyed at a later date completed the second survey, 51.7 per cent were unreachable, and 23.5 per cent refused to take part for various reasons. We use this coding to statistically explore attrition in more detail. We assess whether more Russia-oriented respondents were less likely to take part in the second wave of the survey. If there is a relationship between identifiers of Russian orientation and unit non-response, then the risks of bias are heightened because it is likely to be correlated with views of the war. As noted, Russian identification and ethnicity in Ukraine are debated concepts and difficult to measure. We follow Onuch and Hale (2018) and include four measures that capture different dimensions related to ethnic identity: individual language preference, language embeddedness, ethnolinguistic identity, and nationality. For individual language preference, we code whether the interview was conducted in Russian. Onuch and Hale (2018) argue that social pressure is lower with this approach than with direct questions about languages spoken at home or work. In 2019, almost 51 per cent of interviews were conducted in Russian. To measure language embeddedness, we ask respondents the language that they speak at home. Over 41 per cent of the total sample stated that they speak Russian. Our measure for Russian ethnolinguistic identity is whether respondents say that Russian is their native language (31 per cent). Finally, we include whether respondents identify as being part of the Russian national group/nationality (natsionalnost) (just 8 per cent).

Table 3 shows the results of three logit regression analyses in which the binary dependent variable indicates whether respondents completed the survey (column 1), refused to take part (column 2), 12 or were coded as unreachable (column 3). All models include controls for age, gender, education, and income measured in 2019. The results indicate that different measures related to Russian ethnicity are generally not associated with attrition. While respondents who identify as having Russian nationality are more likely to be unreachable, those who identify Russian as their native language are less likely. In sum, these contradictory findings indicate that potentially more Russia-oriented respondents are equally likely to drop out of the sample in government-controlled areas. There is no evidence that the 2022 survey suffers from unit non-response for salient identity groups. 13 While surveys 12 We also modelled each reason for refusal separately and found no statistically significant relationship for differen t measures of Russian orientation. 13 There is a statistically significant relationship between education and attrition: respondents with higher levels of education are more likely take part in both surveys. There are several reasons why this could be the case. First, it is possible that they are better shielded from armed conflict, potentially because they are less likely to join the army or conducted during the war do not account for the views of those who have fled abroad and remain in violent or occupied areas (as detailed in the under-coverage section above), we find no evidence that the survey suffers from unit non-response bias for salient identity groups in areas where enumerators have access. We conclude that samples in accessible areas are likely to be representative of those areas, which should be reassuring to researchers who rely on surveys in Ukraine.

For those who do take surveys, do some respondents avoid certain questions or conceal their true preferences? Here, there are parallels with research on collecting public opinion in repressive regimes, including in Russia (Frye 2023;Frye et al. 2017) and during Russia's war in Ukraine (Rosenfeld 2023). There is evidence that this challenge is greater in times of war. For instance, Lyall et al. (2013) find high levels of non-response to questions about militant groups in Afghanistan.

There are two ways in which respondents may conceal their true preferences. First, individuals may avoid questions either by stating that they do not know or simply by to answer, known as item non-response or the 'don't know's (Naylor and O'Loughlin 2021). There are several reasons why respondents do this. First, they may simply not know the answer. This is likely if the answer is complex. Second, respondents may not understand because the researchers asked a confusing question. Third, and related to unit non-response bias, respondents may refuse to answer sensitive questions. Refusal to answer is particularly problematic and likely during wartime. Indeed, while war generates a 'rally-around-the-flag' effect, it also leads some people to engage in strategic hedging on sensitive questions. Respondents in Ukraine opposed to NATO membership may respond 'don't know' because opposition may be socially undesirable.

We can assess item non-response bias by drawing on survey responses from the 20 per cent who completed surveys in 2019 and 2022. A Little's test (Little 1988) of questions on potentially politically sensitive topics reveals that missing data are not missing completely at random (MCAR). This indicates that respondent characteristics are related to item non-response. To assess the types of people who may be avoiding sensitive questions, we focus on respondents who were potentially Russia-oriented in the 2019 survey. Building on the above analyses, we take as our independent variable respondents' answer to the question: 'Do you agree that your country should be in NATO?'. We focus on these respondents, as they may feel social pressure to respond in certain ways and thus avoid questions.

We compare the average number of item non-responses per respondent. To do so, we count the number of times that respondents answered 'don't know' or refused to answer across 14 potentially sensitive political questions (listed along the y-axis in Figure 4). The results of this analysis, which include demographic controls and oblast-fixed effects, are shown in Table 4. On average, if a respondent was against joining NATO in 2019, they provide 0.75 more non-responses than those who were in favour of NATO membership. This analysis shows a statistically significant relationship between non-response and pre-war political orientation but ignores important variation across questions. Figure 4 shows the results of a series of logit regression models in which the dependent variable is whether respondents provided a non-response to each question and the key independent variable is whether they were against joining NATO in 2019. As indicated by the red coefficient lines, those who were against joining NATO in 2019 are more likely to provide a volunteer to contribute to the war effort in another capacity. Equally, they may work in industries less affected by the war and therefore have the financial means to remain in Ukraine. In the model for completing the survey, age is also statistically significant. This provides evidence that older respondents may be less likely to move during the war, for which there is also anecdotal evidence from the frontlines (Amnesty International 2022a).

non-response to almost half of the potentially sensitive questions. It is important to note that these respondents may simply not know the answer to sensitive questions. The brutality of the Russian invasion, evidence of war crimes in Bucha and Izyum, and indiscriminate bombing across the Ukrainian territory may have caused them to question their own beliefs. Indeed, emerging evidence suggests that many Russian speakers are learning Ukrainian to distance themselves from Russia (Gormezano 2022;Harding 2023;Kulyk 2023). We cannot rule out this dynamic. For many possible reasons, our analysis suggests strategic hedging among respondents who were more Russia-oriented in 2019. Item non-response bias and preference falsification have similar underlying causal mechanisms.

One can measure preference falsification with experimental designs aimed at eliciting respondents' true preferences, which we discuss in detail below. We did not develop an experimental approach to measure preference falsification, but we note that the existence of item non-response indicates that the likelihood of preference falsification is high in 2022. Based on our analysis of item nonresponse, it is possible that preference falsification is affecting interpretation of the survey data collected in 2022.

Ways forward

What can researchers do when faced with these methodological challenges? The two challenges, under-coverage and response bias, necessitate different responses.

There is no panacea for under-coverage. In the light of this, researchers should take steps in their analysis, interpretation, and dissemination to actively communicate the uncertainty around accurately measuring public opinion. This may require more than simply stating that the 'survey was conducted in government-controlled territories', but, additionally, explicitly stating that we cannot know the views of those who have fled Ukraine or remained in the most violent and occupied territories, as well as acknowledging that these views may differ from the views of those covered.

Researchers who aim to elicit political preferences on sensitive topics, as opposed to immediate needs, should develop research strategies to overcome response biases. Such efforts mostly take place before a survey is fielded. How can we design research to elicit truthful responses and avoid both non-response and preference falsification? There is a large and growing literature on survey methodology that focuses specifically on indirect methods to elicit responses to sensitive questions.

One such technique is list experiments (Blair et al. 2020;Glynn 2013). As noted by Reisinger et al. (2023), these allow flexibility, can be combined with direct questions to increase accuracy, and shed light on the types of people most likely to misreport. In the most basic set-up, respondents are randomly assigned two lists, one of which includes a sensitive item. They are asked how many of the items on the list to them. The difference in mean between the lists provides an estimate of the prevalence of the sensitive item. More advanced multivariate regression analysis is also possible (Imai 2011;Lyall et al. 2013). Because respondents do not directly say whether the sensitive item applies to them, it allows them to provide their true preference and thus overcome response biases. While susceptible to design effects, including floor and ceiling effects (e.g. Lyall et al. 2013), list experiments provide an effective way to measure sensitive items.

There are several other techniques for measuring sensitive questions. Rosenfeld et al. (2016) evaluate list experiments alongside other techniques, including indirect questioning techniques, endorsement experiments, and randomized response techniques (see also Blair et al. 2015).

Researchers need to consider the trade-offs of the various approaches. For example, some techniques are more difficult to implement with poorly educated populations. In these cases, the trade-off will also be informed by resource constraints. Direct questioning allows researchers to ask a range of questions in a limited timeframe, which keeps interview time (and, by extension, respondent loss of attention) and costs of surveying down. Eliciting responses to sensitive questions may require more effort per item, but the payoff in terms of reducing bias is high.

Conclusion

Historic population movements, a massive military mobilization, and huge loss of life mean that under-coverage is affecting survey samples in Ukraine. We show that respondents in eastern and southern oblasts are less likely to be re-surveyed-no doubt due to Russia's military violence and occupation in these oblasts. Even in areas less affected by violence, the sample may be affected by unit and item non-response. Reassuringly, we find that salient identity measures in 2019 are not associated with unit non-response in 2022. This means that the latter survey does not suffer from unit non-response bias for salient identity groups. However, we observe that those who were against NATO membership in 2019 were less likely to answer politically sensitive questions in the 2022 follow-up survey. In the absence of an experimental set-up, it is difficult for us to empirically measure preference falsification. However, based on our findings for non-response, we advise that researchers employ experimental survey designs to elicit accurate preferences on politically sensitive questions, especially those pertaining to war aims.

Wartime polls tell us important things-and it is clear that public opinion in governmentcontrolled areas of Ukraine has shifted as a result of the war-but there are significant challenges related to representativeness and expressed preferences. As researchers, we need to communicate the uncertainty these challenges entail. This is not to say that researchers should shy away from conducting public opinion surveys in Ukraine or other war-torn settings. Indeed, wars underscore the importance of rigorous and independent academic research, as it can help guide policy-makers and first responders, and give a voice to those suffering from violence. This is certainly the case for Ukraine. In 2022 alone, the United Nations High Commissioner for Human Rights recorded over 6,000 civilian casualties, most of which were due to shelling, rocket attacks, missiles, and air strikes (OHCHR 2022b Ukraine should pursue its own path. 15 Results in 2022 show a drastic shift towards the West. The single largest category becomes 'towards the West'. Compared with 2019, this category increases by over 300 per cent. A significant minority still select the middle of the scale, although their number is now dwarfed by those who favour greater rapprochement with the West. We note that, while the number of respondents who report wanting to move slightly more towards Russia (scores 6 to 9) shrinks significantly, the single category of 'towards Russia' (10) increases greatly compared with 2019.

The numbers are small, and no firm conclusions can be drawn. However, they could indicate two things. First, it is possible that the war has polarized those who were oriented more towards Russia, some shifting towards the West and others inclining even more towards Russia. Second, and alternatively, some respondents may choose to move towards Russia as a reconciliatory measure. They may support a negotiated end to the conflict and understand that this can only be achieved by moving towards Russia. Both dynamics could be at play, and we cannot separate them. In any case, the clear conclusion from comparing across survey waves is that Russia's war in Ukraine has caused people to drastically shift towards greater rapprochement with the West.

Comparing between the re-surveyed sample and the nationally representative sample, it is clear that the re-surveyed sample in 2022 (top right plot) leaned more towards the West in 2019 compared with the full sample, which leaned slightly more towards Russia (top left plot). The difference in means between those who were re-surveyed and those who were not is statistically significant (p <0.001). 16 In other words, those who were re-surveyed in 2022 are more likely to have wanted to move towards the West in 2019. What does this mean for surveys conducted in 2022? It could mean that researchers are more likely to sample more West-oriented respondents since the war started. Clearly, individual preferences have shifted significantly, as is visible in the comparison across waves, although it is still too early to know whether this is a short-lived rallyaround-the-flag effect or a more enduring shift in preferences. Crucially, it appears that the type of people taking part in surveys has changed-an issue that is only visible to researchers using panel data. If this is the case, researchers who rely on surveys may be over-estimating the rate of change. Any visible differences in orientation between the 2019 samples all but disappear in 2022. 17

15 We cannot easily tell these apart. Raaijmakers et al. (2000) show that respondents who are not certain often select the middle category, despite having the option to say 'don't know'. Indeed, a consistent finding in foreign policy preference research, that the general population lack interest in foreign policy and geopolitics (Holsti 2009), would back this up. However, lack of interest might not be the case in a conflicted state like Ukraine, where foreign policy is a salient issue (O'Loughlin et al. 2014). The predominance of the middle category may also be due to social desirability or acquiescence bias (e.g. Nadler et al. 2015).

16 See Table A2. 17 Table A2 shows that statistically significant differences also disappear.