I. INTRODUCTION

One of the core problems in security is detection of malicious behavior, with examples including detection of malicious software, emails, websites, and network traffic. There is a vast literature on detection approaches, ranging from signaturebased to machine-learning based [8], [26], [34]. Despite best efforts, however, false positives are inevitable. Moreover, one cannot in general reduce the rate of false alarms without missing some real attacks as a result. Under the pressure of practical considerations such as liability and accountability, these systems are often configured to produce a large amount of alerts in order to be sufficiently sensitive to capture most attacks. As a consequence, cybersecurity professionals are routinely inundated with alerts, and must sift through these overwhelmingly uninteresting logs to identify alerts that should be prioritized for closer inspection.

A considerable literature has therefore emerged attempting to reduce the number of false alerts without significantly affecting the ability to detect malicious behavior [16], [30], [13]. Most of these attempt to add meta-reasoning on top of detection systems that capture broader system state, combining related alerts, escalating priority based on correlated observations, or using alert correlation to dismiss false alarms [38]. Nevertheless, despite significant advances, there are typically still vastly more alerts than time to investigate them. With this state of affairs, alert prioritization approaches have emerged, but rely predominantly on predefined heuristics, such as sorting alerts by suspiciousness score or by potential associated risk [2]. However, any policy that deterministically orders alerts potentially opens the door for determined attackers who can simply choose attacks that are rarely investigated, thereby evading detection.

Building on the observation of the fundamental tradeoff between false alert and attack detection rate, we propose a novel computational approach for robust alert prioritization to address the challenge. Our approach assumes a strong attacker who knows the full state of the detection environment including which alerts have been triggered, which have been investigated in the past, and even the defender's policy. We also assumed that the adversary is capable of finding and utilizing a near optimal attack strategy against the defender policy based on his knowledge of the system and defending policy. To defend against such a strong attacker, we propose to compute the optimal stochastic dynamic defender policy that chooses the alerts to investigate as a function of the observable state, and that is robust to our threat model. At the core of our technical approach is a combination of game theory with adversarial reinforcement learning (ARL). Specifically, we model the problem of robust alert prioritization as a game in which the defender chooses its stochastic and dynamic policy for prioritizing alerts, while the attacker chooses which attacks to execute, also dynamically with full knowledge of the system state. Our computational approach first uses neural reinforcement learning to compute approximately optimal policies for either player in response to a fixed stochastic policy of their counterpart. It then uses these (approximate) best response oracles as a part of a double-oracle framework, which iterates two steps: 1) solve a game involving a restricted set of policies by both players, and 2) augment the policy sets by calling the best response oracle for each player. Note that our approach is completely orthogonal to methods for reducing the number of false positive alerts, such as alert correlation, and is meant to be used in combination with these, rather than as an alternative. In particular, we can first apply alert correlation to obtain a reduced set of alerts, and subsequently use our approach for selecting which alerts to investigate. Since alert correlation cannot be overly aggressive in order to ensure that we still capture actual attacks, the number of alerts often still significantly exceeds the investigation budget.

We evaluate our approach experimentally in two application domains: intrusion detection, where we use the Suricata opensource intrusion-detection system (IDS) with a network IDS dataset, and fraud detection, with a detector learned from data using machine learning. In both settings, we show that our approach is significantly more effective than alternatives with respect to our threat model. Furthermore, we demonstrate that our approach remains highly effective, and better than baseline alternatives in nearly all cases, even when certain assumptions of our threat model are violated.

II. SYSTEM MODEL

A. Overview

As displayed in Figure 1, our system is partitioned into four major components: a group of regular users (RU), an adversary (also called attacker), a defender, and an attack detection environment (ADE).

The regular users (RU) are the authorized users of a system. In contrast, the adversary is a sophisticated actor who attacks the target computer system. The attack detection environment (ADE) models the combination of the software artifact that is responsible for monitoring the system (e.g., network traffic, files, emails) and raising alerts for observed suspicious behavior, as well as relevant system state. System state includes attacks that have been executed (unknown to the defender), and alerts that have been investigated (known to both the attacker and defender). Crucially, the alerts triggered in the ADE may correspond either to behavior of the normal users RU, or to malicious behavior (attacks) by the adversary. We divide time into a series of discrete time periods. The defender is limited in how many alerts it can investigate in each time period and must select a small subset of alerts for investigation, while the adversary is limited in how many attacks it executes in each time period. The full system operates as follows for a representative time period (see again the schematic in Figure 1):

1) Benign alerts are generated by the ADE.

2) These alerts, and the remaining ADE system state (such as which alerts from past time periods have not yet been investigated, but could be investigated in the future), are observed by the attacker, who executes a collection of attacks.

3) The attacks trigger new alerts. These are arbitrarily mixed into the full collection of alerts, which is then observed by the defender. 4) The defender chooses a subset of alerts to investigate.

The ADE state is updated accordingly, and the process repeats in the next time period. Next, we describe our model of the alert detection environment, our threat model, and our defender model. The full list of notation that we use in the model is presented in Table I.

B. Attack Detection Environment (ADE) Model

Our model of the attack detection environment (ADE) captures a broad array of detection settings, including credit card fraud, intrusion, and malware detection. In this model, the ADE is composed of two parts: an alert generator (such as an intrusion detection system, like Suricata) and system state.

An alert generator produces a sequence of alerts in each time period. We aggregate alerts based on a finite predefined set of types T . For example, an alert type may be based on the application layer it was generated for (HTTP, DNS, etc), port number or range, destination IP address, and any other information that's informative for determining the nature and relative priority of alerts. We can also define alert types for meaningful sequences of alerts. Indeed, the notion of alert types is entirely without loss of generality-we can define each type to be a unique sequence of alerts, for examplebut in practice it is useful (indeed, crucial for scalability) to aggregate semantically similar alerts.

At the end of each time period the system generates a collection of alert counts for each alert type t ∈ T . We assume that normal or benign behavior generates alerts according to a known distribution F, where F t (n) is the marginal probability that n alerts of type t are generated. We also refer to this as the distribution of false alarms, since if the defender were omniscient, they would never trigger such alerts. Note that in practice it is not difficult to obtain the distribution F. Specifically, we can use past logs of all alerts over some time period to learn the distribution F. Since the vast majority chooses attack actions, which are then executed by the attack generator, triggering alerts and thereby modifying the state of the ADE. Below we present our approach for approximating the optimal attack policies.

Adversary's Goals. The adversary aims to successfully execute attacks. Success entails avoiding being detection by the defender, which only happens if alerts associated with an attack are inspected. Thus, if an attack triggers a collection of alerts, but none of these are chosen by the defender to be inspected in the current round, the attack succeeds. Different attacks, however, entail different consequences and, therefore, different rewards to the attacker (and loss to the defender). As a result, the adversary will ultimately need to balance rewards to be gained from successful attacks and the likelihood of being detected.

D. Defender Model

Defender's Knowledge. Unlike the adversary, the defender can only partially observe the state of the ADE. In particular, the defender only observes N (k) , the numbers of remaining uninvestigated alerts, grouped by alert type (since clearly the defender cannot directly observe actually attacks). In addition, we assume that the defender knows the attack budget and costs of (representative) attacks. In our experiments, we study the impact of relaxing this assumption (see Sections V-C5 and V-B5), and provide practical guidance on this issue.

Defender's Capabilities. The defender chooses subsets of alerts in N (k) to investigate in each time period k. This choice is constrained by the defender's budget, which in practice can translate to time the defender has to investigate alerts. Since different types of alerts may need different amounts of time to investigate, or more generally, incur varying investigation costs, the budget constraint is on the total cost of investigating chosen alerts. Formally, let C t be the investigation cost of an alert of type t, and let α (k) +1,t be the number of alerts of type t chosen to be investigated by the defender in period k. Then the budget constraint takes the following mathematical form:

An additional constraint imposed by the problem definition is that the defender can only investigate existing alerts:

Just as with the adversary, it is useful to represent the defender as consisting of two modules: Defense Oracle and Alert Analyzer, as shown in Figure 1. The defense oracle runs a policy, which maps partially observed state of the ADE to the choice of a subset of alerts to be investigated. In each time period, after observing the set of as yet uninvestigated alerts, the defense oracle chooses which alerts to investigate, and this policy is then implemented by the alert analyzer, which thereby modifies ADE state (marking the selected alerts as having been investigated). Below we present our approach for approximately computing optimal defense policies that are robust to attacks as defined in our threat model above.

Defender's Goals. The goal of the defender is to guard a computer system or network by detecting attacks through alert inspection. To achieve its goal, the defender develops an investigation policy to allocate its limited budget to investigation activities in order to minimize consequences of successful attacks, where we assume that an attack will fail to accomplish its primary objectives if the alerts it causes the ADE to emit are investigated in a timely manner.

E. An Illustrative Example

Since our system is built on top of an abstracted model of alert investigation, the results are generally applicable to a wide range of real-world problems. We will use intrusion detection as an illustrative example in this section. Port Scan reconnaissance attack is one of the most common initial steps in remote exploitation and is a common occurrence faced by many enterprise IT professionals. In a Suricata IDS system, each alert item has different levels of categorization. For example, at the lowest layer, the port scan may trigger two types of alert, 1) Httprecon Web Server Fingerprint Scan, and 2) ET SCAN NMAP -sO. At a higher level, these alerts can be categorized into attempted-recon (since both reflect potential reconnaissance efforts by the attacker), as is the case in the Emerging Threats Ruleset of Suricata. A defender can choose different granularities of attack categorization to map the IDS alert types into the abstracted types in our proposed model based on individual needs. Besides categorization, the defender can also make use of other attributes in the IDS alerts to aid in abstracted type assignment. For example, a port scan on the enterprise file server can be assigned to the abstracted type of high-risk-recon, while a port scan on employee desktop can be assigned to attempted-recon.

In addition to the alerts corresponding to an actual attack action, normal user behavior can generate false positive alerts. For example, a user who is scraping the web for weather data monitoring may trigger the ET POLICY POSSIBLE Web Crawl using Curl, which is grouped into the attempted-recon type by the same Emerging Threats Suricata ruleset. Leveraging the proposed game-theoretic model on these abstracted alerts, it is possible for the defender to devise an optimal defense policy for a wide range of alert applications even in the face of possible false positives.

III. GAME THEORETIC MODEL OF ROBUST ALERT PRIORITIZATION

We now turn to the proposed approach for robust alert prioritization. We model the interaction between the defender and attacker as a zero-sum game, which allows us to define and subsequently compute robust stochastic inspection policies for the defender. In this section, we formally describe the game model. We then present the computational approach for solving it in Section IV.

The game has two players: the defender (denoted by v = +1) and the adversary (denoted by v = -1). Each player's strategies are policies, that is, mappings from an observed ADE state to the probability distribution over actions to take in that state. In a given state, the defender chooses a subset of alerts to investigate; thus, the defender's set of possible actions is the set of all alert subsets that satisfy the constraints (2) and (3). The attacker's choices in a given state correspond to subsets of actions A to take. Consequently, the set of adversary's actions is the set of all subsets of attacks satisfying constraint (1). Note that the combinatorial nature of both players' action spaces and of the state space makes even representing deterministic policies non-trivial; we will deal with this issue in Section IV. Moreover, we will consider stochastic policies. An equivalent way to represent stochastic policies is as probability distributions over deterministic policies, which map observed state to a particular action (subset of alerts for the defender, subset of attacks for the adversary). Henceforth, we call deterministic policies of the players their pure strategies and stochastic policies are termed mixed strategies, following standard terminology in game theory. 4Let π -1 denote the attacker's policy, which maps the fully observed state of ADE, O +1,t } t∈T are the counts of alerts chosen to be investigated for each type t. Now, notice that all alerts of type t are equivalent by definition; consequently, it makes no difference to the defender which of these are chosen, and we therefore choose the fraction

of alerts of type t uniformly at random.

Let Π v be player v's set of pure strategies, where each pure strategy π v ∈ Π v is a policy as defined above. A mixed strategy of player v is then a probability distribution σ v = {σ v (π v )} πv∈Πv over the player's pure strategies Π v where σ v (π v ) is the probability that player v uses policy π v . Since a mixed strategy σ v is a distribution over a finite set of pure strategies, it satisfies 0 ≤ σ v (π v ) ≤ 1 and πv∈Πv σ v (π v ) = 1. Let Σ v denote the set of all mixed strategies of player v.

For any strategy profile of the two players, (π v , π -v ), we denote the utility of each player v by

Similarly, the expected utility of player v when it chooses the mixed strategy σ v ∈ Σ v and its opponent play the mixed strategy σ

Next, we describe how to compute the utility of player v, U v (π v , π -v ), when its policy is π v and the opponent's policy π -v are given.

Consider arbitrary pure strategies of both players, π +1 and π -1 . The game begins with an initial system state N (0) , M (0) , S (0) = 0, 0, 0 . The system state is then updated in each time period k as follows:

1) Alert investigation. The defender first investigates a subset of alerts produced thus far. Specifically, the defender chooses the number of alerts of each type to investigate {α

where C(n, r) is the number of possible combinations of r objects from a set of n objects. p (k) a is then the probability that attack a is not detected by the defender.

2) Attack generation. The adversary produces attacks by executing actions according to its policy {α

-1,a for each a ∈ A.

3) Triggering alerts. Each attack a ∈ A can trigger alerts as follows. For each attack a ∈ A and alert type t ∈ T , if

= n with probability P a,t (n) for n ≥ 0. This probability can be estimated, for example, by feeding inputs which include representative attacks into an attack detector and observing relative frequencies of alerts that are triggered. In addition, false alerts are generated according to the distribution F t , which we can estimate from data of normal behavior and associated alert counts. Let f (k) t be the number of false alerts of type t ∈ T that have been generated. Then the total number of alerts in the next time period

In order to define the reward received by the defender in time period k, we make the following assumption: if any of the alerts raised by an attack is chosen to be inspected, then the attack is detected; otherwise, the attack is not detected. Let L a be the loss incurred by the defender when an attack a ∈ A is not detected. Then the reward of the defender obtained in time period k is

For an arbitrary pure strategy profile of the defender and adversary, (π +1 , π -1 ), the defender's utility from the game is the expected total discounted sum of the reward accrued in each time period:

where τ ∈ (0, 1) is a temporal discounting factor which implies that future rewards are less important than current rewards. That is, imminent losses are more important to the defender than potential future losses. The adversary's utility is then

Our goal of finding robust alert investigation policies amounts to computing a mixed-strategy Nash equilibrium (MSNE) of our game by the well-known equivalence between MSNE, maximin, and minimax solutions in zero-sum games [17]. A mixed-strategy profile (σ * v , σ * -v ) of the two players is an MSNE if it satisfies the following condition for all v ∈ {+1, -1}

That is, each player v chooses a stochastic policy σ * v that is the best response (is optimal for v) when its opponent chooses σ * -v .

IV. COMPUTING ROBUST ALERT PRIORITIZATION POLICIES

A. Solution Overview

For given sets of policies, Π +1 and Π -1 , a standard approach to computing the MSNE of a zero-sum game is to solve a linear program of the following form:

) where in our case the optimal solution σ * +1 yields the robust alert prioritization policy for the defender. However, using this approach for our problem entails two principal technical challenges: 1) the space of policies for both players is intractably large, and 2) it is even intractable to explicitly represent individual policies, since they map a combinatorial set of states to a combinatorial set of actions for both players.

We propose an adversarial reinforcement learning approach to address these challenges, which combines a double oracle framework [25] with neural reinforcement learning. The general double oracle approach is illustrated in Figure 2. We start with an arbitrary small collection of policies for both players, (Π +1 , Π -1 ), and solve the linear program (10), obtaining provisional equilibrium mixed strategies (σ +1 , σ -1 ) of the restricted game. Next, we query the attack oracle to compute the adversary's best response π -1 (σ +1 ) to the defender's equilibrium mixed strategy σ +1 , and, similarly, query the defense oracle to compute the defender's best response π +1 (σ -1 ) to the adversary's equilibrium mixed strategy σ -1 . The best response policies are then added to the policy sets (Π +1 , Π -1 ) of the players, and we then re-solve the linear program and repeat the process. The process stops when neither player's best response policy yields appreciable improvement in utility compared to the provisional equilibrium mixed strategy. Since the space of possible policies in our case is infinite, this process may not converge. However, in our experiments the procedure converged in fewer than 15 iterations (see Figure 12 in Appendix B), with the fast convergence in part due to the way we represent policies, as discussed below. The main question that remains is how to compute or approximate the best response oracles for both players. To this end, we use reinforcement learning techniques with policies represented using neural networks. Below, we explain both our double oracle approach and our neural reinforcement learning methods (including the specific way in which we represent policies) in further detail.

B. Policy-based Double Oracle Method

As displayed in Figure 2, our game solver is an extension of the double oracle algorithm proposed in [36] and is partitioned into four parts: a policy container, a linear programming (LP) optimizer, a defense oracle, and an attack oracle. The policy container stores the policies of the two players, Π +1 and Π -1 , as well as a utility matrix U , whose elements are U +1 (π +1 , π -1 ) for all π +1 ∈ Π +1 and π -1 ∈ Π -1 . The LP optimizer solves the game by computing the current mixedstrategy Nash equilibrium given the utility matrix U . The defense and attack oracles are agents that apply reinforcement learning to compute the optimal responses to their opponents' mixed strategies, which are provided by the LP optimizer.

Our solver works in an iterative manner such that the players' policies and the utility matrix grow incrementally. Initially, Π +1 , Π -1 can be set up with some basic policies, for example, uniformly allocating each player's budget among their choices. Then, the policy sets, jointly encapsulated in a policy container, are updated in each iteration as follows:

1) First, the LP optimizer computes the mixed-strategy Nash Equilibrium (σ

) of the current iteration by solving the optimization problems presented in Equation (10).

2) The oracle of player v computes the best response policy π ′ v given that its opponent uses its equilibrium mixed-strategy σ ′ -v , for v ∈ {+1, -1}.

) for all v ∈ {+1, -1}, the double oracle algorithm terminates and returns (σ ′ +1 , σ ′ -1 ) as the approximate MSNE. Otherwise, add π ′ v to the corresponding Π v , update the utility matrix U and continue from Step 2. adversary's budget over attack actions; and Greedy, a policy which allocates the budget to attacks in the order of expected adversary utility. Specifically, the Greedy adversary prioritizes the attack actions according to L a • min{ D ca , 1}, where D is the available attack budget, adding actions in this priority order until the adversary's budget is exhausted.

We first conduct our experiments by assuming that the defender knows the adversary's capabilities. Subsequently, we evaluate the robustness of our approach when the defender is uncertain about the adversary's capabilities, and use it to provide practical guidance. We also provide results on the computational cost of our approach in Appendix B.

B. Case Study I: Intrusion Detection

Our first case study involves a signature-based intrusion detection scenario, using the Suricata, a state-of-the-art open source intrusion detection system (IDS), combined with the CICIDS2017 dataset. Our case study evaluates our alert prioritization method in two cases: i) the defender has full knowledge of the adversary; and ii) the defender is uncertain about the adversary's capabilities.

1) CICIDS2017 dataset: The CICIDS2017 dataset [33] records benign and malicious network flows in pcap format, captured in a real-world network between 07/03/2017 and 07/27/2017. The network consists of 10 desktops belonging to regular users and 5 laptops owned by attackers. The desktops are used to generate natural benign background traffic by using a profile system that abstracts the behaviors of regular users. The laptops are employed to produce malicious traffic of the following classes of attacks: Brute Force, Botnet, DDoS, DoS, Heartbleed, Infiltration, Portscan, and Web Attack.

2) Suricata IDS: We employ Suricatafoot_1 to conduct our case study on the CICIDS2017 dataset. Suricata is an open-source network intrusion detection system which performs analysis of passing traffic on a network by using a set of signatures (also called rules). If a traffic pattern matches any of the signatures, then a corresponding alert is triggered and sent to the network administrator.

A Suricata signature contains the following parts: action, header, rule options, and priority. Action describes the operation of Suricata when a signature is matched, which can be either dropping a packet or raising an alert. Header defines the protocol, port, and IP addresses of the source and destination in a signature. Rule options include a list of keywords, for example, the corresponding alert type associated with a priority. Finally, the priority keyword comes with a numerical value ranging from 1 to 255 where 1 indicates the highest priority and 255 the lowest.

In our experiments, we use Suricata to scan the pcap files in the CICIDS2017 dataset. Specifically, we use the Emerging Threats Ruleset (ETR) 6 to analyze the network traffic in the dataset. ETR defines a total of 33 alert types, and we select the 10 most common alert types exhibited during our experiments, which are shown in Table III.

3) Experimental Setup: We use the following steps to set up our experiments for the case study. First, we used 30 minutes as the fixed length of each time period. Then, we utilized the Suricata IDS to scan and detect intrusions for both malicious and benign traffic in the CICIDS2017 data. By doing so, we obtained the number of alerts of each type raised by each attack action, as well as the number of false alerts in each time period. In the preprocessing step we pruned alert types that were triggered only by malicious traffic, as discussed in Section IV-D. As a result, we were left with 7 out of the 10 alert types to consider using our full adversarial RL framework. In addition, we filtered out the attack actions that do not raise any alerts, since those attacks will never be detected using Suricata, leaving 7 out of 8 representative attacks for our experiments. The final attack actions and alert types that we use in the experiments are given in Table IV.

We used Poisson distribution to fit the distribution of alerts raised by benign traffic in each time period. Since the benign traffic in the CICIDS2017 dataset was captured from only 10 desktop which is far less than the number of computers in a real-world local area network, we amplified the corresponding mean of each type of alert by a factor of 100. The resulting average numbers are shown in Table V. We set the cost of investigating each alert to 1.0 (i.e., equal for all alerts).

C. Alert Management and Prioritization

A multitude of research efforts have studied the problem of reducing the number of alerts without significantly reducing the probability of attack detection [16]. One of the most common approaches is alert correlation and clustering, which attempt to group related alerts together, thereby reducing the set of messages that are presented [30]. In distributed systems, collaborative intrusion detection systems may be deployed, which include several monitoring components and correlate alerts among the monitors to create a holistic view [38]. Since the number of alerts may be too high even after correlation, research efforts have also investigated the prioritization of alerts. For example, Alsubhi et al. introduced a fuzzy-logic based alert management system, called FuzMet, which uses several metrics and fuzzy logic to score and prioritize alerts [2]. However, these approaches do not consider the possibility of an attacker adapting to the prioritization.

D. Game Theory for Alert Prioritization and Security Audits

Prior work has successfully applied game theory to a variety of security problems, ranging from physical security [3] to network security and privacy [24].

Our approach is most closely related to alert-prioritization games. Laszka et al. introduced the first game-theoretic model for alert prioritization, which they solved with the help of a greedy heuristic [19]. The performance of this approach, which we denoted GAIN in our experiments, is limited by its restrictive assumptions about the defender's decision making. In particular, GAIN assumes that the defender's policy is a strict prioritization that investigates all higher-priority alerts before investigating any lower-priority ones, and the prioritization is chosen before observing the actual number of alerts. Moreover, the model considers only a single time slot, which further limits its usefulness. Yan et al. improved upon GAIN by allowing the defender to specify a maximum budget that may be spent on each alert types, thereby relaxing the strict prioritization of GAIN [43]. However, this improved approach, which we denoted RIO in our experiments, still assumes that the prioritization is chosen before observing any alerts and considers only a single time slot. As our numerical results demonstrate, these restrictions can lead to significantly higher losses for the defender. Schlenker et al. introduced a similar model, called Cyber-alert Allocation Game, which further simplifies the problem by assuming that the number of false alerts is fixed and known by both parties in advance [32].

Our approach also resembles audit games, which study the problem of allocating a limited amount of audit resources to a fixed number of audit targets [6], [7]. However, despite the resemblance, audit games are ill-suited for prioritizing alerts since these games assume that the attacker knows the exact set of targets, which would correspond to individual alerts, before launching its attack. Due to the unpredictability of false alerts, this assumption does not hold for alert prioritization.

VII. DISCUSSION AND CONCLUSION

Since even after applying techniques for reducing the alert burden (e.g., alert correlation) there often remain vastly more alerts than time to investigate them, the success of detection often hinges on how defenders prioritize certain alerts over others. In practice, prioritization is typically based on nonstrategic heuristics (e.g., Suricata's built-in priority values), which may easily be exploited by a strategic attacker who can adapt to the prioritization. Strategic prioritization approaches attempt to prevent this by using game-theory to capture adaptive attackers; however, existing strategic approaches severely restrict the defender's policy (e.g., strict prioritization) for the sake of computational tractability.

In contrast, we introduced a general model of alert prioritization that does not impose any restrictions on the defender's policy, and we proposed a novel double oracle and reinforcement learning based approach for finding approximately optimal prioritization policies efficiently. Our experimental results-based on case studies of IDS and fraud detectiondemonstrate that these policies significantly outperform nonstrategic prioritization and prior game-theoretic approaches. Further, to demonstrate the strength of our attacker model, we also showed that the attacker policies found by our approach outperform multiple baseline policies.

For practitioners, the key task in applying our approach is estimating the parameter values of our model. In our case studies, we showed how to estimate parameters in two domains (e.g., for IDS, using CVSS score to estimate attack impact and CVSS complexity for attack cost). The most difficult parameter to estimate is the attacker's budget; however, our experimental results show that our approach is robust to uncertainty in the attacker's budget and outperforms other approaches even when the budget is misestimated. We leave studying the sensitivity to other parameters to future work.