In many AI+human applications of behavior change, AI agents assist the human in performing frictionful tasks, where making progress toward the human's goal requires sustained effort over This work is licensed under a Creative Commons Attribution International 4.0 License.
Proc. of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024), N. Alechina, V. Dignum, M. Dastani, J.S. Sichman (eds.), May 6 -10, 2024, Auckland, New Zealand. © 2024 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org).
time with little immediate gratification. Examples include physical therapy (PT) programs, adherence to scheduled medication, or passing an online course. Two key challenges for AI agents in these settings are rapid personalization [26,35,43] and learning interpretable policies for intervention [41,44]. In frictionful tasks, since effort exerted by the human does not reap immediate benefits, the AI agent must learn a personalized intervention policy for each human in a small number of interactions, or risk disengagement. These policies must also be interpretable to experts in behavioral science so that they can discover which interventions work for which individuals, and investigate why.
Grounded in behavioral literature that treats humans as sequential decision-makers (e.g. [24,33,37,38,50]), we model the human as an agent planning under a "maladapted" Markov Decision Process (MDP). In maladapted human MDPs, the optimal policy does not reach the human's stated goal; for example, in physical therapy (PT), the goal may be a rehabilitated shoulder and the maladapted MDP parameter may be an extremely low discount rate, Ą. This results in myopic decision-making, wherein an individual forgoes the long-term goal (rehabilitated shoulder) to avoid experiencing friction in the short-term (unpleasantness of PT). The AI agent helps the individual achieve their long-term goal by changing the maladapted human MDP (and thereby the optimal policy).
While there is existing reinforcement learning (RL) literature for optimizing interventions on human utility functions (i.e. reward) in maladapted MDPs [19,46,50], interventions on Ą have not been optimized from an RL perspective. On the other hand, in behavioral science, humans have been observed to use a problematically low Ą [34] and scientists have developed interventions to change a human's Ą (e.g. [17]). However, no work optimizes for when and with what mechanisms to intervene on the parameters of the human's maladapted MDP.
In this paper, we introduce a flexible and behaviorally interpretable framework called "Behavior-Model RL" (BMRL). In BMRL, the human is modeled as an RL agent, whose actions are behaviors, such as performing or skipping PT; the AI agent provides personalized assistance by delivering interventions on the human's maladapted MDP parameters. By linking the behaviors of our human agents to their MDP parameters, BMRL allows us to interpret the mechanism behind the human's maladapted decision-making. Our framework is also more flexible than existing ones since we allow the AI agent's actions to include operations on any part of the human MDP (such as Ą). By solving for the AI agent's optimal policy, we learn the best set of interventions to change the human agent's behavior and to help the human reach their goal.
Unfortunately, current RL approaches have two major drawbacks when used to solve for the optimal AI agent policy in BMRL. First, most planning methods are too data-intensive for our setting, in which personalization occurs online. For example, online algorithms in robotics require thousands of interactions to learn reasonable policies (e.g. in [39,40,45]), but in frictionful tasks, we are limited to tens to hundreds of interactions [41]. Second, existing planning methods model the human as a black-box transition or value function. Unfortunately, in learning black-box representations of the human agent, we lose the ability to interpretably attribute human behavior to their MDP parameters.
In this paper, we propose a tractable planning method for the AI agent in our BMRL framework. Our method provides the AI agent with a useful inductive bias, in the form of a human model that captures important behavioral patterns in frictionful tasks. Specifically, we identify a small, behaviorally grounded model of the human that the AI agent can leverage to rapidly personalize interventions, including previously under-explored interventions on Ą. Then, we introduce the concept of "AI equivalence" to identify a class of more complex human models for which AI policies learned in our simple human model can be lifted with no loss of performance. In our empirical analysis, we test whether AI planning with our small model is robust to complex human models that are not covered by our equivalence result. Throughout all of this, our small model preserves scientific interpretability-in fact, it has an analytical solution for the human behavior policy-which allows experts to inspect and learn from the AI policies.
Computational modeling of human behaviors. Behavioral scientists have developed and verified several computational models of dynamic human decision-making. Unlike static models, such as Social Cognitive Theory [2], dynamic models of decision-making apply to interactive human-AI settings, since they capture person-level variation and changes over time, as in Zhang et al. [47]. Scientists developed these models to explain o ine data from frictionful settings such as health (e.g. [18,43,48]), energy [20], and experience sampling [14] or to capture broader behaviors such as risk [16] and adherence [27]. However, these models involve too many latent variables-corresponding to internal human processes-to facilitate rapid AI learning from online data. In contrast, we propose a minimal, behaviorally grounded model, one whose set of latent parameters is small and structured enough that our AI can learn.
Computational modeling of human agent deficiencies. RL is frequently used to model the complex mechanisms underlying human behavior, from the firing of dopaminergic neurons in the brain (e.g. in [24,33]) to frictionful tasks such as mindful eating [37], weight loss [1], and smoking cessation [38]. Although these works use RL to model humans, the models themselves are not used to enrich planning for an AI agent. One exception is inverse reinforcement learning, in which the AI agent infers the human agent's rewards (e.g. [3,49]), transitions [30], discount factor [11], or entire MDP [7,13,32], but does not intervene on the parts of the human MDP Human Environment AI Agent Human Agent AI action a AI Human action a h r' h s' h r' AI Human reward r h Human state s h AI reward r h AI Environment that are maladapted. When the AI agent does intervene, the changes are limited to the human's reward [19,36,46,50] or states [6,31]. Our BMRL framework is flexible enough to incorporate AI interventions on multiple parts of the human MDP, including the discount factor or transitions.
Equivalence of (human) MDP models. In RL, there are notions of equivalence that can reduce larger human MDPs to smaller, more manageable ones. Equivalence, as defined in bisimulations [10], homomorphism [28], and approximate homomorphisms (e.g. [29,42]), requires that one human MDP strictly preserves the transition and reward functions of another, given a mapping between the state and action spaces. State abstraction methods, which equate the optimal value function between two human-level MDPs, are less strict [15]. However, these equivalences are still stricter than necessary in our setting, where we only care that the human MDPs are similar enough that the AI agent policy will not differ. Furthermore, the simpler MDPs recovered by these methods are not guaranteed to be behaviorally valid or interpretable. In our approach, we define two human MDPs as equivalent if they lead to the same AI optimal policies, and we use this definition to build up to more complex human MDPs from a behaviorally interpretable one.
We define a formal framework, called BMRL, in which an AI agent learns to intervene on a human agent's maladapted MDP parameters (overview in fig. 1).
In BMRL, human agents perform optimal planning on (subconscious) knowledge of their MDP,
where ĩ ĝ , ĩ Ě ∈ S ℎ are absorbing goal (e.g. a rehabilitated shoulder) and disengagement states (e.g. quitting PT).
Though in general, it is possible for the human's perception of the states S ℎ , actions A ℎ and transitions Đ ℎ to be maladapted, in this paper we assume that the human's perception matches the true environment. On the other hand, we allow the human's rewards Ď ℎ and discount Ą ℎ to vary by perception. For example, one may skip PT because of a tendency to ignore long-term rewards (low Ą ℎ ) while another may skip PT because they find the workout to be extremely unpleasant (bad Ď ℎ ).
We assume that at any point the human subconsciously "knows" their own MDP, solves for the optimal policy, and uses it to select actions. In future work, BMRL can extend to sub-optimal human planning. Despite being optimal, our human agents are still boundedly rational because their MDP is maladapted. That is, under certain values of Ą ℎ , Ď ℎ , even an optimal human policy will never lead to the goal state (e.g. if the path to the goal reward is laced with extremely negative rewards). The existence of maladapted MDPs in humans is shown in behavior science, where myopic discounting has been linked to excessive alcohol intake [34] or miscalibrated rewards have been linked to unhealthy eating [37]. Despite subconscious knowledge of their own MDP, our human agents are still boundedly rational because (1) they may not be conscious of their deficiencies and unable to target them; (2) even if aware, they may still struggle to change their deficiencies. In both cases, behavioral interventions (delivered by the AI agent) can help.
Our AI agent encourages the human agent toward the goal by intervening on the human's decision-making parameters, such as Ą ℎ . To do so, the AI agent plans according to an MDP,
with known rewards Ď ýą and unknown transitions Đ ýą . Upon observing state ĩ ýą = [ĩ ℎ , ė ℎ ], which consists of the human's current state and previous action, the AI agent must decide whether to intervene on the human's discount (ė ýą = ė Ą ), reward (ė ýą = ė Ď ), or to do nothing (ė ýą = 0). In practice, a discounting intervention ė Ą could be "episodic future thinking," where individuals imagine future events as if they are presently occurring [4]; this could executed as a guided activity in-app. A common intervention on reward ė Ď is to offer extrinsic rewards, such as badges [8]. Domain experts would determine how the interventions are executed, e.g. if the burden intervention should be a badge, motivational message, or cash.
To encourage policies that quickly lead to the goal state, the AI agent receives a positive reward when the human reaches the goal state, a negative reward when the human disengages, and a negative reward for the "cost" of intervening. The AI's transitions factorize into two probability distributions,
The first distribution refers to the human-level transitions Đ ℎ . The second distribution is over human actions; it is the human policy that results from the AI's intervention on the human's MDP. Importantly, we assume that the effect of AI actions on the human MDP is temporary. For example, if the AI agent increases the human's discount factor Ą ℎ to Ą ′ ℎ in the current time step, the human's discounting will have reverted to Ą ℎ at the next time step.
In table 1, we provide a comparison on what the AI and human agents separately know and observe. Note that all of the AI agent's unknown parameters pertain to the human MDP M ℎ and are contained in the AI's transitions Đ ýą . Instead of explicitly learning M ℎ to form Đ ýą , we could directly estimate Đ ýą or č * ýą using standard model-based or model-free techniques. However, by learning M ℎ , we take advantage of the known structure of the problem; the better the AI's model of M ℎ , the better the inductive bias for forming Đ ýą (and therefore ÿ * ýą ).
In this section, we define chainworlds, a class of simple human MDPs that the AI agent will use as a stand-in model for the true human decision-making process. Chainworlds are based on the observation that many frictionful tasks contain a notion of human progress toward a goal; for example, in PT, the progress toward a rehabilitated shoulder may be summarized by the current strength of the joint. We summarize these "progress-based" settings with a "progress-only" class of human MDPs, shown in fig. 2, which we call chainworlds and denote ℳ chain . Each element of ℳ chain is as follows:
and represent progress toward the goal. The goal state at the end of the chain, ĩ Ċ = ĩ ĝ means that the human has rehabilitated their shoulder. The disengagement state ĩ Ě means that the human has disengaged from PT. • Actions ė ℎ ∈ {0, 1}. The human decides to perform (ė ℎ = 1) or not perform (ė ℎ = 0) the goal-directed behavior. In the future, this could be extended to categorical actions. That said, many important applications have binary actions, such as "exercise or not" in PT, "smoke or not" in smoking cessation, and "adhered or not" in medication adherence. • Rewards. The human's utility function is the reward,
Goal behaviors, such as doing PT, incur a cost representing burden Ĩ Ę < 0. Similarly, losing progress incurs Ĩ ℓ < 0. The goal and disengagement states have positive utility, Ĩ ĝ > 0 and Ĩ Ě > 0. • Transitions. The human knows that there is Ħ ĝ probability that they will move toward the goal as a result of the behavior, Ħ ℓ probability that they will lose progress from abstaining, and Ħ Ě probability that they will disengage from abstaining. These probabilities are fixed across states, except for the first state ĩ 0 , which has a separate probability of disengagement Ħ Ě0 g Ħ Ě .
• Discount. The human exponentially discounts future rewards via Ą ℎ ∈ [0, 1). We leave other behaviorally relevant forms of discounting, such as hyperbolic discounting [9], as future work. • Effect of AI interventions. When ė ýą = ė Ą the human's discount Ą ℎ increases by Ą > 0, and when ė ýą = ė Ę the human's burden Ĩ Ę < 0 decreases by Ę . We clip Ą ℎ + Ą to be between 0 and 1.
Each individual is an instance of the chainworld,
For example, some people tend to prioritize short-term rewards (with a low Ą ℎ ) while others prioritize long-term rewards (with a high Ą ℎ ). The parameters ĉ must be inferred by the AI.
Closed-form Solutions for Human Policies in Chainworlds. Chainworlds are inspectable to behavioral experts because there is an analytical solution for the optimal value function (all derivations in appendix A [25]). For a chainworld MDP M ĉ ∈ ℳ chain , the optimal value function maximizes between the value of a policy that always pursues the goal, ÿ ĝ (ĩ Ĥ ) = 1, and the value of a policy that always chooses to disengage, ÿ Ě (ĩ Ĥ ) = 0, where ĩ Ĥ for Ĥ ∈ 0, . . . , Ċ refers to the Ĥ-th state on the chain. The value of goal pursuit is,
where
The value of goal pursuit, Ē ÿ ĝ ĉ (ĩ Ĥ ), trades off between the long-term utility of the goal (the Ĩ ĝ term) and the burden one accumulates to get there (the Ĩ Ę term). The value of disengagement is,
where
The first term in the equation (with Ĩ Ě ), represents the value of disengagement from state 0, after having lost all prior progress. The second term represents the value of disengagement after state 0, which factors in the cost of disengagement Ĩ Ě and of losing progress Ĩ ℓ . These equations allow us to hypothesize about the diverse space of AI actions that will encourage the human towards the goal, such as actions to increase the human's level of motivation (increasing Ĩ ĝ ) or that highlight the consequences of quitting (decreasing Ĩ Ě ).
At this point, we have fully specified an AI MDP as defined in section 3.2, in which the human MDP is a chainworld M ĉ ∈ ℳ chain . Solving this AI MDP will yield an optimal AI policy, which is the best intervention plan for a given human with parameters ĉ . Importantly fig. 3 demonstrates that personalization is necessary because humans with different ĉ require different optimal AI policies.
In this section, we define an equivalence class of more complex human MDPs for which an AI agent that plans with the chainworld can still learn the optimal policy. Definition 5.1 (AI equivalence of human MDPs). We consider two human MDPs M ℎ ≡ M ℎ under state mapping Ĝ : S ℎ → S ℎ and action mapping ĝ ĩ : A ℎ → A ℎ if the corresponding optimal AI policies are equal, so that ÿ
The state mapping Ĝ and (state-specific) action mapping ĝ ĩ let us map from the state and action space of one MDP to the other. In terms of the chainworld, our definition states that if the optimal AI action in the chainworld MDP is the same as the optimal AI action in the true MDP for all states (after applying the mappings), then the two are equivalent.
Our equivalence in definition 5.1 is not as strict as the homomorphisms equivalence. Unlike homomorphisms, we do not seek human MDPs that have the same rewards and transitions as chainworld. In fact, we do not even seek MDPs that result in the same optimal human policy as chainworld. Instead, we only care that the two human MDPs are similar enough to result in the same optimal AI policy. As a result, we get the largest set of human MDPs that admits simple planning of optimal interventions by the AI agent.
Under definition 5.1, the class of MDPs that is equivalent to chainworlds is determined by the space of AI policies that chainworlds can express. In this section, we show that all chainworld MDPs M ĉ ∈ ℳ chain result in AI optimal policies that follow a "threewindow format, " which we refer to as Π. Throughout this section, we describe the AI policy in terms of the chainworld states, ĩ Ĥ , where Ĥ refers to Ĥ-th state on the chain; even though the previous human actions are technically part of the AI state, they do not affect the best current action in the AI's optimal policy.
A "three-window" AI policy consists of: window 1 (no intervention is effective enough to make human perform the behavior), window 2 (intervention window), and window 3 (human performs behavior without intervention). Two examples are in fig. 3. The size of these windows varies and may even be 0. For example, if the interventions have no effect ( Ą = 0, Ę = 0) then the intervention window will be size 0. The three windows are a consequence of how the AI's action affects the human's optimal policy; when the AI agent intervenes on the human, it changes the human's MDP parameters, which in turn, might change the human's optimal policy.
To succinctly describe the human's optimal policy, we introduce "human thresholds" Ī in definition 5.2; when the human is in a state past the threshold, their optimal policy is to pursue the goal. A human with a smaller threshold Ī will pursue the goal from farther away. An effective AI action moves the threshold Ī to a state preceding the human's current state, so that the human chooses to move.
For a chainworld M ĉ ∈ ℳ chain , define Ī ∈ {0, . . . , Ċ -1} as the threshold where ÿ * ĉ (ĩ Ĥ ) = 0 for Ĥ f Ī and ÿ * ĉ (ĩ Ĥ ) = 1 for Ĥ > Ī. Even if the AI agent can intervene to prompt the human toward the goal, whether or not the optimal AI does intervene depends on the configuration of the AI rewards. If intervening has negligible cost, then the AI agent will intervene as soon as it is able. On the other hand, if there is a high cost, then the AI agent will wait until the human is closer to the goal, to minimize the total number of interventions needed. We define AI threshold Ī ýą below, as the point at which the reward of reaching the goal outweighs the cost of interventions required to reach it: Definition 5.3 (AI threshold). For a human chainworld M ĉ ∈ ℳ chain and AI MDP M ýą , define AI threshold Ī ýą ∈ {0, . . . , Ċ -1} as the chainworld state in which the value of the goal is greater than the value of disengagement. For states ĩ Ĥ where Ĥ > Ī ýą , the AI values are Ē
ýą (ĩ Ĥ ), and for states where Ĥ f Ī ýą , the AI values are Ē
ýą . The human and AI thresholds define the intervention windows for the AI policy in theorem 5.4.
Theorem 5.4 (Chainworld AI policies). Suppose we are given:
• An AI MDP M ýą = ïS ýą , A ýą ,Đ ýą , Ď ýą , Ą ýą ð, where the actions are to do nothing (ė ýą = 0), intervene on the discount (ė ýą = ė Ą ), or to intervene on burden (ė ýą = ė Ę ) • A human MDP M ĉ ∈ ℳ chain , which results in human thresholds Ī 0 ℎ , Ī Ą ℎ , and Ī Ę ℎ under AI actions 0, ė Ą , and ė Ę , respectively Let Ī min ℎ = min Ī 0 ℎ , Ī Ą ℎ , Ī Ę ℎ denote the earliest human threshold as a result of any AI action. Let Ī ýą denote the AI intervention threshold, as in definition 5.3. Then, the optimal AI policy is,
and ÿ * ýą belongs to the three-window policy class, Π. The proof is in appendix B.1 [25]. Note that if both ė Ę and ė Ą are valid options in the intervention window (when Ī ýą < Ĥ f Ī 0 ℎ ), then the AI agent will prefer the less expensive intervention. Theorem 5.4 shows that every chainworld results in an optimal AI policy belonging to Π. Theorem 5.5 shows the reverse; for any human MDP whose corresponding AI policy is ÿ ýą ∈ Π, there exists a chainworld MDP whose AI policy is also ÿ ýą .
Proof in appendix B.2 [25]. Theorem 5.5 means that any human MDP that results in a three-window AI policy-that is, consists of three regions: impossible to help, can be helped by the AI, and does not need help-belongs to the chainworld equivalence class. In section 6, we will show that the AI agent can plan interventions using chainworld as a substitute for another human MDP in the same class, without any loss in performance.
Ultimately, we care that the chainworld equivalence class contains realistic models of humans that align with the behavioral literature.
In this section, we provide examples of human MDPs that capture a meaningful behavior not covered by chainworlds, yet whose optimal AI policy is still in the equivalence class Π.
Monotonic chainworlds. In monotonic chainworlds, the closer one gets to the goal, the higher the relative value of pursuing it.
Definition 5.6 (Monotonic chainworlds). For a monotonic chainworld M, the value of goal-pursuit increases closer to the goal:
For example, consider chainworlds in which the probability of disengagement Ħ Ě decreases the closer the agent is to the goal (the human is less likely the quit the closer they are to recovery). Monotonic chainworlds relate to the goal-gradient hypothesis, which states that motivation to reach a goal increases with proximity [23]. In appendix C.1 [25], we prove that all monotonic chainworlds are AI equivalent to our chainworld.
Progress worlds. Progress worlds, while potentially multi-dimensional, have a one-dimensional notion of progress. Definition 5.7 (Progress worlds). Suppose M is a Ā dimensional, path-connected graph with an absorbing goal state ĩ ĝ , an absorbing disengagement state ĩ Ě , and actions that allow movement between states on the graph. Let Ě (ĩ, ĩ ′ ) denote the shortest graph distance from ĩ to ĩ ′ . M is a progress world if Ě (ĩ, ĩ Ě ) = Ě (ĩ ′ , ĩ Ě ) and Ě (ĩ, ĩ ĝ ) = Ě (ĩ ′ , ĩ ĝ ) for all pairs of ĩ, ĩ ′ ∈ S.
In our PT example, "progress" may depend on a combination of metrics such as joint strength, the ability to perform daily tasks, and so on. We show in appendix C.4 [25] that worlds in which states can be mapped to a one-dimensional distance are equivalent to our chainworlds. This type of equivalence is simple yet useful, as it lets us reduce high-dimensional worlds to a single dimension of interest. Definition 5.7 restricts us to graphs in which all shortest paths between the disengagement and goal state are of the same length. Intuitively, this means that a single chainworld can represent all paths (and therefore, the entire world). Though not all graphs are progress worlds, in our empirical experiments, we test the chainworld AI's robustness to graphs that break this definition.
Multi-chain worlds. In multi-chain worlds, there is a principle dimension that corresponds to progress toward the goal (as in our simple chainworld) but there may be several additional dimensions associated with different ways of dropping out.
A multi-chain world M consists of ÿ chains, each of length Ċ ę . The first chain, ę = 0, is the goal chain; when the human reaches the end of this chain, they have reached the goal. The remaining chains, ĩ 1 , . . . , ĩ ÿ -1 , are disengagement chains; when the human reaches the end of any of these chains, they disengage. When ė = 1, the human moves along the goal chain with probability Ħ 0 while staying still in the disengagement chains. When ė = 0, the human stays still in the goal chain and (independently) moves along each of the ę disengagement chains with probability Ħ ę .
In our PT example, the principle chain might still correspond to the overall strength of the joint as a measure of progress toward recovery. Additional chains, corresponding to the level of motivation, level of pain, etc., may all represent mechanisms that cause disengagement. This form of multi-chain reflects how disengagement is described in the behavioral literature (e.g. [21,22]). In appendix C.5.1 [25], we show equivalence to multi-chain worlds whose disengagement chains are of length 2, which corresponds to realworld situations in which one of many factors can abruptly trigger disengagement at any point (e.g. the PT patient is re-injured).
Negative effect worlds. These are chainworlds in which the AI intervention has the opposite intended effect on the human. Definition 5.9 (Negative effect worlds). A negative effect world M is defined exactly as the chainworld, except that Ą < 0 (AI intervention on discount Ą ℎ decreases it) or Ę > 0 (AI intervention on burden Ĩ Ę increases it).
The efficacy of a behavioral intervention is known to vary by individual (e.g. [5]). In appendix C.2 [25], we prove that negative effect worlds result in AI policies that correspond to chainworlds where the intervention has no effect (i.e. Ą = 0 and Ę = 0).
We test how AI planning using chainworld benefits performance, especially as we remove our assumptions and make the true human model dissimilar to chainworld.
All experiments are over 200 trials of 15 episodes each, and each trial corresponds to a human whose MDP parameters ĉ are sampled. Not all settings of ĉ correspond to individuals that can reach their goal-for example, consider a human whose burden is so high that no AI intervention can make them act. Here, we report results for the subset of sampled humans that can reach the goal under the oracle AI policy. Doing so preserves the relative ordering of method performances and reduces noise; in appendix fig 11 [25] we give an example of results that include individuals who never reach the goal.
Baselines. Our baselines are ways to learn the AI policy online. Using data D ýą = {(ĩ ýą , ė ýą , ĩ ′ ýą , Ĩ ýą )}, the model-free approach directly estimates č * ýą via Q-learning. The model-based method estimates Đ ýą using the observed transitions and then solves for ÿ * ýą with certainty equivalence. Both approaches bypass the need for explicitly solving for a human policy. The always ā and always þ are "no personalization, " in which the AI policy is to always intervene on Ą and þ, respectively. Our method, chainworld, estimates the parameters ĉ from D ýą .
In perfect conditions, the AI agent can use chainworld to reach oracle-level performance in the fewest episodes. When the true human matches our inductive bias, i.e. both are chainworlds, we achieve the fastest personalization in fig. 4. In contrast, modelfree requires hundreds of episodes before it learns policies that are better than random (which we demonstrate in fig. 10 of the appendix [25]).
Oracle Chainworld (ours) Random Model-free Model-based Always γ Always rb 5 10 15 Episode -30 -20 -10 AI Rewards Chain world Our method's performance scales to high-dimensional human models equivalent to the chainworld. In the prior theoretical section, we provided examples of human MDPs that reduce to the chainworld. The gridworld in fig. 5a is one such world since it is a type of distance world. In fig. 5, our method still personalizes the fastest in increasingly large state spaces, because the Oracle Chainworld (ours) Random Model-free Model-based Always γ Always rb 0.2 0.4 Noise level -50 -25 0 AI Rewards Noise in r g (a) Chainworld robust to low and high mis. 0.2 0.4 Noise level -50 -25 0 AI Rewards Noise in γ (b) Chainworld is robust to low mis. 0.2 0.4 Noise level -50 -25 0 AI Rewards Noise in r b (c) Environment challenges all methods. 6b, and all methods, including oracle, struggle to perform well in fig. 6c. Details and plots for all environments in appendix D.1 and E.3 [25], respectively. longer a chainworld, but a gridworld as in fig. 5a. However, the gridworld in this experiment is no longer equivalent to our chainworld because the goal state ĩ ĝ is not in the lower-right corner at [Ĕ, 0]. In fact, the equivalence degrades as Ċ increases for [Ĕ, Ċ]. We set the grid dimensions as Ĕ = 8, ĕ = 5 and define low misspecification as Ċ = 1 and high misspecification as Ċ = 4.
We are robust to low levels of misspecification. In table 2, our method outperforms baselines in 9 out of 12 robustness experiments under low levels of misspecification. With high misspecification, when our method is not the best, it falls within two standard errors of the next-best method in all but one condition. Some humans are difficult to intervene on overall, even for the oracle. All methods, including the oracle, earn fewer rewards when the burden parameter Ĩ Ę is noisy (see fig. 6c). This indicates that it is particularly important to model Ĩ Ę well in frictionful tasks. For example, we may ensure that features predictive of burden, such as mood, are part of the AI's state space, so that we can estimate Ĩ Ę .
To reduce (non-equivalent) human models to the chainworld, it is important that we capture distance to goal well. Since chainworld is one-dimensional, it can only represent worlds whose multi-dimensional states can be mapped to one dimension. When such a mapping is not possible, we must choose between capturing progress toward goal (e.g. how far does the patient feel from shoulder recovery?) or distance from disengagement (e.g. how close to giving up does the patient feel?). Under the "wrong distance to goal / disengagement mapping" condition in table 2, we show that capturing progress toward goal matters more. This implies that chainworlds can still be applied to settings where we cannot model all factors that lead to disengagement, so long as we have an accurate way of measuring the human's progress to the goal.
In this paper, we introduced Behavior Model Reinforcement Learning (BMRL), a framework for AI agents to intervene on human agents performing frictionful tasks. We proposed a simple model of the human agent-the chainworld-that the AI agent can use to rapidly personalize. Using a novel definition of equivalence between human models in BMRL, we defined a theoretical class of human MDPs that chainworld can generalize to and showed that this class contains behaviorally meaningful models of humans.
Our chainworlds are not psychologically verified human models; in future work, we will formally test the modeling assumptions with user studies. To apply BMRL in the real world, we must also consider the ethics of AI intervention. Mainly, we must ensure the AI does not manipulate the human. BMRL should only be used for people who already have a long-term goal, and the AI must not change that goal. Subgroup fairness should also be considered during learning and personalization.
Although we aimed to be comprehensive in testing chainworld's robustness, there were limitations to our approach. First, we did not evaluate how multiple misspecifications may compound to affect AI performance. Second, our analyses assumed that the mapping from the true MDP to the chainworld is given. In some applications this is reasonable; in PT, a domain expert is likely to know which factors contribute to a patient's perception of "progress" (the mapping from a distance world to a chainworld). In other cases, one will need to learn this mapping in conjunction with the chainworld parameters.
We made several simplifying assumptions on the human + AI interactions. We avoided a POMDP formulation by assuming that there are no delayed effects of the AI's actions on the human MDP. However, habituation (reduced effectiveness of repeated interventions) is a well-studied phenomenon in digital interventions (e.g. [12]). Furthermore, we avoided multi-agent RL by assuming that the human is not learning, and instead, is solving an (implicitly) known MDP at each time step. We did not consider suboptimality of the human agent's planning, such as (small) fixed-horizon planning. Finally (and excitingly), BMRL is adaptable to more diverse AI interventions. Our paper focused exclusively on interventions to the human's discount and reward. In many applications, the human's perception of state, actions, and transitions may also be impaired. Similarly, behavioral interventions on perceptions of state, actions, and transitions exist and could be incorporated into our framework.