More Like this

1. Exploring the Impact of Simple Explanations and Agency on Batch Deep Reinforcement Learning Induced Pedagogical Policies.

   Sanz Ausin, M; Maniktala, M; Barnes, T; Chi, M. (January 2020, Artificial Intelligence in Education. AIED 2020)

   null (Ed.)

   In recent years, Reinforcement learning (RL), especially Deep RL (DRL), has shown outstanding performance in video games from Atari, Mario, to StarCraft. However, little evidence has shown that DRL can be successfully applied to real-life human-centric tasks such as education or healthcare. Different from classic game-playing where the RL goal is to make an agent smart, in human-centric tasks the ultimate RL goal is to make the human-agent interactions productive and fruitful. Additionally, in many real-life human-centric tasks, data can be noisy and limited. As a sub-field of RL, batch RL is designed for handling situations where data is limited yet noisy, and building simulations is challenging. In two consecutive classroom studies, we investigated applying batch DRL to the task of pedagogical policy induction for an Intelligent Tutoring System (ITS), and empirically evaluated the effectiveness of induced pedagogical policies. In Fall 2018 (F18), the DRL policy is compared against an expert-designed baseline policy and in Spring 2019 (S19), we examined the impact of explaining the batch DRL-induced policy with student decisions and the expert baseline policy. Our results showed that 1) while no significant difference was found between the batch RL-induced policy and the expert policy in F18, the batch RL-induced policy with simple explanations significantly improved students’ learning performance more than the expert policy alone in S19; and 2) no significant differences were found between the student decision making and the expert policy. Overall, our results suggest that pairing simple explanations with induced RL policies can be an important and effective technique for applying RL to real-life human-centric tasks. 

   more » « less
2. The Impact of Batch Deep Reinforcement Learning on Student Performance: A Simple Act of Explanation Can Go A Long Way

   Sanz_Ausin, Markel; Maniktala, Mehak; Barnes, Tiffany; Chi, Min (April 2023, International journal of artificial intelligence in education)

   While Reinforcement learning (RL), especially Deep RL (DRL), has shown outstanding performance in video games, little evidence has shown that DRL can be successfully applied to human-centric tasks where the ultimate RL goal is to make the \textit{human-agent interactions} productive and fruitful. In real-life, complex, human-centric tasks, such as education and healthcare, data can be noisy and limited. Batch RL is designed for handling such situations where data is \textit{limited yet noisy}, and where \textit{building simulations is challenging}. In two consecutive empirical studies, we investigated Batch DRL for pedagogical policy induction, to choose student learning activities in an Intelligent Tutoring System. In Fall 2018 (F18), we compared the Batch DRL policy to an Expert policy, but found no significant difference between the DRL and Expert policies. In Spring 2019 (S19), we augmented the Batch DRL-induced policy with \textit{a simple act of explanation} by showing a message such as \textit{"The AI agent thinks you should view this problem as a Worked Example to learn how some new rules work."}. We compared this policy against two conditions, the Expert policy, and a student decision making policy. Our results show that 1) the Batch DRL policy with explanations significantly improved student learning performance more than the Expert policy; and 2) no significant differences were found between the Expert policy and student decision making. Overall, our results suggest that \textit{pairing simple explanations with the Batch DRL policy} can be an important and effective technique for applying RL to real-life, human-centric tasks. 

   more » « less
3. RLang: A Declarative Language for Describing Partial World Knowledge to Reinforcement Learning Agents

   Rodriguez-Sanchez, Rafael; Spiegel, Benjamin; Wang, Jennifer; Patel, Roma; Tellex, Stefanie; Konidaris, George (July 2023, Proceedings of the 40th International Conference on Machine Learning)

   We introduce RLang, a domain-specific language (DSL) for communicating domain knowledge to an RL agent. Unlike existing RL DSLs that ground to single elements of a decision-making formalism (e.g., the reward function or policy), RLang can specify information about every element of a Markov decision process. We define precise syntax and grounding semantics for RLang, and provide a parser that grounds RLang programs to an algorithm-agnostic partial world model and policy that can be exploited by an RL agent. We provide a series of example RLang programs demonstrating how different RL methods can exploit the resulting knowledge, encompassing model-free and model-based tabular algorithms, policy gradient and value-based methods, hierarchical approaches, and deep methods. 

   more » « less
4. Parental rights or parental wrongs: Parents’ metacognitive knowledge of the factors that influence their school choice decisions

   https://doi.org/10.1371/journal.pone.0301768  

   Cash, Trent N; Oppenheimer, Daniel M (April 2024, PLOS ONE)

   Nooripour, Roghieh (Ed.)

   School choice initiatives–which empower parents to choose which schools their children attend–are built on the assumptions that parents know what features of a school are most important to their family and that they are capable of focusing on the most important features when they make their decisions. However, decades of psychological research suggest that decision makers lack metacognitive knowledge of the factors that influence their decisions. We sought to reconcile this discrepancy between the policy assumptions and the psychological research. To do so, we asked participants to complete Choice-Based Conjoint surveys in which they made series of choices between different hypothetical schools. We then asked participants to self-report the weight they placed on each attribute when making their choices. Across four studies, we found that participants did not know how much weight they had placed on various school attributes. Average correlations between stated and revealed weights ranged fromr= .34–.54. Stated weights predicted different choices than revealed weights in 16.41–20.63% of decisions. These metacognitive limitations persisted regardless of whether the participants were parents or non-parents (Study 1a/1b), the nature of the attributes that participants used to evaluate alternatives (Study 2), and whether or not decision makers had access to school ratings that could be used as metacognitive aids (Study 3). In line with prior psychological research–and in contract to policy assumptions–these findings demonstrate that decision makers do not have particularly strong metacognitive knowledge of the factors that influence their school choice decisions. As a result, parents making school choice decisions are likely to seek out and use the wrong information, thus leading to suboptimal school choices. Future research should replicate these results in more ecologically valid samples and test new approaches to school choice that account for these metacognitive limitations. 

   more » « less
5. Reinforcement learning‐guided control strategies for CAR T‐cell activation and expansion

   https://doi.org/10.1002/bit.28753  

   Ferdous, Sakib; Shihab, Ibne Farabi; Chowdhury, Ratul; Reuel, Nigel F (September 2024, Biotechnology and Bioengineering)

   Abstract Reinforcement learning (RL), a subset of machine learning (ML), could optimize and control biomanufacturing processes, such as improved production of therapeutic cells. Here, the process of CAR T‐cell activation by antigen‐presenting beads and their subsequent expansion is formulated in silico. The simulation is used as an environment to train RL‐agents to dynamically control the number of beads in culture to maximize the population of robust effector cells at the end of the culture. We make periodic decisions of incremental bead addition or complete removal. The simulation is designed to operate in OpenAI Gym, enabling testing of different environments, cell types, RL‐agent algorithms, and state inputs to the RL‐agent. RL‐agent training is demonstrated with three different algorithms (PPO, A2C, and DQN), each sampling three different state input types (tabular, image, mixed); PPO‐tabular performs best for this simulation environment. Using this approach, training of the RL‐agent on different cell types is demonstrated, resulting in unique control strategies for each type. Sensitivity to input‐noise (sensor performance), number of control step interventions, and advantages of pre‐trained RL‐agents are also evaluated. Therefore, we present an RL framework to maximize the population of robust effector cells in CAR T‐cell therapy production. 

   more » « less