More Like this

1. Self-Annotation Methods for Aligning Implicit and Explicit Human Feedback in Human-Robot Interaction

   https://doi.org/10.1145/3568162.3576986  

   Zhang, Qiping; Narcomey, Austin; Candon, Kate; Vázquez, Marynel (March 2023, HRI '23: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction)

   Recent research in robot learning suggests that implicit human feedback is a low-cost approach to improving robot behavior without the typical teaching burden on users. Because implicit feedback can be difficult to interpret, though, we study different methods to collect fine-grained labels from users about robot performance across multiple dimensions, which can then serve to map implicit human feedback to performance values. In particular, we focused on understanding the effects of annotation order and frequency on human perceptions of the self-annotation process and the usefulness of the labels for creating data-driven models to reason about implicit feedback. Our results demonstrate that different annotation methods can influence perceived memory burden, annotation difficulty, and overall annotation time. Based on our findings, we conclude with recommendations to create future implicit feedback datasets in Human-Robot Interaction. 

   more » « less
2. A Preliminary Investigation into Learning Behaviors in Complex Environments for Human-in-the-Loop Cyber-Physical Systems

   https://doi.org/10.1177/1071181321651222  

   Luster, Maya S.; Pitts, Brandon J. (September 2021, Proceedings of the Human Factors and Ergonomics Society Annual Meeting)

   The field of Cyber-Physical Systems (CPS) is increasingly recognizing the importance of integrating Human Factors for Human-in-the-loop CPS (HiLCPS) developments. This is because psychological, physiological, and behavioral characteristics of humans can be used to predict human-machine interactions. The goal of this pilot study is to collect initial data to determine whether driving and eye tracking metrics can provide evidence of learning for a CPS project. Six participants performed a series of 12 repeated obstacle avoidance tasks in manual driving. Lane deviations and fixation-related eye data were recorded for each trial. Overall, participants displayed either conservation/safe or aggressive/risky in their lateral position with respect to the obstacle during successive trials. Also, eye tracking metrics were not significantly affected by trial number, but observational trends suggest their potential for aiding in understanding adjustments humans make in learning. Results can inform predictive modeling algorithms that can anticipate and mitigate potential problems in real-time. 

   more » « less
3. Toward Foundation Models for Online Complex Event Detection in CPS-IoT: A Case Study

   https://doi.org/10.1145/3722565.3727198  

   Han, Liying; Dong, Gaofeng; Ouyang, Xiaomin; Kaplan, Lance; Cerutti, Federico; Srivastava, Mani (May 2025, ACM)

   Complex events (CEs) play a crucial role in CPS-IoT applications, enabling high-level decision-making in domains such as smart monitoring and autonomous systems. However, most existing models focus on short-span perception tasks, lacking the long-term reasoning required for CE detection. CEs consist of sequences of short-time atomic events (AEs) governed by spatiotemporal dependencies. Detecting them is difficult due to long, noisy sensor data and the challenge of filtering out irrelevant AEs while capturing meaningful patterns. This work explores CE detection as a case study for CPS-IoT foundation models capable of long-term reasoning. We evaluate three approaches: (1) leveraging large language models (LLMs), (2) employing various neural architectures that learn CE rules from data, and (3) adopting a neurosymbolic approach that integrates neural models with symbolic engines embedding human knowledge. Our results show that the state-space model, Mamba, which belongs to the second category, outperforms all methods in accuracy and generalization to longer, unseen sensor traces. These findings suggest that state-space models could be a strong backbone for CPS-IoT foundation models for long-span reasoning tasks. 

   more » « less
4. Hierarchical Imitation Learning of Team Behavior from Heterogeneous Demonstrations

   Seo, Sangwon; Unhelkar, Vaibhav (June 2025, Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS))

   Successful collaboration requires team members to stay aligned, especially in complex sequential tasks. Team members must dynamically coordinate which subtasks to perform and in what order. However, real-world constraints like partial observability and limited communication bandwidth often lead to suboptimal collaboration. Even among expert teams, the same task can be executed in multiple ways. To develop multi-agent systems and human-AI teams for such tasks, we are interested in data-driven learning of multimodal team behaviors. Multi-Agent Imitation Learning (MAIL) provides a promising framework for data-driven learning of team behavior from demonstrations, but existing methods struggle with heterogeneous demonstrations, as they assume that all demonstrations originate from a single team policy. Hence, in this work, we introduce DTIL: a hierarchical MAIL algorithm designed to learn multimodal team behaviors in complex sequential tasks. DTIL represents each team member with a hierarchical policy and learns these policies from heterogeneous team demonstrations in a factored manner. By employing a distribution-matching approach, DTIL mitigates compounding errors and scales effectively to long horizons and continuous state representations. Experimental results show that DTIL outperforms MAIL baselines and accurately models team behavior across a variety of collaborative scenarios. 

   more » « less
5. Do Speech-Based Collaboration Analytics Generalize Across Task Contexts?

   https://doi.org/10.1145/3506860.3506894  

   Pugh, Samuel L.; Rao, Arjun; Stewart, Angela E.B.; D'Mello, Sidney K. (March 2022, Proceedings of the 12th International Learning Analytics and Knowledge Conference)

   We investigated the generalizability of language-based analytics models across two collaborative problem solving (CPS) tasks: an educational physics game and a block programming challenge. We analyzed a dataset of 95 triads (N=285) who used videoconferencing to collaborate on both tasks for an hour. We trained supervised natural language processing classifiers on automatic speech recognition transcripts to predict the human-coded CPS facets (skills) of constructing shared knowledge, negotiation / coordination, and maintaining team function. We tested three methods for representing collaborative discourse: (1) deep transfer learning (using BERT), (2) n-grams (counts of words/phrases), and (3) word categories (using the Linguistic Inquiry Word Count [LIWC] dictionary). We found that the BERT and LIWC methods generalized across tasks with only a small degradation in performance (Transfer Ratio of .93 with 1 indicating perfect transfer), while the n-grams had limited generalizability (Transfer Ratio of .86), suggesting overfitting to task-specific language. We discuss the implications of our findings for deploying language-based collaboration analytics in authentic educational environments. 

   more » « less