More Like this

1. Collaboration Requirement Planning Protocol for HUB-CI in Factories of the Future

   https://doi.org/10.1016/j.promfg.2020.01.327  

   Puwadol Oak Dusadeerungsikul, Maitreya Sreeram (January 2019, Procedia manufacturing)

   Rapid advances in production systems’ models and technology continually challenge manufacturers preparing for the factories of the future. To address the complexity issues typically coupled with the improvements, we have developed a brain-inspired model for production systems, HUBCI. It is a virtual Hub for Collaborative Intelligence, receiving human instructions from a human-computer interface; and in turn, commanding robots via ROS. The purpose of HUB-CI is to manage diverse local information and real-time signals obtained from system agents (robots, humans, and warehouse components, e.g., carts, shelves, racks) and globally update real-time assignments and schedules for those agents. With Collaborative Control Theory (CCT) we first develop the protocol for collaborative requirement planning for a HUB-CI, (CRP-H), through which we can synchronize the agents to work smoothly and execute rapidly changing tasks. This protocol is designed to answer: Which robot(s) should perform each human-assigned task, and when should this task be performed? The primary two phases of CRP-H, CRP-I (task assignment optimization) and CRP-II (agents schedule harmonization) are developed and validated for two test scenarios: a two-robot collaboration system with five tasks; and a two-robot-and-helper-robot collaboration system with 25 tasks. Simulation results indicate that under CRP-H, both operational cost and makespan of the production work are significantly reduced in the two scenarios. 

   more » « less
2. Inclusive Portraits: Race-Aware Human-in-the-Loop Technology

   https://doi.org/10.1145/3617694.3623235  

   Flores-Saviaga, Claudia; Curtis, Christopher; Savage, Saiph (October 2023, ACM)

   AI has revolutionized the processing of various services, including the automatic facial verification of people. Automated approaches have demonstrated their speed and efficiency in verifying a large volume of faces, but they can face challenges when processing content from certain communities, including communities of people of color. This challenge has prompted the adoption of "human-inthe-loop" (HITL) approaches, where human workers collaborate with the AI to minimize errors. However, most HITL approaches do not consider workers’ individual characteristics and backgrounds. This paper proposes a new approach, called Inclusive Portraits (IP), that connects with social theories around race to design a racially-aware human-in-the-loop system. Our experiments have provided evidence that incorporating race into human-in-the-loop (HITL) systems for facial verification can significantly enhance performance, especially for services delivered to people of color. Our findings also highlight the importance of considering individual worker characteristics in the design of HITL systems, rather than treating workers as a homogenous group. Our research has significant design implications for developing AI-enhanced services that are more inclusive and equitable. 

   more » « less
3. FAIRO: Fairness-aware Sequential Decision Making for Human-in-the-Loop CPS

   https://doi.org/10.1109/ICCPS61052.2024.00015  

   Zhao, Tianyu; Taherisadr, Mojtaba; Elmalaki, Salma (May 2024, IEEE)

   Achieving fairness in sequential decision making systems within Human-in-the-Loop (HITL) environments is a critical concern, especially when multiple humans with different behavior and expectations are affected by the same adaptation decisions in the system. This human variability factor adds more complexity since policies deemed fair at one point in time may become discriminatory over time due to variations in human preferences resulting from inter- and intra-human variability. This paper addresses the fairness problem from an equity lens, considering human behavior variability, and the changes in human preferences over time. We propose FAIRO, a novel algorithm for fairness-aware sequential decision making in HITL adaptation, which incorporates these notions into the decision-making process. In particular, FAIRO decomposes this complex fairness task into adaptive sub-tasks based on individual human preferences through leveraging the Options reinforcement learning framework. We design FAIRO to generalize to three types of HITL application setups that have the shared adaptation decision problem. Furthermore, we recognize that fairness-aware policies can sometimes conflict with the application’s utility. To address this challenge, we provide a fairness-utility tradeoff in FAIRO, allowing system designers to balance the objectives of fairness and utility based on specific application requirements. Extensive evaluations of FAIRO on the three HITL applications demonstrate its generalizability and effectiveness in promoting fairness while accounting for human variability. On average, FAIRO can improve fairness compared with other methods across all three applications by 35.36%. 

   more » « less
4. Remote Instrumentation Science Environment for Intelligent Image Analytics

   https://doi.org/10.1109/eScience55777.2022.00023  

   Alarcon, Mauro Lemus; Wang, Songjie; Nguyen, Nguyen; Pandey, Ashish; Bunyak, Filiz; Maschmann, Matthew; Palaniappan, Kannappan; Calyam, Prasad (December 2022, 2022 IEEE 18th International Conference on e-Science)

   Current scientific experiments frequently involve control of specialized instruments (e.g., scanning electron mi- croscopes), image data collection from those instruments, and transfer of the data for processing at simulation centers. This process requires a “human-in-the-loop” to perform those tasks manually, which besides requiring a lot of effort and time, could lead to inconsistencies or errors. Thus, it is essential to have an automated system capable of performing remote instrumentation to intelligently control and collect data from the scientific instruments. In this paper, we propose a Remote Instrumentation Science Environment (RISE) for intelligent im- age analytics that provides the infrastructure to securely capture images, determine process parameters via machine learning, and provide experimental control actions via automation, under the premise of “human-on-the-loop”. The machine learning in RISE aids an iterative discovery process to assist researchers to tune instrument settings to improve the outcomes of experiments. Driven by two scientific use cases of image analytics pipelines, one in material science, and another in biomedical science, we show how RISE automation leverages a cutting-edge integration of cloud computing, on-premise HPC cluster, and a Python programming interface available on a microscope. Using web services, we implement RISE to perform automated image data collection/analysis guided by an intelligent agent to provide real-time feedback control of the microscope using the image analytics outputs. Our evaluation results show the benefits of RISE for researchers to obtain higher image analytics accuracy, save precious time in manually controlling the microscopes, while reducing errors in operating the instruments. 

   more » « less
5. Human Supervision is Key to Achieving Accurate AI-assisted Wildlife Identifications in Camera Trap Images

   https://doi.org/10.5334/cstp.752  

   Huebner, Sarah E; Palmer, Meredith S; Packer, Craig (January 2024, Citizen Science: Theory and Practice)

   Fortson, Lucy; Crowston, Kevin; Kloetzer, Laure; Ponti, Marisa (Ed.)

   Using public support to extract information from vast datasets has become a popular method for accurately labeling wildlife data in camera trap (CT) images. However, the increasing demand for volunteer effort lengthens the time interval between data collection and our ability to draw ecological inferences or perform data-driven conservation actions. Artificial intelligence (AI) approaches are currently highly effective for species detection (i.e., whether an image contains animals or not) and labeling common species; however, it performs poorly on species rarely captured in images and those that are highly visually similar to one another. To capitalize on the best of human and AI classifying methods, we developed an integrated CT data pipeline in which AI provides an initial pass on labeling images, but is supervised and validated by humans (i.e., a “human-in-the-loop” approach). To assess classification accuracy gains, we compare the precision of species labels produced by AI and HITL protocols to a “gold standard” (GS) dataset annotated by wildlife experts. The accuracy of the AI method was species-dependent and positively correlated with the number of training images. The combined efforts of HITL led to error rates of less than 10% for 73% of the dataset and lowered the error rates for an additional 23%. For two visually similar species, human input resulted in higher error rates than AI. While integrating humans in the loop increases classification times relative to AI alone, the gains in accuracy suggest that this method is highly valuable for high-volume CT surveys. 

   more » « less