Objective This work examines two human–autonomy team (HAT) training approaches that target communication and trust calibration to improve team effectiveness under degraded conditions. Background Human–autonomy teaming presents challenges to teamwork, some of which may be addressed through training. Factors vital to HAT performance include communication and calibrated trust. Method Thirty teams of three, including one confederate acting as an autonomous agent, received either entrainment-based coordination training, trust calibration training, or control training before executing a series of missions operating a simulated remotely piloted aircraft. Automation and autonomy failures simulating degraded conditions were injected during missions, and measures of team communication, trust, and task efficiency were collected. Results Teams receiving coordination training had higher communication anticipation ratios, took photos of targets faster, and overcame more autonomy failures. Although autonomy failures were introduced in all conditions, teams receiving the calibration training reported that their overall trust in the agent was more robust over time. However, they did not perform better than the control condition. Conclusions Training based on entrainment of communications, wherein introduction of timely information exchange through one team member has lasting effects throughout the team, was positively associated with improvements in HAT communications and performance under degraded conditions. Training that emphasized the shortcomings of the autonomous agent appeared to calibrate expectations and maintain trust. Applications Team training that includes an autonomous agent that models effective information exchange may positively impact team communication and coordination. Training that emphasizes the limitations of an autonomous agent may help calibrate trust.
more »
« less
Assessing Communication and Trust in an AI Teammate in a Dynamic Task Environment
This research examines the relationship between anticipatory pushing of information and trust in human– autonomy teaming in a remotely piloted aircraft system - synthetic task environment. Two participants and one AI teammate emulated by a confederate executed a series of missions under routine and degraded conditions. We addressed the following questions: (1) How do anticipatory pushing of information and trust change from human to human and human to autonomous team members across the two sessions? and (2) How is anticipatory pushing of information associated with the trust placed in a teammate across the two sessions? This study demonstrated two main findings: (1) anticipatory pushing of information and trust differed between human-human and human-AI dyads, and (2) anticipatory pushing of information and trust scores increased among human-human dyads under degraded conditions but decreased in human-AI dyads.
more »
« less
- Award ID(s):
- 1828010
- NSF-PAR ID:
- 10344408
- Date Published:
- Journal Name:
- 2021 IEEE 2nd International Conference on Human-Machine Systems (ICHMS)
- Page Range / eLocation ID:
- 1 to 6
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
In activities such as dancing and sports, people synchronize behaviors in many different ways. Synchroni- zation between people has traditionally been characterized as either perfect mirroring (1:1 in-phase synchronization, spontaneous synchrony, and mimicry) or reflectional mir- roring (1:1 antiphase synchronization), but most activities require partners to synchronize more complicated patterns. We asked visually coupled dyads to coordinate finger move- ments to perform multifrequency ratios (1:1, 2:1, 3:1, 4:1, and 5:1). Because these patterns are coordinated across and not just within individual physiological and motor systems, we based our predictions on frequency-locking dynamics, which is a general coordination principle that is not limited to physiological explanations. Twenty dyads performed five multifrequency ratios under three levels of visual coupling, with half using a subcritical visual information update rate. The dynamical principle was supported, such that multi- frequency performance tends to abide by the strictures of frequency locking. However, these constraints are relaxed if the visual information rate is beyond the critical informa- tion update rate. An analysis of turning points in the oscil- latory finger movements suggested that dyads did not rely on this visual information to stabilize coordination. How the laboratory findings align with naturalistic observa- tions of multifrequency performance in actual sports teams (Double Dutch) is discussed. Frequency-locking accounts not only for the human propensity for perfect mirroring but also for variations in performance when dyads deviate from mirroring.more » « less
-
Objective Over the past decade, we developed and studied a face-to-face video-based analysis-of-practice professional development (PD) model. In a cluster randomized trial, we found that the face-to-face model enhanced elementary science teacher knowledge and practice and resulted in important improvements to student science achievement (student treatment effect, d = 0.52; Taylor et al, 2017; Roth et al, 2018). The face-to-face PD model is expensive and difficult to scale. In this paper, we present the results of a two-year design-based research study to translate the face-to-face PD into a facilitated online PD experience. The purpose is to create an effective, flexible, and cost-efficient PD model that will reach a broader audience of teachers. Perspective/Theoretical Framework The face-to-face PD model is grounded in situated cognition and cognitive apprenticeship frameworks. Teachers engage in learning science content and effective science teaching practices in the context in which they will be teaching. There are scaffolded opportunities for teachers to learn from analysis of model videos by experienced teachers, to try teaching model units, to analyze video of their own teaching efforts, and ultimately to develop their own unit, with guidance. The PD model attends to the key features of effective PD as described by Desimone (2009) and others. We adhered closely to the design principles of the face-to-face model as described by Authors, 2019. Methods We followed a design-based research approach (DBR; Cobb et al., 2003; Shavelson et al., 2003) to examine the online program components and how they promoted or interfered with the development of teachers’ knowledge and reflective practice. Of central interest was the examination of mechanisms for facilitating teacher learning (Confrey, 2006). To accomplish this goal, design researchers engaged in iterative cycles of problem analysis, design, implementation, examination, and redesign (Wang & Hannafin, 2005) in phase one of the project before studying its effect. Data Three small pilot groups of teachers engaged in both synchronous and asynchronous components of the larger online course which began implementation with a 10-week summer course that leads into study groups of participants meeting through one academic year. We iteratively designed, tested, and revised 17 modules across three pilot versions. On average, pilot groups completed one module every two weeks. Pilot 1 began the work in May 2019; Pilot 2 began in August 2019, and Pilot 3 began in October 2019. Pilot teachers responded to surveys and took part in interviews related to the PD. The PD facilitators took extensive notes after each iteration. The development team met weekly to discuss revisions. We revised all modules between each pilot group and used what we learned to inform our development of later modules within each pilot. For example, we applied what we learned from testing Module 3 with Pilot 1 to the development of Module 3 for Pilots 2, and also applied what we learned from Module 3 with Pilot 1 to the development of Module 7 for Pilot 1. Results We found that community building required the same incremental trust-building activities that occur in face-to-face PD. Teachers began with low-risk activities and gradually engaged in activities that required greater vulnerability (sharing a video of themselves teaching a model unit for analysis and critique by the group). We also identified how to contextualize technical tools with instructional prompts to allow teachers to productively interact with one another about science ideas asynchronously. As part of that effort, we crafted crux questions to surface teachers’ confusions or challenges related to content or pedagogy. We called them crux questions because they revealed teachers’ uncertainty and deepened learning during the discussion. Facilitators leveraged asynchronous responses to crux questions in the synchronous sessions to push teacher thinking further than would have otherwise been possible in a 2-hour synchronous video-conference. Significance Supporting teachers with effective, flexible, and cost-efficient PD is difficult under the best of circumstances. In the era of covid-19, online PD has taken on new urgency. NARST members will gain insight into the translation of an effective face-to-face PD model to an online environment.more » « less
-
The culture within engineering colleges and departments has been historically quiet when considering social justice issues. Often the faculty in those departments are less concerned with social issues and are primarily focused on their disciplines and the concrete ways that they can make impacts academically and professionally in their respective arena’s. However, with the social climate of the United States shifting ever more towards a politically charged climate, and current events, particularly the protests against police brutality in recent years, faculty and students are constantly inundated with news of injustices happening in our society. The murder of George Floyd on May 25th 2020 sent shockwaves across the United States and the world. The video captured of his death shared across the globe brought everyone’s attention to the glaringly ugly problem of police brutality, paired with the COVID-19 pandemic, and US election year, the conditions were just right for a social activist movement to grow to a size that no one could ignore. Emmanuel Acho spoke out, motivated by injustices seen in the George Floyd murder, initially with podcasts and then by writing his book “Uncomfortable Converstations with a Black Man” [1]. In his book he touched on various social justice issues such as: racial terminology (i.e., Black or African American), implicit biases, white privilege, cultural appropriation, stereotypes (e.g., the “angry black man”), racial slurs (particularly the n-word), systemic racism, the myth of reverse racism, the criminal justice system, the struggles faced by black families, interracial families, allyship, and anti-racism. Students and faculty at Anonymous University felt compelled to set aside the time to meet and discuss this book in depth through the video conferencing client Zoom. In these meetings diverse facilitators were tasked with bringing the topics discussed by Acho in his book into conversation and pushing attendees of these meetings to consider those topics critically and personally. In an effort to avoid tasking attendees with reading homework to be able to participate in these discussions, the discussed chapter of the audiobook version of Acho’s book was played at the beginning of each meeting. Each audiobook chapter lasted between fifteen and twenty minutes, after which forty to forty-five minutes were left in the hour-long meetings to discuss the content of the chapter in question. Efforts by students and faculty were made to examine how some of the teachings of the book could be implemented into their lives and at Anonymous University. For broader topics, they would relate the content back to their personal lives (e.g., raising their children to be anti-racist and their experiences with racism in American and international cultures). Each meeting was recorded for posterity in the event that those conversations would be used in a paper such as this. Each meeting had at least one facilitator whose main role was to provide discussion prompts based on the chapter and ensure that the meeting environment was safe and inclusive. Naturally, some chapters address topics that are highly personal to some participants, so it was vital that all participants felt comfortable and supported to share their thoughts and experiences. The facilitator would intervene if the conversation veered in an aggressive direction. For example, if a participant starts an argument with another participant in a non-constructive manner, e.g., arguing over the definition of ethnicity, then the facilitator will interrupt, clear the air to bring the group back to a common ground, and then continue the discussion. Otherwise, participants were allowed to steer the direction of the conversation as new avenues of discussion popped up. These meetings were recorded with the goal of returning to these conversations and analyzing the conversations between attendees. Grounded theory will be used to first assess the most prominent themes of discussion between attendees for each meeting [2]. Attendees will be contacted to expressly ask their permission to have their words and thoughts used in this work, and upon agreement that data will begin to be processed. Select attendees will be asked to participate in focus group discussions, which will also be recorded via Zoom. These discussions will focus around the themes pulled from general discussion and will aim to dive deeper into the impact that this experience has had on them as either students or faculty members. A set of questions will be developed as prompts, but conversation is expected to evolve organically as these focus groups interact. These sessions will be scheduled for an hour, and a set of four focus groups with four participants are expected to participate for a total of sixteen total focus group participants. We hope to uncover how this experience changed the lives of the participants and present a model of how conversations such as this can promote diversity, equity, inclusion, and access activities amongst faculty and students outside of formal programs and strategic plans that are implemented at university, college, or departmental levels.more » « less
-
more » « less
While there is increased interest in how trust spreads in Human Autonomy Teams (HATs), most trust measurements are subjective and do not examine real-time changes in trust. To develop a trust metric that consists of objective variables influenced by trust/distrust manipulations, we conducted an Interactive hybrid Cognitive Task Analysis (IhCTA) for a Remotely Piloted Aerial System (RPAS) HAT. The IhCTA adapted parts of the hybrid Cognitive Task Analysis (hCTA) framework. In this paper, we present the four steps of the IhCTA approach, including 1) generating a scenario task overview, 2) generating teammate-specific event flow diagrams, 3) identifying interactions and interdependencies impacted by trust/distrust manipulations, and 4) processing RPAS variables based on the IhCTA to create a metric. We demonstrate the application of the metric through a case study that examines how the influence of specific interactions on team state changes before and after the spread of distrust.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/ICHMS53169.2021.9582626
