Learning to effectively work in teams in co-located settings remains irreplaceable in many professional sectors, even though online teamwork practices have become more common [29]. Embodied team learning is one such setting where students can freely move around a physical learning space to learn how to interact effectively with resources and other students [5]. This is common practice in high-risk sectors where inadequate teamwork skills have been linked to failures and safety issues [15] such as in firefighting [2] and healthcare [9]. The conditions in such settings can lead to high complexity in communication dynamics, as dialogue segments of team members can happen in parallel at different physical locations with varied team member configurations. Figure 1 illustrates this situation in the context of nursing simulation, where a team of students may need to temporarily split themselves into two sub-teams to complete various tasks.
The dialogue within those two sub-teams can be unrelated, resulting in dialogue segments being spatially distributed. In such a situation, teachers can find it difficult to assess the team's performance as critical events can happen simultaneously. Students can also struggle to reflect on their team dynamics since they cannot easily have a comprehensive view of their activity as a whole [21]. Rapid advancements in multimodal sensing [14] and artificial intelligence (AI) innovations are enabling new opportunities to automatically model dialogue in education. For instance, previous studies have demonstrated that teacherstudent classroom communication can be modelled to provide feedback to teachers on the authority level of their questions [6] and their classroom discourse skills [22]. It has also been proposed that modelling student-student communication can enable the analysis of teamwork [34] and collaborative problem-solving skills [1]. Recent works have demonstrated the value of rendering student verbal communication data visible using word-based interfaces to support the assessment of group collaboration [16] and interfaces showing key communication events to help teachers identify groups of students that may require more help [28].
These previous works suggest that the automated analysis of the content of group communication is becoming feasible and can be highly valuable to study and support learning. However, in these studies, students have been expected to work physically together, therefore generating one dialogue segment throughout a learning session. None of these works targeted highly dynamic embodied team learning situations where students can freely create sub-teams, which can lead to distributed dialogue segments, making it hard to extract meaning from students' logged dialogue. The only previous work studying such a highly dynamic learning situation was presented by Zhao et al. [34] in which authors combined spatial data with audio to enable extracting meaning from distributed dialogue segments. However, the authors did not consider the order of occurrence of key constructs that may augment the meaningfulness of the dialogue analysis. As suggested by the most of other previous works, it has been critical to consider such order for analysing verbal communication [6,16,17,22]. Against this gap in the literature, we formulate the following research question (RQ): To what extent can the order of occurrence of key high-order team constructs be modelled from students' distributed coded dialogue to identify effective team learning practices?
Our work addresses this question, and goes beyond previous work, by presenting a study that illustrates how audio and spatial data of students can be used to model critical high-order constructs of embodied team communication. The study was conducted in the context of a highly dynamic healthcare simulation setting. We used ordered network analysis (ONA) to model the order of occurrence of coded co-located dialogue data and identify effective team learning practices by analysing key differences between high and low performing teams.
Learning Context. The study involved a series of immersive team simulations, held over four weeks in 2021, which were part of the regular activities of an undergraduate course of the Bachelor of Nursing at Undisclosed University. These were conducted in a specialised classroom space equipped with four patient manikins and medical equipment simulating a hospital ward (see Fig. 1). Multimodal data (details provided below) from 228 consenting students (aged 20 to 23), grouped in 57 teams of four students, were collected. Due to limitations of the microphone hardware (e.g., signal interference) and other practical challenges (e.g., students accidentally turning microphones off), only the high-quality data from 60 students grouped in 15 teams were used in this study. Three nursing teachers, who designed the simulations, monitored the simulation from a control room and assessed the students' performance based on their observations. The learning design included the following four phases. (1) An initial handover, in which the first two students would enter the room and listen to an introduction conveyed by the doctor (played by a teacher). (2) The initial assessment, in which the same students would make a plan and would need to react to an unexpected event pre-programmed by the teachers, involving the identification of a serious problem suffered by the patient (in bed 4) and escalating the situation by calling for help. (3) Resolving emergency, in which the two other students in the team would enter the room, receive handover information and collaborate to help the patient at risk (defined by teachers as the primary task) while completing tasks for the three other patients (the secondary tasks). (4) Emergent diagnosis, in which an emergency doctor would enter the room and students would provide an update of the situation.
Each simulation was between 15 and 30 min long (avg = 20.25 min.; st. dev. = 8.13 min). Teachers assessed students' team performance in phases 2 and 3, since students' team dynamics mostly happened in these two phases.
Apparatus. Portable wireless (Xiaokoa) headset microphones were provided to consenting students to capture their voices. A multi-channel (TASCAM US-16×08) audio interface was used to synchronise the audio streams and store them into individual files. For spatial data, waist bags each containing a positioning sensor (Pozyx) were provided to each student. These data included each student's body orientation and their x-y spatial coordinates. The team assessment results were collected using a questionnaire filled by the teachers who evaluated the simulation. The questionnaire assessed their teamwork effectiveness based on a 7-point Likert scale. All data collection devices were synchronised automatically. Ethical approval was obtained from the Undisclosed University.
Modelling Multimodal Sensor Data. Utterance intervals (when individual speech started and ended) were automatically extracted from audio signals captured by each microphone using a voice activity detection script created via the Python library py-webrtcvad. The utterance intervals were used to organise students' utterances in the sequence of their turns of talking, and utterance content would be coded using the coding scheme described below. The utterance content was transcribed using a third-party transcription service.
Since students could be chatting at completely different locations and have conversations in parallel, spatial data was needed to organise students' dialogue into corresponding dialogue segments. We used the body orientation and spatial coordinates to detect dialogue segments by adopting the f-formations theory [23]. An f-formation appears whenever multiple people sustain a spatial and orientational relationship for collaboration in close proximity [23]. As a previous study in healthcare [27] suggested, communication between healthcare professionals commonly happens in close proximity (less than 1.5 m), we applied an f-formation detection algorithm [35] to differentiate dialogue segments. Furthermore, the spatial data were also used to detect Spaces of Interests (SoIs), namely the spaces of primary tasks (bed 4) and secondary tasks (beds 1-3).
Coding Scheme and Procedure. To analyse the content of dialogue segments, a coding scheme was designed for embodied teamwork communication in this context by adapting previous coding schemes [13,19] and a team theory framework [20]. The coding scheme includes four higher-order teamwork constructs and nine communication behaviours for coding the dialogue. The first construct, shared leadership, captures instances when students assigned others or themselves to specific tasks or provided handover information to bring others on board as no student was formally appointed as a leader. The second construct, situation awareness, captures the communication related to identification and reaction to patients' emergency state [19]. The third construct, shared mental model, captures the communication for establishing a shared understanding of the current situation and a potential plan to tackle this [32]. The last construct, closed-loop communication, refers to the double-checking of information or acknowledging the receipt of information [13]. More details about the definitions and examples for each communication behaviour are provided in Table 1.
The coding was done at an utterance level and each utterance could have multiple codes. Two researchers coded 20% of the dialogue data. Cohen's kappa was used to measure inter-rater reliability (0.6 was the threshold for acceptance) [12]. The kappa for each code was greater than 0.7 (see column 5 in Table 1). The coding of the remaining 80% of the data was completed by one researcher.
As suggested by theories of communication in healthcare teams [15], the sequential order of communication is essential to demonstrate effective healthcare teamwork. For example, the order of communication from information sharing to task allocation can illustrate assigning tasks a student provides information responding to a previously asked question 0.804
Acknowledgement [4] a student acknowledges receipt of information or instructions from others "Yes", "I agree", "Okay" 0.858 Checking-back [4] a student double-checks the information or instructions from others A: "Can you give her 1ml IV fluid?" B: "1ml IV fluid" (Check-back) 0.922 based on clinical evidence, which can be indicative of an effective team coordination strategy [15]. Thus, we used ordered network analysis (ONA) [30].
ONA is a technique to quantify and visualise the directed connection within coded data. The ONA algorithm employs similar functions and procedures as Epistemic Network Analysis (ENA), a widely used network analysis technique for the modeling and comparison of learning phenomena [3,24,25]. The key difference is that ONA accounts for the order of connections during the modeling and visualising processes. ONA starts processing the data by accumulating directed connections within units of analysis (i.e., subjects of research interest, such as a team) as high-dimensional vectors. These accumulations operate on the line level (i.e., the fundamental unit of meaning in the data, such as an utterance) by counting the order of codes' co-occurrences within stanza windows (i.e., the temporal context formed by span of lines). Then, the ordered co-occurrences are aggregated across each conversation (i.e., collections of meaningfully related stanza windows, such as all stanza windows of a sub-team) for each unit of analysis to obtain cumulative connection vectors. Next, a dimensional reduction via Singular Value Decomposition (SVD) or Means Rotation (MR) is applied to the collection of those cumulative vectors to project them as points in a two dimensional space.
In our study, we applied MR to maximise the variances between two groups of units on the x-axis of the space. Then, ONA networks are visualised in this two-dimensional space using two coordinated representations for each unit of analysis: (1) a projected point, which represents the location of its network in the low-dimensional projected space (shown as red or blue points in our study), and (2) a directed weighted network where nodes correspond to the codes, and edges reflect the relative frequency of connection between two codes. Specifically, the node size is proportional to the frequency of its represented code being connected with other codes; and the size of the coloured inner circle of a node is proportional to self-connections, i.e., the frequency of the code making a connection with itself. Between each pair of nodes, the edge consists of a pair of triangles with varied sizes to illustrate the frequency of directed connections. The bigger and darker a triangle is, the higher frequency of connections is. A black chevron is placed on the more frequent side of an edge to support recognizing the direction of connections. For example, a chevron pointing from node A to node B represents that the directed connection from A to B is more frequent.
To conduct ONA, several parameters need to be specified, namely: lines, conversations, stanza windows, units of analysis, and codes. In our study, students' utterances were used as lines and co-located dialogue segments as conversations. We used a stanza window size of three lines to accumulate connections, since we tested multiple configurations of stanza window and found this setting provided the highest variance in the ONA model [26]. The units of analysis in this study were the unique combinations of phase (i.e., phases 2 and 3) and SoIs (i.e., primary tasks and secondary tasks). Regarding codes, we included all codes in Table 1 except for checking-back, since it had extremely low frequency (n = 105) compared to other codes (mean = 791.7, st. dev. = 644.3). We omitted this code to maximise the clarity of analysis as suggested in [31]. We also excluded provision of handover information code in the secondary task phase 3 model for a similar reason. We built ONA models using the R implementation of ONA [10].
To address our RQ, we divided the 15 teams into seven low-performing and eight high-performing teams based on the median score of their team performance. Using these two groups, we can identify prominent differences of team communication in the mean ONA networks of high and low performing teams. To demonstrate the prominent differences, we created ONA mean network subtractions by subtracting the two groups' mean networks. We also conducted Mann-Whitney U tests on the distribution of projected points to compare if the differences between the groups were statistically significant.
Primary Task: Phase 2. As shown in Fig. 2 (left), for the case of dialogue data at bed 4 (the primary task), several directed connections appeared in high- performing teams, while the connection from planning to acknowledgement was the only major directed connection in low-performing teams. This is not a surprising result since the primary task in phase 2 was to identify the patient at risk and announce escalation, so team behaviours related to information exchange and escalation were expected to be more frequent for high-performing teams. The Mann-Whitney U tests also showed the differences between the high-performing teams (N = 8, Mdn = 0.40, Q1 = 0.19, Q3 = 0.37) and the low-performing teams (N = 7, Mdn = -0.301, Q1 = -0.34, Q3 = -0.20) on the first dimension (U = 54, p = 0.001, r = 0.58) were significant.
Specifically, although both groups of teams showed connections to information request, a key difference is that high-performing teams requested information more frequently after contributing with new information (sharing information) or after announcing an emergency (escalation). In contrast, information requests were limited in low-performing teams and happened only after planning, Considering that the directed connection from planning to acknowledgement is stronger than from planning to information request, we can conclude that low-performing teams demonstrated inefficient communication while planning. Moreover, evidence of the effective dynamics of high-performing teams is also illustrated by the directed connections to responding to request. For example, information request, and escalation are pointing towards responding to request. This suggests that students in high-performing teams exchanged information related to the critical task (announcing escalation) more frequently which was critical in this phase. Although similar connections occurred for low-performing teams, the strength of those connections was relatively weak, as indicated by the faded red edges.
Primary Task: Phase 3. The primary task in phase 3 mainly consisted of providing handover information to new team members and then offering emergent medical support to the patient at risk (in bed 4) while taking care of stable patients in the other beds. As shown in Fig. 2 (right), dialogue from the high-performing teams showed stronger connections among a small set of nodes on the right side of the x-axis; while low-performing teams had stronger connections on the left side. The Mann-Whitney U tests showed the difference between the high-performing teams (N = 8, Mdn = 0.152, Q1 = 0.02, Q3 = 0.31) and the low-performing teams (N = 7, Mdn = -0.13, Q1 = -0.19, Q3 = -0.10) on the first dimension (U = 52, p = 0.004, r = 0.52) was significant.
Although both groups of teams showed various connections to the node provision of handover information, the connections to other nodes were different. The blue circle in the node provision of handover information shows that the high-performing teams repeatedly provided handover information, and also when team members explicitly requested such information (see directed connection from information request to provision of handover information). This shows that the high-performing teams could provide handover information fluently and respond to questions from others. In contrast, the low-performing teams communicated about handover information infrequently, as indicated by the faded red edges connected to provision of handover information. This indicates that some team members in the low-performing teams may have not had complete information relevant for the primary task. Additionally, the red edges show that the low-performing teams communication focused on task allocation, information sharing, planning, and escalation announcement to gather information, which are constructs that were expected to occur in the previous phase, rather than focusing on provision of handover information which was critical in this phase.
The secondary tasks were related to the three patients with stable physical conditions. Students were expected to put less effort into the secondary tasks but still perform them as they would prioritise aiding the patient at risk. Secondary Tasks: Phase 2 and Phase 3. As shown in Fig. 3 (left), the majority of edges are in red. This is expected since the teams were expected to put less effort into secondary tasks, and teams rated as low-performing were not aligned with this expectation. The Mann-Whitney U tests showed a significant difference between the high-performing teams (N = 8, Mdn = 0.17, Q1 = 0.01, Q3 = 0.29) and the low-performing teams (N = 7, Mdn = -0.12, Q1 = -0.32, Q3 = -0.02) on the first dimension (U = 50, p = 0.012, r = 0.53).
The high-performing teams only made one strong connection from acknowledgement to information request. This suggests that the high-performing teams frequently asked questions in the middle of a dialogue segment, which may demonstrate their engagement in team communication. Yet, the larger number of red edges suggests that low-performing teams may have over-emphasised the secondary tasks. The node escalation also reveals important differences between the two groups. Specifically, regarding all the edges connected to escalation, the ones pointed towards escalation, namely from task allocation, information request, and acknowledgement, are all from the low-performing teams. This suggests that the low-performing teams sometimes allocated tasks or exchanged information and then announced escalation. Since this happened in the space of the secondary tasks, the low-performing teams may have incorrectly announced escalation on stable patients. In contrast, the edges pointing away from escalation are all from the high-performing teams. This shows that the high-performing teams sometimes announced escalation and then arranged tasks or exchanged information. This may happen in a situation that the student working on primary tasks went to the space of secondary tasks to inform the student working in this space about escalation of the patient at risk and asked a student for help.
Regarding the communication in the secondary tasks' space in phase 3, the Mann-Whitney U tests also showed a significant difference between the highperforming teams (N = 8, Mdn = 0.013, Q1 = 0.01, Q3 = 0.16) and the lowperforming teams (N = 7, Mdn = -0.13, Q1 = -0.26, Q3 = -0.08) on the first dimension (U = 56, p = 0.001, r = 0.50). The main finding is that lowperforming teams still overemphasised the secondary tasks in phase 3, as the majority of edges in Fig. 3 (right) appeared in low-performing teams.
To address our RQ, we used multimodal data and ONA to model the ordered communication behaviours in embodied team learning where dialogue segments were spatially distributed. From the analysis of mean network subtractions, the effective team learning practice in this setting involves the following behaviours.
(1) Prioritising primary tasks. Through the network subtractions for primary tasks and secondary tasks, we found that high-performing teams prioritised the primary tasks while low-performing teams prioritised secondary tasks. This is aligned with the healthcare literature that suggests that developing effective patient prioritisation skills is critical in this kind of learning settings [7]. (2) Timely and correctly performing critical tasks (i.e., escalation and providing handover information). The results illustrate that high-performing teams announced escalation on the correct patient timely (in phase 2) and provided abundant handover information. In contrast, low-performing teams more frequently announced escalation on the wrong (stable) patients or late in phase 3, putting less effort into providing handover information. This suggests that timely and correctly performing critical tasks is an effective team practice, as announcing escalation correctly and timely is related to the effective application of medical resources to maximise patient safety [7], and providing handover information can contribute to effective team coordination [15]. (3) Coordinating the team efficiently. All four network subtractions demonstrated that low-performing teams put more effort into communicating about team coordination (stronger connections to planning and task allocation). This suggests that low-performing teams were less efficient in coordinating their teams, so they had to invest more effort on coordination, resulting in failure to demonstrate other key effective team behaviours.
This study has several implications for research in embodied teamwork education and co-located collaborative learning. We demonstrated how ONA graphs can be useful to analyse communication in an embodied team learning setting by modelling multimodal data. Other researchers can adapt this method to model and analyse verbal communication in similar learning settings, such as collaborative problem-solving [17] and laboratory classroom teaching [11]. We also considered the implication for students and teachers. However, current ONA visualisations can be hard for them to understand due to their complexity. Yet, future work can explore ways to effectively communicate the insights from ONA in ways that students or teachers can understand.
Regarding limitations and future work, the first limitation is the generalisability of the findings. The sample size in this study is limited (60 students in 15 teams) and the interpretations of the results are based on a specific learning design, so the findings are not meant to be generalisable. Another limitation results from the manual transcription and coding of the communication contents. This currently limits the portability and scalability [33] of this method. Overcoming this limitation is our future work. We plan to fully automate the transcription and coding procedures using speech-to-text (e.g., the recently released OpenAI whisper [18]) and natural language processing techniques [8]. Another future work is to design a method to convey the findings in ONA to students and teachers in an intuitive way to enable practical application.
Regarding ethical considerations, although spatial data in this study was anonymous, the audio recordings can lead to privacy concerns. However, it is a common practice to record audio data in similar studies [16,17]. To minimise privacy concerns, we used colours to represent the students and never collected any identity information. Furthermore, we controlled the access to our dataset to prevent any unintended use [33].
In conclusion, we presented a method to extract communication behaviours in embodied learning settings from multimodal data and analyse them using ONA. We illustrated the capability of this method to identify the key factors for differentiating high and low-performing teams in a team-based embodied healthcare simulation. The method in this paper can benefit practitioners to support their teaching and researchers studying embodied teamwork.