skip to main content


Title: Objective Communication Patterns Associated With Team Member Effectiveness in Real-World Virtual Teams
Objective

We explore the relationships between objective communication patterns displayed during virtual team meetings and established, qualitative measures of team member effectiveness.

Background

A key component of teamwork is communication. Automated measures of objective communication patterns are becoming more feasible and offer the ability to measure and monitor communication in a scalable, consistent and continuous manner. However, their validity in reflecting meaningful measures of teamwork processes are not well established, especially in real-world settings.

Method

We studied real-world virtual student teams working on semester-long projects. We captured virtual team meetings using the Zoom video conferencing platform throughout the semester and periodic surveys comprising peer ratings of team member effectiveness. Leveraging audio transcripts, we examined relationships between objective measures of speaking time, silence gap duration and vocal turn-taking and peer ratings of team member effectiveness.

Results

Speaking time, speaking turn count, degree centrality and (marginally) speaking turn duration, but not silence gap duration, were positively related to individual-level team member effectiveness. Time in dyadic interactions and interaction count, but not interaction length, were positively related to dyad-level team member effectiveness.

Conclusion

Our study highlights the relevance of objective measures of speaking time and vocal turn-taking to team member effectiveness in virtual project-based teams, supporting the validity of these objective measures and their use in future research.

Application

Our approach offers a scalable, easy-to-use method for measuring communication patterns and team member effectiveness in virtual teams and opens the opportunity to study these patterns in a more continuous and dynamic manner.

 
more » « less
Award ID(s):
1910117
PAR ID:
10387538
Author(s) / Creator(s):
 ;  ;  ;  ;  ;  ;  ;  
Publisher / Repository:
SAGE Publications
Date Published:
Journal Name:
Human Factors: The Journal of the Human Factors and Ergonomics Society
Volume:
66
Issue:
5
ISSN:
0018-7208
Format(s):
Medium: X Size: p. 1414-1430
Size(s):
p. 1414-1430
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract Background

    Effectively facilitating teamwork experiences, particularly in the context of large-size courses, is difficult to implement. This study seeks to address the challenges of implementing effective teamwork experiences in large courses. This study integrated teamwork pedagogy to facilitate a semester-long project in the context of a large-size class comprising 118 students organized into 26 teams. The data for this study were collected from two online teamwork sessions when teams collaborated and self-recorded during the in-class time. The video recordings were qualitatively analyzed to identify patterns in team dynamics processes through visualizations. The study aims to provide insights into the different ways team members engaged in team dynamics processes during different phases of the semester.

    Results

    Findings suggest that members of teams were mostly active and passive during meetings and less constructive and interactive in their engagement. Team members mainly engaged in communication, team orientation, and feedback behaviors. Over time, team members' interactions with one another remained about the same, with feedback behaviors tending to diminish and coordination behaviors staying about the same or slightly increasing over time.

    Conclusion

    The implications of this study extend to both practice and theory. Practically, combining cooperative learning and scrum practices enabled a blend of collaborative and cooperative work, which suggests providing teams with tools and structures to coordinate teamwork processes and promote interaction among team members. From a theoretical perspective, this study contributes to the understanding of temporal aspects of teamwork dynamics by examining how team interactions evolve during working sessions at different points in time. Overall, this research provides valuable insights for educators, practitioners, and researchers aiming to enhance teamwork experiences in large courses, particularly in software development disciplines.

     
    more » « less
  2. Abstract Background

    Teamwork has become a central element of engineering education. However, the race‐ and gender‐based marginalization prevalent in society is also prevalent in engineering student teams. These problematic dynamics limit learning opportunities, isolate historically marginalized students, and ultimately push students away from engineering, further reinforcing the demographic imbalances in the profession.

    Purpose

    While there are strategies to improve the experiences of marginalized students within teams, there are few tools for detecting marginalizing behaviors as they occur. The purpose of this work is to examine how peer evaluations collected as a normal part of an engineering course can be used as a window into team dynamics to reveal marginalization as it occurs.

    Method

    We used a semester of peer evaluation data from a large engineering course in which a team project is the central assignment and peer evaluation occurs four times during the course. We designed an algorithm to identify teams where marginalization may be occurring. We then performed qualitative analyses using a sociolinguistic analysis.

    Results

    Results show that the algorithm helps identify teams where marginalization occurs. Qualitative analyses of four illustrative cases demonstrated the stealth appearance and evolution of marginalization, providing strong evidence that hidden within language of peer evaluation are indicators of marginalization. Based on the wider dataset, we present a taxonomy (eight categories) of linguistic marginalization appearing in peer comments.

    Conclusion

    Both peer evaluation scores and the language used in peer evaluations can reveal team inequities and may serve as a near‐real‐time mechanism to interrupt marginalization within engineering teams.

     
    more » « less
  3. Assessing team software development projects is notoriously difficult and typically based on subjective metrics. To help make assessments more rigorous, we conducted an empirical study to explore relationships between subjective metrics based on peer and instructor assessments, and objective metrics based on GitHub and chat data. We studied 23 undergraduate software teams (n= 117 students) from two undergraduate computing courses at two North American research universities. We collected data on teams’ (a) commits and issues from their GitHub code repositories, (b) chat messages from their Slack and Microsoft Teams channels, (c) peer evaluation ratings from the CATME peer evaluation system, and (d) individual assignment grades from the courses. We derived metrics from (a) and (b) to measure both individual team members’contributionsto the team, and theequalityof team members’ contributions. We then performed Pearson analyses to identify correlations among the metrics, peer evaluation ratings, and individual grades. We found significant positive correlations between team members’ GitHub contributions, chat contributions, and peer evaluation ratings. In addition, the equality of teams’ GitHub contributions was positively correlated with teams’ average peer evaluation ratings and negatively correlated with the variance in those ratings. However, no such positive correlations were detected between the equality of teams’ chat contributions and their peer evaluation ratings. Our study extends previous research results by providing evidence that (a) team members’ chat contributions, like their GitHub contributions, are positively correlated with their peer evaluation ratings; (b) team members’ chat contributions are positively correlated with their GitHub contributions; and (c) the equality of team’ GitHub contributions is positively correlated with their peer evaluation ratings. These results lend further support to the idea that combining objective and subjective metrics can make the assessment of team software projects more comprehensive and rigorous. 
    more » « less
  4. null (Ed.)
    Engineers need to develop professional skills, including the ability to work successfully in teams and to communicate within and outside of their discipline, in addition to required technical skills. A collaborative multi-disciplinary service learning project referred to as Ed+gineering was implemented in a 100-level mechanical engineering course. In this collaboration, mechanical engineering students, primarily in the second semester of their freshman year or first semester of their second year, worked over the course of a semester with education students taking a foundations course to develop and deliver engineering lessons to fourth or fifth graders. Students in comparison engineering classes worked on a team project focused on experimental design for a small satellite system. The purpose of this study was to determine if participating in the Ed+gineering collaboration had a positive effect on teamwork effectiveness and satisfaction when compared to the comparison class. In both team projects, the five dimensions of the Comprehensive Assessment of Team Member Effectiveness (CATME) system were used as a quantitative assessment. The five dimensions of CATME Behaviorally Anchored Ratings Scale (BARS) (contribution to the team’s work, interacting with teammates, keeping the team on track, expecting quality, and having relevant Knowledge, Skills, and Abilities - KSAs) were measured. Additionally, within the CATME platform team satisfaction, team interdependence and team cohesiveness were measured. ANCOVA analysis was used to assess the quantitative data from CATME. Preliminary results suggest that students in the treatment classes had higher team member effectiveness and overall satisfaction scores than students in the comparison classes. Qualitative data from reflections written at the completion of the aforementioned projects were used to explore these results. 
    more » « less
  5. Objective This work examines two human–autonomy team (HAT) training approaches that target communication and trust calibration to improve team effectiveness under degraded conditions. Background Human–autonomy teaming presents challenges to teamwork, some of which may be addressed through training. Factors vital to HAT performance include communication and calibrated trust. Method Thirty teams of three, including one confederate acting as an autonomous agent, received either entrainment-based coordination training, trust calibration training, or control training before executing a series of missions operating a simulated remotely piloted aircraft. Automation and autonomy failures simulating degraded conditions were injected during missions, and measures of team communication, trust, and task efficiency were collected. Results Teams receiving coordination training had higher communication anticipation ratios, took photos of targets faster, and overcame more autonomy failures. Although autonomy failures were introduced in all conditions, teams receiving the calibration training reported that their overall trust in the agent was more robust over time. However, they did not perform better than the control condition. Conclusions Training based on entrainment of communications, wherein introduction of timely information exchange through one team member has lasting effects throughout the team, was positively associated with improvements in HAT communications and performance under degraded conditions. Training that emphasized the shortcomings of the autonomous agent appeared to calibrate expectations and maintain trust. Applications Team training that includes an autonomous agent that models effective information exchange may positively impact team communication and coordination. Training that emphasizes the limitations of an autonomous agent may help calibrate trust. 
    more » « less