Title: What Did They Learn? Objective Assessment Tools Show Mixed Effects of Training on Science Communication Behaviors
There is widespread agreement about the need to assess the success of programs training scientists to communicate more effectively with non-professional audiences. However, there is little agreement about how that should be done. What do we mean when we talk about “effective communication”? What should we measure? How should we measure it? Evaluation of communication training programs often incorporates the views of students or trainers themselves, although this is widely understood to bias the assessment. We recently completed a 3-year experiment to use audiences of non-scientists to evaluate the effect of training on STEM (Science, Technology, Engineering and Math) graduate students’ communication ability. Overall, audiences rated STEM grad students’ communication performance no better after training than before, as we reported in Rubega et al. 2018. However, audience ratings do not reveal whether training changed specific trainee communication behaviors (e.g., jargon use, narrative techniques) even if too little to affect trainees’ overall success. Here we measure trainee communication behavior directly, using multiple textual analysis tools and analysis of trainees’ body language during videotaped talks. We found that student use of jargon declined after training but that use of narrative techniques did not increase. Flesch Reading Ease and Flesch-Kincaid Grade Level scores, used as indicators of complexity of sentences and word choice, were no different after instruction. Trainees’ movement of hands and hesitancy during talks was correlated negatively with audience ratings of credibility and clarity; smiling, on the other hand, was correlated with improvement in credibility, clarity and engagement scores given by audience members. We show that objective tools can be used to measure the success of communication training programs, that non-verbal cues are associated with audience judgments, and that an intensive communication course does change some, if not all, communication behaviors. more »« less
Rubega, Margaret A.; Burgio, Kevin R.; MacDonald, A. Andrew; Oeldorf-Hirsch, Anne; Capers, Robert S.; Wyss, Robert(
, Science Communication)
null
(Ed.)
As the science community has recognized the vital role of communicating to the public, science communication training has proliferated. The development of rigorous, comparable approaches to assessment of training has not kept pace. We conducted a fully controlled experiment using a semester-long science communication course, and audience assessment of communicator performance. Evaluators scored the communication competence of trainees and their matched, untrained controls, before and after training. Bayesian analysis of the data showed very small gains in communication skills of trainees, and no difference from untrained controls. High variance in scores suggests little agreement on what constitutes “good” communication.
King-Kostelac, Amelia; Gomez, Eres A.; Finucane, Mary M.; Gorton, Sarah; Killian, Jamie L.; Walker, Kenneth; Bush, Janis K.; Smith, Jennifer A.(
, Frontiers in Environmental Science)
Graduate students emerging from STEM programs face inequitable professional landscapes in which their ability to practice inclusive and effective science communication with interdisciplinary and public audiences is essential to their success. Yet these students are rarely offered the opportunity to learn and practice inclusive science communication in their graduate programs. Moreover, minoritized students rarely have the opportunity to validate their experiences among peers and develop professional sensibilities through research training. In this article, the authors offer the Science Communication (Sci/Comm) Scholar’s working group at The University of Texas at San Antonio as one model for training graduate students in human dimensions and inclusive science communication for effective public engagement in thesis projects and beyond. The faculty facilitated peer-to-peer working group encouraged participation by women who often face inequities in STEM workplaces. Early results indicate that team-based training in both the science and art of public engagement provides critical exposure to help students understand the methodological care needed for human dimensions research, and to facilitate narrative-based citizen science engagements. The authors demonstrate this through several brief profiles of environmental science graduate students’ thesis projects. Each case emphasizes the importance of research design for public engagement via quantitative surveys and narrative-based science communication interventions. Through a faculty facilitated peer-to-peer working group framework, research design and methodological care function as an integration point for social scientific and rhetorical training for inclusive science communication with diverse audiences.
A recently launched National Science Foundation Research Traineeship (NRT) aims to enhance graduate education by integrating research and professional skill development within a diverse, inclusive and supportive academy. This contribution will describe three initial interventions within this NRT, namely, an onboarding and orientation event, a career exploration symposium, and a multidisciplinary introductory course. In addition, the assessment of each of these interventions – and the outcomes thereof – will be presented and discussed.
Prior to the onboarding and orientation event, trainees received the event’s agenda and checklists summarizing pre- and post-event assignments. Pre-event assignments were designed to familiarize trainees with the NRT, the process of drafting an individual development plan (IDP), and the consent form required for traineeship evaluation purposes. During the event – held online due to COVID-19 – and following introductions, trainees were given the opportunity to ask questions stemming from the pre-event assignments. Subsequently, trainees were introduced to several tools (e.g., checklists as well as sample developmental network maps and mentoring contracts) to guide and track their development and progression through the traineeship. The event concluded with a discussion on topics that also constituted post-event assignments, including registering and preparing for both the career exploration symposium and the multidisciplinary introductory course. Survey data collected after the event indicated that trainees valued the opportunity to learn more about the NRT, ask questions, and meet faculty who expressed a commitment to student success.
Shortly thereafter, trainees attended a career exploration symposium and moderated sessions featuring speakers representing careers of interest. Indeed, the symposium was purposely designed to expose trainees to a wide range of career pathways. In addition, practical career tools and skills for STEM professionals were discussed in several breakout sessions. Finally, the symposium ended with a panel discussion comprising four diverse and accomplished recent Ph.D. graduates, who discussed mental health and communication issues prior to answering questions asked by trainees. Trainee responses to a post-symposium survey were also positive as trainees reported the following: an increase in knowledge of career paths and hiring sectors, an appreciation for the diversity of the presenters and career paths, and the attainment of at least one new skill or strategy they felt would aid in their graduate school success.
In their first semester in the NRT, trainees take an interdisciplinary course covering the high priority convergent research topic targeted by the traineeship. This course is co-taught by faculty of seven different departments and is composed of four units, each focused on a research question requiring extensive interdisciplinary collaboration to be answered. Teams of at least three core faculty with the cumulative expertise needed to answer each question co-teach each unit, emphasizing concepts that students must understand to address the question at hand. During this course, four multi-departmental interdisciplinary student teams are formed, each focusing on – and conducting a critical review of the literature in – one of the research questions. Indeed, emphasis is placed on providing students with the knowledge and tools to find, critically evaluate, summarize, and present literature on the topic.
Art is a common approach for communicating and educating about science, yet it remains unclear the extent to which science art can benefit varied audiences in varied contexts. To examine this gap, we developed an art exhibit based on the findings of two publications in disease ecology. In study 1, we asked visitors with varying formal science, technology, engineering, and math (STEM) education backgrounds to complete a survey about their interest in science research before and after viewing the exhibit. In study 2, we recruited upper-level ecology undergraduate students to receive one of three treatments: engage with the art exhibit, read the abstracts of the papers, or do neither. Students completed a comprehension quiz immediately after their learning treatment and again 2 weeks later to evaluate retention. Following the exhibit, visitors who did not report a career or major in STEM showed a greater increase in research interest than visitors who did report a career or major in STEM. For the ecology undergraduate students, comprehension quiz scores were higher for students in the abstract group than the art exhibit group, while both groups scored higher than the control group. Retention of information did not significantly differ between the three groups. Overall, these findings suggest that science art exhibits are an effective method for increasing the accessibility of science to broader audiences and that audience identifiers (e.g., level of formal education in STEM) play an important role in audience experience of science communication and science education initiatives.
Plaisime, Marie V.; Jipguep‐Akhtar, Marie; Locascio, Joseph J.; Belcher, Harolyn M. E.; Hardeman, Rachel R.; Picho‐Kiroga, Katherine; Perry, Sylvia P.; Phelan, Sean M.; van Ryn, Michelle; Dovidio, John F.(
, Health Services Research)
AbstractObjective
To examine the experience of interracial anxiety among health professionals and how it may affect the quality of their interactions with patients from racially marginalized populations. We explored the influence of prior interracial exposure—specifically through childhood neighborhoods, college student bodies, and friend groups—on interracial anxiety among medical students and residents. We also examined whether levels of interracial anxiety change from medical school through residency.
Data Source
Web‐based longitudinal survey data from the Medical Student Cognitive Habits and Growth Evaluation Study.
Study Design
We used a retrospective longitudinal design with four observations for each trainee. The study population consisted of non‐Black US medical trainees surveyed in their 1st and 4th years of medical school and 2nd and 3rd years of residency. Mixed effects longitudinal models were used to assess predictors of interracial anxiety and assess changes in interracial anxiety scores over time.
Principal Findings
In total, 3155 non‐Black medical trainees were followed for 7 years. Seventy‐eight percent grew up in predominantly White neighborhoods. Living in predominantly White neighborhoods and having less racially diverse friends were associated with higher levels of interracial anxiety among medical trainees. Trainees' interracial anxiety scores did not substantially change over time; interracial anxiety was highest in the 1st year of medical school, lowest in the 4th year, and increased slightly during residency.
Conclusions
Neighborhood and friend group composition had independent effects on interracial anxiety, indicating that premedical racial socialization may affect medical trainees' preparedness to interact effectively with diverse patient populations. Additionally, the lack of substantial change in interracial anxiety throughout medical training suggests the importance of providing curricular tools and structure (e.g., instituting interracial cooperative learning activities) to foster the development of healthy interracial relationships.
Capers, Robert S., Oeldorf-Hirsch, Anne, Wyss, Robert, Burgio, Kevin R., and Rubega, Margaret A. What Did They Learn? Objective Assessment Tools Show Mixed Effects of Training on Science Communication Behaviors. Retrieved from https://par.nsf.gov/biblio/10342400. Frontiers in Communication 6. Web. doi:10.3389/fcomm.2021.805630.
Capers, Robert S., Oeldorf-Hirsch, Anne, Wyss, Robert, Burgio, Kevin R., & Rubega, Margaret A. What Did They Learn? Objective Assessment Tools Show Mixed Effects of Training on Science Communication Behaviors. Frontiers in Communication, 6 (). Retrieved from https://par.nsf.gov/biblio/10342400. https://doi.org/10.3389/fcomm.2021.805630
Capers, Robert S., Oeldorf-Hirsch, Anne, Wyss, Robert, Burgio, Kevin R., and Rubega, Margaret A.
"What Did They Learn? Objective Assessment Tools Show Mixed Effects of Training on Science Communication Behaviors". Frontiers in Communication 6 (). Country unknown/Code not available. https://doi.org/10.3389/fcomm.2021.805630.https://par.nsf.gov/biblio/10342400.
@article{osti_10342400,
place = {Country unknown/Code not available},
title = {What Did They Learn? Objective Assessment Tools Show Mixed Effects of Training on Science Communication Behaviors},
url = {https://par.nsf.gov/biblio/10342400},
DOI = {10.3389/fcomm.2021.805630},
abstractNote = {There is widespread agreement about the need to assess the success of programs training scientists to communicate more effectively with non-professional audiences. However, there is little agreement about how that should be done. What do we mean when we talk about “effective communication”? What should we measure? How should we measure it? Evaluation of communication training programs often incorporates the views of students or trainers themselves, although this is widely understood to bias the assessment. We recently completed a 3-year experiment to use audiences of non-scientists to evaluate the effect of training on STEM (Science, Technology, Engineering and Math) graduate students’ communication ability. Overall, audiences rated STEM grad students’ communication performance no better after training than before, as we reported in Rubega et al. 2018. However, audience ratings do not reveal whether training changed specific trainee communication behaviors (e.g., jargon use, narrative techniques) even if too little to affect trainees’ overall success. Here we measure trainee communication behavior directly, using multiple textual analysis tools and analysis of trainees’ body language during videotaped talks. We found that student use of jargon declined after training but that use of narrative techniques did not increase. Flesch Reading Ease and Flesch-Kincaid Grade Level scores, used as indicators of complexity of sentences and word choice, were no different after instruction. Trainees’ movement of hands and hesitancy during talks was correlated negatively with audience ratings of credibility and clarity; smiling, on the other hand, was correlated with improvement in credibility, clarity and engagement scores given by audience members. We show that objective tools can be used to measure the success of communication training programs, that non-verbal cues are associated with audience judgments, and that an intensive communication course does change some, if not all, communication behaviors.},
journal = {Frontiers in Communication},
volume = {6},
author = {Capers, Robert S. and Oeldorf-Hirsch, Anne and Wyss, Robert and Burgio, Kevin R. and Rubega, Margaret A.},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
