There is widespread agreement about the need to assess the success of programs training scientists to communicate more effectively with non-professional audiences. However, there is little agreement about how that should be done. What do we mean when we talk about “effective communication”? What should we measure? How should we measure it? Evaluation of communication training programs often incorporates the views of students or trainers themselves, although this is widely understood to bias the assessment. We recently completed a 3-year experiment to use audiences of non-scientists to evaluate the effect of training on STEM (Science, Technology, Engineering and Math) graduate students’ communication ability. Overall, audiences rated STEM grad students’ communication performance no better after training than before, as we reported in Rubega et al. 2018. However, audience ratings do not reveal whether training changed specific trainee communication behaviors (e.g., jargon use, narrative techniques) even if too little to affect trainees’ overall success. Here we measure trainee communication behavior directly, using multiple textual analysis tools and analysis of trainees’ body language during videotaped talks. We found that student use of jargon declined after training but that use of narrative techniques did not increase. Flesch Reading Ease and Flesch-Kincaid Grade Level scores, used as indicators of complexity of sentences and word choice, were no different after instruction. Trainees’ movement of hands and hesitancy during talks was correlated negatively with audience ratings of credibility and clarity; smiling, on the other hand, was correlated with improvement in credibility, clarity and engagement scores given by audience members. We show that objective tools can be used to measure the success of communication training programs, that non-verbal cues are associated with audience judgments, and that an intensive communication course does change some, if not all, communication behaviors.
more »
« less
Assessment by Audiences Shows Little Effect of Science Communication Training
As the science community has recognized the vital role of communicating to the public, science communication training has proliferated. The development of rigorous, comparable approaches to assessment of training has not kept pace. We conducted a fully controlled experiment using a semester-long science communication course, and audience assessment of communicator performance. Evaluators scored the communication competence of trainees and their matched, untrained controls, before and after training. Bayesian analysis of the data showed very small gains in communication skills of trainees, and no difference from untrained controls. High variance in scores suggests little agreement on what constitutes “good” communication.
more »
« less
- Award ID(s):
- 2022036
- NSF-PAR ID:
- 10286231
- Date Published:
- Journal Name:
- Science Communication
- Volume:
- 43
- Issue:
- 2
- ISSN:
- 1075-5470
- Page Range / eLocation ID:
- 139 to 169
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Despite being disproportionately impacted by health disparities, Black, Hispanic, Indigenous, and other underrepresented populations account for a significant minority of graduates in biomedical data science-related disciplines. Given their commitment to educating underrepresented students and trainees, minority serving institutions (MSIs) can play a significant role in enhancing diversity in the biomedical data science workforce. Little has been published about the reach, curricular breadth, and best practices for delivering these data science training programs. The purpose of this paper is to summarize six Research Centers in Minority Institutions (RCMIs) awarded funding from the National Institute of Minority Health Disparities (NIMHD) to develop new data science training programs. A cross-sectional survey was conducted to better understand the demographics of learners served, curricular topics covered, methods of instruction and assessment, challenges, and recommendations by program directors. Programs demonstrated overall success in reach and curricular diversity, serving a broad range of students and faculty, while also covering a broad range of topics. The main challenges highlighted were a lack of resources and infrastructure and teaching learners with varying levels of experience and knowledge. Further investments in MSIs are needed to sustain training efforts and develop pathways for diversifying the biomedical data science workforce.more » « less
-
more » « less
Abstract Interdisciplinary teams are on the rise as scientists attempt to address complex environmental issues. While the benefits of team science approaches are clear, researchers often struggle with its implementation, particularly for new team members. The challenges of large projects often weigh on the most vulnerable members of a team: trainees, including undergraduate students, graduate students, and post‐doctoral researchers. Trainees on big projects have to navigate their role on the team, with learning project policies, procedures, and goals, all while also training in key scientific tasks such as co‐authoring papers. To address these challenges, we created and participated in a project‐specific, graduate‐level team science course. The purposes of this course were to: (1) introduce students to the goals of the project, (2) build trainees' understanding of how big projects operate, and (3) allow trainees to explore how their research interests dovetailed with the overall project. Additionally, trainees received training regarding: (1) diversity, equity & inclusion, (2) giving and receiving feedback, and (3) effective communication. Onboarding through the team science course cultivated psychological safety and a collaborative student community across disciplines and institutions. Thus, we recommend a team science course for onboarding students to big projects to help students establish the skills necessary for collaborative research. Project‐based team science classes can benefit student advancement, enhance the productivity of the project, and accelerate the discovery of solutions to ecological issues by building community, establishing a shared project vocabulary, and building a workforce with collaborative skills to better answer ecological research questions.
-
null (Ed.)A recently launched National Science Foundation Research Traineeship (NRT) aims to enhance graduate education by integrating research and professional skill development within a diverse, inclusive and supportive academy. This contribution will describe three initial interventions within this NRT, namely, an onboarding and orientation event, a career exploration symposium, and a multidisciplinary introductory course. In addition, the assessment of each of these interventions – and the outcomes thereof – will be presented and discussed. Prior to the onboarding and orientation event, trainees received the event’s agenda and checklists summarizing pre- and post-event assignments. Pre-event assignments were designed to familiarize trainees with the NRT, the process of drafting an individual development plan (IDP), and the consent form required for traineeship evaluation purposes. During the event – held online due to COVID-19 – and following introductions, trainees were given the opportunity to ask questions stemming from the pre-event assignments. Subsequently, trainees were introduced to several tools (e.g., checklists as well as sample developmental network maps and mentoring contracts) to guide and track their development and progression through the traineeship. The event concluded with a discussion on topics that also constituted post-event assignments, including registering and preparing for both the career exploration symposium and the multidisciplinary introductory course. Survey data collected after the event indicated that trainees valued the opportunity to learn more about the NRT, ask questions, and meet faculty who expressed a commitment to student success. Shortly thereafter, trainees attended a career exploration symposium and moderated sessions featuring speakers representing careers of interest. Indeed, the symposium was purposely designed to expose trainees to a wide range of career pathways. In addition, practical career tools and skills for STEM professionals were discussed in several breakout sessions. Finally, the symposium ended with a panel discussion comprising four diverse and accomplished recent Ph.D. graduates, who discussed mental health and communication issues prior to answering questions asked by trainees. Trainee responses to a post-symposium survey were also positive as trainees reported the following: an increase in knowledge of career paths and hiring sectors, an appreciation for the diversity of the presenters and career paths, and the attainment of at least one new skill or strategy they felt would aid in their graduate school success. In their first semester in the NRT, trainees take an interdisciplinary course covering the high priority convergent research topic targeted by the traineeship. This course is co-taught by faculty of seven different departments and is composed of four units, each focused on a research question requiring extensive interdisciplinary collaboration to be answered. Teams of at least three core faculty with the cumulative expertise needed to answer each question co-teach each unit, emphasizing concepts that students must understand to address the question at hand. During this course, four multi-departmental interdisciplinary student teams are formed, each focusing on – and conducting a critical review of the literature in – one of the research questions. Indeed, emphasis is placed on providing students with the knowledge and tools to find, critically evaluate, summarize, and present literature on the topic.more » « less
-
Graduate training often takes a monodisciplinary approach that is not informed by best practices, ignores the needs and preferences of students, and overlooks the increasingly interdisciplinary and international nature of research. This is unfortunate, particularly since graduate education that is fully integrated with interdisciplinary research can help students become part of a trained and diverse workforce equipped to meet society’s many challenges. Against this backdrop, a National Science Foundation Research Traineeship (NRT) program is being established at the University of Kentucky leveraging the most effective instruments for the training of STEM professionals, such as network-based graduate student mentoring and career preparation encompassing both technical and professional skillsets. Briefly, the training graduate students will receive – in a way that is fully integrated with the research they perform – includes: 1) tools such as individual development plans and developmental network maps; 2) a multi-departmental and interdisciplinary course on research-related content; 3) a seminar course on transferrable skills (ethics, research, communication, teaching, mentoring, entrepreneurship, teamwork, management, leadership, outreach, etc.); 4) a certificate to be awarded once students complete the two courses above and garner additional credits from an interdisciplinary curriculum of research-related courses; 5) summer internships at other departments and at external institutions (other universities, industry, national laboratories) nationwide or abroad; 6) an annual research-related symposium including all elements of a scientific conference; 7) internal collaborative research grants for participants to fund and pursue their own ideas; 8) fields trips to facilities related to the research; and 9) coaching on job hunting as well as résumé, motivation letter and interview preparation. Since a workforce equipped to meet society’s challenges must be both well trained and diverse, multiple initiatives will ensure that this NRT will broaden participation in STEM. Recruitment-wise, close collaboration with a number of entities will provide this NRT with a broad recruitment pool of talented and diverse students. Moreover, collaboration with these entities will provide trainees with ample opportunities to acquire, practice and refine their professional skills, as trainees present their results and recruit in conferences, meetings and outreach events organized by these entities, become members and/or join their leadership, and expand their professional and mentoring network in the process. In addition, minority trainees will be surveyed periodically to probe their feelings of well-being, preparation, acceptance, belonging and distress, as well as their perception of how well structured their departments and programs are. According to recent literature, these factors determine whether or not they perform (i.e., publish) at rates comparable to their male majority peers. Saliently, the evaluation of the educational model employed will afford a comprehensive understanding not only of the academy components that were more utilized and impactful, but will reveal the individual mentoring and skill-building facets of the program driving its successful implementation. The evaluation plan includes outcomes, performance measures, an evaluation timetable, benchmarks and a description of how formative evaluation will improve practice, the evaluation process also extending to research activities.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1177/1075547020971639
