skip to main content

Title: Assessment by Audiences Shows Little Effect of Science Communication Training
As the science community has recognized the vital role of communicating to the public, science communication training has proliferated. The development of rigorous, comparable approaches to assessment of training has not kept pace. We conducted a fully controlled experiment using a semester-long science communication course, and audience assessment of communicator performance. Evaluators scored the communication competence of trainees and their matched, untrained controls, before and after training. Bayesian analysis of the data showed very small gains in communication skills of trainees, and no difference from untrained controls. High variance in scores suggests little agreement on what constitutes “good” communication.
Authors:
; ; ; ; ;
Award ID(s):
2022036
Publication Date:
NSF-PAR ID:
10286231
Journal Name:
Science Communication
Volume:
43
Issue:
2
Page Range or eLocation-ID:
139 to 169
ISSN:
1075-5470
Sponsoring Org:
National Science Foundation
More Like this
  1. There is widespread agreement about the need to assess the success of programs training scientists to communicate more effectively with non-professional audiences. However, there is little agreement about how that should be done. What do we mean when we talk about “effective communication”? What should we measure? How should we measure it? Evaluation of communication training programs often incorporates the views of students or trainers themselves, although this is widely understood to bias the assessment. We recently completed a 3-year experiment to use audiences of non-scientists to evaluate the effect of training on STEM (Science, Technology, Engineering and Math) graduate students’ communication ability. Overall, audiences rated STEM grad students’ communication performance no better after training than before, as we reported in Rubega et al. 2018. However, audience ratings do not reveal whether training changed specific trainee communication behaviors (e.g., jargon use, narrative techniques) even if too little to affect trainees’ overall success. Here we measure trainee communication behavior directly, using multiple textual analysis tools and analysis of trainees’ body language during videotaped talks. We found that student use of jargon declined after training but that use of narrative techniques did not increase. Flesch Reading Ease and Flesch-Kincaid Grade Level scores,more »used as indicators of complexity of sentences and word choice, were no different after instruction. Trainees’ movement of hands and hesitancy during talks was correlated negatively with audience ratings of credibility and clarity; smiling, on the other hand, was correlated with improvement in credibility, clarity and engagement scores given by audience members. We show that objective tools can be used to measure the success of communication training programs, that non-verbal cues are associated with audience judgments, and that an intensive communication course does change some, if not all, communication behaviors.« less
  2. A recently launched National Science Foundation Research Traineeship (NRT) aims to enhance graduate education by integrating research and professional skill development within a diverse, inclusive and supportive academy. This contribution will describe three initial interventions within this NRT, namely, an onboarding and orientation event, a career exploration symposium, and a multidisciplinary introductory course. In addition, the assessment of each of these interventions – and the outcomes thereof – will be presented and discussed. Prior to the onboarding and orientation event, trainees received the event’s agenda and checklists summarizing pre- and post-event assignments. Pre-event assignments were designed to familiarize trainees with the NRT, the process of drafting an individual development plan (IDP), and the consent form required for traineeship evaluation purposes. During the event – held online due to COVID-19 – and following introductions, trainees were given the opportunity to ask questions stemming from the pre-event assignments. Subsequently, trainees were introduced to several tools (e.g., checklists as well as sample developmental network maps and mentoring contracts) to guide and track their development and progression through the traineeship. The event concluded with a discussion on topics that also constituted post-event assignments, including registering and preparing for both the career exploration symposium andmore »the multidisciplinary introductory course. Survey data collected after the event indicated that trainees valued the opportunity to learn more about the NRT, ask questions, and meet faculty who expressed a commitment to student success. Shortly thereafter, trainees attended a career exploration symposium and moderated sessions featuring speakers representing careers of interest. Indeed, the symposium was purposely designed to expose trainees to a wide range of career pathways. In addition, practical career tools and skills for STEM professionals were discussed in several breakout sessions. Finally, the symposium ended with a panel discussion comprising four diverse and accomplished recent Ph.D. graduates, who discussed mental health and communication issues prior to answering questions asked by trainees. Trainee responses to a post-symposium survey were also positive as trainees reported the following: an increase in knowledge of career paths and hiring sectors, an appreciation for the diversity of the presenters and career paths, and the attainment of at least one new skill or strategy they felt would aid in their graduate school success. In their first semester in the NRT, trainees take an interdisciplinary course covering the high priority convergent research topic targeted by the traineeship. This course is co-taught by faculty of seven different departments and is composed of four units, each focused on a research question requiring extensive interdisciplinary collaboration to be answered. Teams of at least three core faculty with the cumulative expertise needed to answer each question co-teach each unit, emphasizing concepts that students must understand to address the question at hand. During this course, four multi-departmental interdisciplinary student teams are formed, each focusing on – and conducting a critical review of the literature in – one of the research questions. Indeed, emphasis is placed on providing students with the knowledge and tools to find, critically evaluate, summarize, and present literature on the topic.« less
  3. Graduate training often takes a monodisciplinary approach that is not informed by best practices, ignores the needs and preferences of students, and overlooks the increasingly interdisciplinary and international nature of research. This is unfortunate, particularly since graduate education that is fully integrated with interdisciplinary research can help students become part of a trained and diverse workforce equipped to meet society’s many challenges. Against this backdrop, a National Science Foundation Research Traineeship (NRT) program is being established at the University of Kentucky leveraging the most effective instruments for the training of STEM professionals, such as network-based graduate student mentoring and career preparation encompassing both technical and professional skillsets. Briefly, the training graduate students will receive – in a way that is fully integrated with the research they perform – includes: 1) tools such as individual development plans and developmental network maps; 2) a multi-departmental and interdisciplinary course on research-related content; 3) a seminar course on transferrable skills (ethics, research, communication, teaching, mentoring, entrepreneurship, teamwork, management, leadership, outreach, etc.); 4) a certificate to be awarded once students complete the two courses above and garner additional credits from an interdisciplinary curriculum of research-related courses; 5) summer internships at other departments and at externalmore »institutions (other universities, industry, national laboratories) nationwide or abroad; 6) an annual research-related symposium including all elements of a scientific conference; 7) internal collaborative research grants for participants to fund and pursue their own ideas; 8) fields trips to facilities related to the research; and 9) coaching on job hunting as well as résumé, motivation letter and interview preparation. Since a workforce equipped to meet society’s challenges must be both well trained and diverse, multiple initiatives will ensure that this NRT will broaden participation in STEM. Recruitment-wise, close collaboration with a number of entities will provide this NRT with a broad recruitment pool of talented and diverse students. Moreover, collaboration with these entities will provide trainees with ample opportunities to acquire, practice and refine their professional skills, as trainees present their results and recruit in conferences, meetings and outreach events organized by these entities, become members and/or join their leadership, and expand their professional and mentoring network in the process. In addition, minority trainees will be surveyed periodically to probe their feelings of well-being, preparation, acceptance, belonging and distress, as well as their perception of how well structured their departments and programs are. According to recent literature, these factors determine whether or not they perform (i.e., publish) at rates comparable to their male majority peers. Saliently, the evaluation of the educational model employed will afford a comprehensive understanding not only of the academy components that were more utilized and impactful, but will reveal the individual mentoring and skill-building facets of the program driving its successful implementation. The evaluation plan includes outcomes, performance measures, an evaluation timetable, benchmarks and a description of how formative evaluation will improve practice, the evaluation process also extending to research activities.« less
  4. Sustainability of the scientific enterprise requires being able to recruit, retain, and prepare ongoing generations of PhD-trained scientists ready to adapt with the evolving needs of the scientific workforce and society. This necessitates a broadened, trainee-centered view in doctoral and postdoctoral training—including a prominent focus on career planning, science across sectors, and development of professional skills. Although there is energy and movement to enhance graduate and postdoctoral training, actions remain disparate, leading to inefficiencies in implementation and lack of systemic change. In 2019, an emerging national initiative, Professional Development Hub (pd|hub), hosted a workshop to bring organizations and individuals together across stakeholder groups to discuss enhancing the development, dissemination, and widespread implementation of evidence-based practices for STEM graduate and postdoctoral education, with specific emphasis on career and professional development for PhD scientists. The fifty workshop participants represented nine key stakeholder groups: career development practitioners, scientific societies, disseminators, education researchers and evaluators, employers of PhD scientists, funders, professional associations, trainees, and university leaders and faculty. In addition, participants spanned different races, ethnicities, genders, disciplines, sectors, geographic locations, career stages, and levels of institutional resources. This report presents cross-cutting themes identified at the workshop, examples of stakeholder-specific perspectives, and recommended next steps.more »As part of the collective effort to develop a foundation for sustainable solutions, several actions were defined, including: incentivizing change at institutions and programs, curating and disseminating resources for evidence-based career and professional development educational practices, expanding evidence for effective training and mentoring, establishing expectations for STEM career and professional development, and improving communication across all stakeholders in STEM PhD education. Furthermore, the report describes national-level actions already moving forward via pd|hub in the months following the workshop. Building on a decade of reports and gatherings advocating for a shift in graduate and postdoctoral education, this workshop represented a key step and catalyst for change toward a more impactful future.« less
  5. null (Ed.)
    To enable the sustainable use of their ocean resources, capacity for ocean science and observations is important for every coastal nation. In many developing areas of the world, capability for ocean science and observations is not yet adequate to meet management needs. International organizations have employed a variety of capacity development approaches to assist developing countries in building self-sustaining ocean science and observational communities. This article describes the lessons learned from visiting scientist programs conducted for more than a decade by the Partnership for Observation of the Global Ocean (POGO) and the Scientific Committee on Oceanic Research (SCOR) that dispatched ocean scientists to developing countries to train hundreds of individuals in a variety of ocean science and observation topics and techniques. From these programs, SCOR and POGO have learned that training in-country has multiple benefits to trainees, host institutions, and trainers, benefits that are not achievable when students leave their countries. These benefits include more cost-effective training on issues relevant to the host institutions using locally available technology, as well as the ability to reach a large number of trainees. Lessons learned from the POGO and SCOR programs can be used to inform the future capacity-development activities of POGO andmore »SCOR, as well as other organizations, to improve, enhance, and expand the use of in-country training and mentoring. Such approaches could contribute to the capacity development efforts of the UN Decade of Ocean Science for Sustainable Development.« less