National Science Foundation (NSF) funded Engineering Research Centers (ERC) must complement their technical research with various education and outreach opportunities to: 1) improve and promote engineering education, both within the center and to the local community; 2) encourage and include the underrepresented populations to participate in Engineering activities; and 3) advocate communication and collaboration between industry and academia. ERCs ought to perform an adequate evaluation of their educational and outreach programs to ensure that beneficial goals are met. Each ERC has complete autonomy in conducting and reporting such evaluation. Evaluation tools used by individual ERCs are quite similar, but each ERC has designed their evaluation processes in isolation, including evaluation tools such as survey instruments, interview protocols, focus group protocols, and/or observation protocols. These isolated efforts resulted in redundant resources spent and lacking outcome comparability across ERCs. Leaders from three different ERCs led and initiated a collaborative effort to address the above issue by building a suite of common evaluation instruments that all current and future ERCs can use. This leading group consists of education directors and external evaluators from all three partners ERCs and engineering education researchers, who have worked together for two years. The project intends to address the four ERC program clusters: Broadening Participation in Engineering, Centers and Networks, Engineering Education, and Engineering Workforce Development. The instruments developed will pay attention to culture of inclusion, outreach activities, mentoring experience, and sustained interest in engineering. The project will deliver best practices in education program evaluation, which will not only support existing ERCs, but will also serve as immediate tools for brand new ERCs and similar large-scale research centers. Expanding the research beyond TEEC and sharing the developed instruments with NSF as well as other ERCs will also promote and encourage continual cross-ERC collaboration and research. Further, the joint evaluation will increase the evaluation consistency across all ERC education programs. Embedded instrumental feedback loops will lead to continual improvement to ERC education performance and support the growth of an inclusive and innovative engineering workforce. Four major deliveries are planned. First, develop a common quantitative assessment instrument, named Multi-ERC Instrument Inventory (MERCII). Second, develop a set of qualitative instruments to complement MERCII. Third, create a web-based evaluation platform for MERCII. Fourth, update the NSF ERC education program evaluation best practice manual. These deliveries together will become part of and supplemented by an ERC evaluator toolbox. This project strives to significantly impact how ERCs evaluate their educational and outreach programs. Single ERC based studies lack the sample size to truly test the validity of any evaluation instruments or measures. A common suite of instruments across ERCs would provide an opportunity for a large scale assessment study. The online platform will further provide an easy-to-use tool for all ERCs to facilitate evaluation, share data, and reporting impacts.
more »
« less
Exploring the Utility of Social-Network-Derived Collaborative Opportunity Temperature Readings for Informing Design and Research of Large-Group Immersive Learning Environments
Large-group (n > 8) co-located collaboration has not been adequately studied because it demands different conceptual framings than those used to study small-group collaboration, while also posing pragmatic constraints on data collection. Working within these pragmatic constraints, we use video data to devise an indicator of the current possibilities for learner collaboration during large-group co-located interactions. We borrow conceptualizations from proxemics and social network analysis to construct collaborative opportunity networks, allowing us to define the concept of collaborative opportunity temperature (COT) readings: a “snapshot” of the current configuration of the different social subgroup structures within a large group, indicating emergent opportunities for collaboration (via talk or shared action) due to proximity. Using a case study of two groups of people (n = 11, n = 12) who interacted with a multi-user museum exhibit, we outline the processes of deriving COT. We show how to quickly detect differences in subgroup configurations that may result from educational interventions and how COT can triangulate with and complement other forms of data (audio transcripts and activity logs) during lengthier analyses. We also outline how COT readings can be used to supply formative feedback on social engagement to learners and be adapted to other learning environments.
more »
« less
- Award ID(s):
- 1822864
- NSF-PAR ID:
- 10355153
- Date Published:
- Journal Name:
- Journal of learning analytics
- Volume:
- 9
- Issue:
- 1
- ISSN:
- 1929-7750
- Page Range / eLocation ID:
- 53-76
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Lankes, R. David (Ed.)Resilience is often treated as a single-dimension system attribute, or various dimensions of resilience are studied separately without considering multi-dimensionality. The increasing frequency of catastrophic natural or man-made disasters affecting rural areas demands holistic assessments of community vulnerability and assessment. Disproportionate effects of disasters on minorities, low-income, hard-to-reach, and vulnerable populations demand a community-oriented planning approach to address the “resilience divide.” Rural areas have many advantages, but low population density, coupled with dispersed infrastructures and community support networks, make these areas more affected by natural disasters. This paper will catalyze three key learnings from our current work in public librarians’ roles in disaster resiliency: rural communities are composed of diverse sub-communities, each which experiences and responds to traumatic events differently, depending on micro-geographic and demographic drivers. Rural citizens tend to be very self-reliant and are committed to strengthening and sustaining community resiliency with local human capital and resources. Public libraries are central to rural life, providing a range of informational, educational, social, and personal services, especially in remote areas that lack reliable access to community resources during disasters. Public libraries and their librarian leaders are often a “crown jewel” of rural areas’ community infrastructure and this paper will present a community-based design and assessment process for resiliency hubs located in and operated through rural public libraries. The core technical and social science research questions explored in the proposed paper are: 1) Who were the key beneficiaries and what did they need? 2) What was the process of designing a resiliency hub? 3) What did library resiliency hubs provide and how can they be sustained? This resiliency hub study will detail co-production of solutions and involves an inclusive collaboration among researchers, librarians, and community members to address the effects of cascading impacts of natural disasters. The novel co-design process detailed in the paper reflects an in-depth understanding of the complex interactions among libraries, residents, governments, and other agencies by collecting sociotechnical hurricane-related data for Calhoun County, Florida, USA, a region devastated by Hurricane Michael (2018) and hard-hit by Covid-19. We analyzed data from newly developed fusing algorithms and incorporating multiple communities and developed a framework and process to co-design resiliency hubs sited in public libraries. This research leverages a unique opportunity to library-centered policies and technologies to establish a new paradigm for developing disaster resiliency in rural settings. Public libraries serve a diverse population who will directly benefit from practical support tailored to their needs. The project will inform efficient plans to ensure that high-need groups are not isolated in disasters. The knowledge and insight gained from the resiliency hub design process will not only improve our understanding of emergency response operations, but also will contribute to the development of new disaster related policies and plans for public libraries, with a broader application to rural communities in many settings.more » « less
-
Lankes, R.David (Ed.)Resilience is often treated as a single-dimension system attribute, or various dimensions of resilience are studied separately without considering multi-dimensionality. The increasing frequency of catastrophic natural or man-made disasters affecting rural areas demands holistic assessments of community vulnerability and assessment. Disproportionate effects of disasters on minorities, low-income, hard-to-reach, and vulnerable populations demand a community-oriented planning approach to address the “resilience divide.” Rural areas have many advantages, but low population density, coupled with dispersed infrastructures and community support networks, make these areas more affected by natural disasters. This paper will catalyze three key learnings from our current work in public librarians’ roles in disaster resiliency: 1) rural communities are composed of diverse sub-communities, each which experiences and responds to traumatic events differently, depending on micro-geographic and demographic drivers; 2) public libraries are central to rural life, providing a range of informational, educational, social, and personal services, especially in remote areas that lack reliable access to community resources during disasters; and 3) rural citizens tend to be very self-reliant and are committed to strengthening and sustaining community resiliency with local human capital and resources. Public libraries and their librarian leaders are often a “crown jewel” of rural areas’ community infrastructure and this paper will present a community-based design and assessment process for resiliency hubs located in and operated through rural public libraries. The core technical and social science research questions explored in the proposed paper are: 1) Who were the key beneficiaries and what did they need? 2) What was the process of designing a resiliency hub? 3) What did library resiliency hubs provide and how can they be sustained? This resiliency hub study will detail co-production of solutions and involves an inclusive collaboration among researchers, librarians, and community members to address the effects of cascading impacts of natural disasters. The novel co-design process detailed in the paper reflects 1) an in-depth understanding of the complex interactions among libraries, residents, governments, and other agencies by collecting sociotechnical hurricane-related data for Calhoun County, Florida, USA, a region devastated by Hurricane Michael (2018) and hard-hit by Covid-19; 2) analyzed data from newly-developed fusing algorithms and incorporating multiple communities; and 3) co-designed resiliency hubs sited in public libraries. This research leverages a unique opportunity for the co-development of integrated library-centered policies and technologies to establish a new paradigm for developing disaster resiliency in rural settings. Public libraries serve a diverse population who will directly benefit from practical support tailored to their needs. The project will inform efficient plans to ensure that high-need groups are not isolated in disasters. The knowledge and insight gained from disseminating the study’s results will not only improve our understanding of emergency response operations, but also will contribute to the development of new disaster-related policies and plans for public libraries, with a broader application to rural communities in many settings.more » « less
-
Hackathons and similar time-bounded events have become a popular form of collaboration in various domains. They are commonly organized as in-person events during which teams engage in intense collaboration over a short period of time to complete a project that is of interest to them. Most research to date has thus consequently focused on studying how teams collaborate in a co-located setting, pointing towards the advantages of radical co-location. The global pandemic of 2020, however, has led to many hackathons moving online, which challenges our current understanding of how they function. In this paper, we address this gap by presenting findings from a multiple-case study of 10 hackathon teams that participated in 4 hackathon events across two continents. By analyzing the collected data, we found that teams merged synchronous and asynchronous means of communication to maintain a common understanding of work progress as well as to maintain awareness of each other's tasks. Task division was self-assigned based on individual skills or interests, while leaders emerged from different strategies (e.g., participant experience, the responsibility of registering the team in an event). Some of the affordances of in-person hackathons, such as the radical co-location of team members, could be partially reproduced in teams that kept open synchronous communication channels while working (i.e., shared audio territories), in a sort of "radical virtual co-location". However, others, such as interactions with other teams, easy access to mentors, and networking with other participants, decreased. In addition, the technical constraints of the different communication tools and platforms brought technical problems and were overwhelming to participants. Our work contributes to understanding the virtual collaboration of small teams in the context of online hackathons and how technologies and event structures proposed by organizers imply this collaboration.more » « less
-
Researchers, evaluators and designers from an array of academic disciplines and industry sectors are turning to participatory approaches as they seek to understand and address complex social problems. We refer to participatory approaches that collaboratively engage/ partner with stakeholders in knowledge creation/problem solving for action/social change outcomes as collaborative change research, evaluation and design (CCRED). We further frame CCRED practitioners by their desire to move beyond knowledge creation for its own sake to implementation of new knowledge as a tool for social change. In March and May of 2018, we conducted a literature search of multiple discipline-specific databases seeking collaborative, change-oriented scholarly publications. The search was limited to include peerreviewed journal articles, with English language abstracts available, published in the last five years. The search resulted in 526 citations, 236 of which met inclusion criteria. Though the search was limited to English abstracts, all major geographic regions (North America, Europe, Latin America/Caribbean, APAC, Africa and the Middle East) were represented within the results, although many articles did not state a specific region. Of those identified, most studies were located in North America, with the Middle East having only one identified study. We followed a qualitative thematic synthesis process to examine the abstracts of peer-reviewed articles to identify practices that transcend individual disciplines, sectors and contexts to achieve collaborative change. We surveyed the terminology used to describe CCRED, setting, content/topic of study, type of collaboration, and related benefits/outcomes in order to discern the words used to designate collaboration, the frameworks, tools and methods employed, and the presence of action, evaluation or outcomes. Forty-three percent of the reviewed articles fell broadly within the social sciences, followed by 26 percent in education and 25 percent in health/medicine. In terms of participants and/ or collaborators in the articles reviewed, the vast majority of the 236 articles (86%) described participants, that is, those who the research was about or from whom data was collected. In contrast to participants, partners/collaborators (n=32; 14%) were individuals or groups who participated in the design or implementation of the collaborative change effort described. In terms of the goal for collaboration and/or for doing the work, the most frequently used terminology related to some aspect of engagement and empowerment. Common descriptors for the work itself were ‘social change’ (n=74; 31%), ‘action’ (n=33; 14%), ‘collaborative or participatory research/practice’ (n=13; 6%), ‘transformation’ (n=13; 6%) and ‘community engagement’ (n=10; 4%). Of the 236 articles that mentioned a specific framework or approach, the three most common were some variation of Participatory Action Research (n=30; 50%), Action Research (n=40; 16.9%) or Community-Based Participatory Research (n=17; 7.2%). Approximately a third of the 236 articles did not mention a specific method or tool in the abstract. The most commonly cited method/tool (n=30; 12.7%) was some variation of an arts-based method followed by interviews (n=18; 7.6%), case study (n=16; 6.7%), or an ethnographic-related method (n=14; 5.9%). While some articles implied action or change, only 14 of the 236 articles (6%) stated a specific action or outcome. Most often, the changes described were: the creation or modification of a model, method, process, framework or protocol (n=9; 4%), quality improvement, policy change and social change (n=8; 3%), or modifications to education/training methods and materials (n=5; 2%). The infrequent use of collaboration as a descriptor of partner engagement, coupled with few reported findings of measurable change, raises questions about the nature of CCRED. It appears that conducting CCRED is as complex an undertaking as the problems that the work is attempting to address.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.18608/jla.2022.7419
