skip to main content


Title: Work-in-Progress: A Review of the Type, Breadth, and Limitations of Publicly Available Educational Technology Products in 2022
One of the major changes in the higher education ecosystem over the last decade has been a rise in the availability of education-based software products, including education-based web-pages and web-services. Globally the investment in education-based startups in 2017 was $9.5B which surged to $18.7B in 2019 [1]. The COVID-19 pandemic further fueled record investment in this sector, with the US seeing $2.2B invested in 130 startups in 2020, up from $1.7B in 2019 and $1.4B in 2018 (see [2] and [3]). Early indicators show that 2021 will again see further increases [4]. While the majority (92%) of these investments are aimed at consumer and corporate sectors, there is potential for the innovations developed to diffuse into both the P-12 and higher education spaces [5]. What is evident from the investment numbers is that an integration of learning technologies specifically into higher education is progressing at a relatively slower pace [5]. It is the goal of this work-in-progress to identify some of the reasons for this slower progress. Our hypothesis is that, while some of these reasons may be obvious, there are also more subtle and/or counterintuitive reasons for the reduced interest in higher education. The motivation and need for the proposed study grew out of an ongoing NSF RED project where we endeavor to fuse the concept of convergence, loosely defined as “deep integration,” into our undergraduate engineering curriculum. Increasingly software and data systems at colleges and universities, and the affordances they do and do not offer, are integral to university structures. If the respective software systems do not support certain activities and functions then the programs are simply not useful to the faculty [6]. Additionally, any subset of systems needs to seamlessly integrate to form a coherent and usable learning support system that faculty, students, and staff can use without issue and/or barrier. The goal of the proposed activity within our grant is, thus, to build structures to collect, analyze, and display data in support of developing skills in addressing convergent problems.  more » « less
Award ID(s):
2022271
NSF-PAR ID:
10356583
Author(s) / Creator(s):
; ; ; ; ; ; ;
Date Published:
Journal Name:
American Society for Engineering Education Annual Conference and Exhibition
Page Range / eLocation ID:
#37421
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. There have been numerous demands for enhancements in the way undergraduate learning occurs today, especially at a time when the value of higher education continues to be called into question (The Boyer 2030 Commission, 2022). One type of demand has been for the increased integration of subjects/disciplines around relevant issues/topics—with a more recent trend of seeking transdisciplinary learning experiences for students (Sheets, 2016; American Association for the Advancement of Science, 2019). Transdisciplinary learning can be viewed as the holistic way of working equally across disciplines to transcend their own disciplinary boundaries to form new conceptual understandings as well as develop new ways in which to address complex topics or challenges (Ertas, Maxwell, Rainey, & Tanik, 2003; Park & Son, 2010). This transdisciplinary approach can be important as humanity’s problems are not typically discipline specific and require the convergence of competencies to lead to innovative thinking across fields of study. However, higher education continues to be siloed which makes the authentic teaching of converging topics, such as innovation, human-technology interactions, climate concerns, or harnessing the data revolution, organizationally difficult (Birx, 2019; Serdyukov, 2017). For example, working across a university’s academic units to collaboratively teach, or co-teach, around topics of convergence are likely to be rejected by the university systems that have been built upon longstanding traditions. While disciplinary expertise is necessary and one of higher education’s strengths, the structures and academic rigidity that come along with the disciplinary silos can prevent modifications/improvements to the roles of academic units/disciplines that could better prepare students for the future of both work and learning. The balancing of disciplinary structure with transdisciplinary approaches to solving problems and learning is a challenge that must be persistently addressed. These institutional challenges will only continue to limit universities seeking toward scaling transdisciplinary programs and experimenting with novel ways to enhance the value of higher education for students and society. This then restricts innovations to teaching and also hinders the sharing of important practices across disciplines. To address these concerns, a National Science Foundation Improving Undergraduate STEM Education project team, which is the topic of this paper, has set the goal of developing/implementing/testing an authentically transdisciplinary, and scalable educational model in an effort to help guide the transformation of traditional undergraduate learning to span academics silos. This educational model, referred to as the Mission, Meaning, Making (M3) program, is specifically focused on teaching the crosscutting practices of innovation by a) implementing co-teaching and co-learning from faculty and students across different academic units/colleges as well as b) offering learning experiences spanning multiple semesters that immerse students in a community that can nourish both their learning and innovative ideas. As a collaborative initiative, the M3 program is designed to synergize key strengths of an institution’s engineering/technology, liberal arts, and business colleges/units to create a transformative undergraduate experience focused on the pursuit of innovation—one that reaches the broader campus community, regardless of students’ backgrounds or majors. Throughout the development of this model, research was conducted to help identify institutional barriers toward creating such a cross-college program at a research-intensive public university along with uncovering ways in which to address these barriers. While data can show how students value and enjoy transdisciplinary experiences, universities are not likely to be structured in a way to support these educational initiatives and they will face challenges throughout their lifespan. These challenges can result from administration turnover whereas mutual agreements across colleges may then vanish, continued disputes over academic territory, and challenges over resource allotments. Essentially, there may be little to no incentives for academic departments to engage in transdisciplinary programming within the existing structures of higher education. However, some insights and practices have emerged from this research project that can be useful in moving toward transdisciplinary learning around topics of convergence. Accordingly, the paper will highlight features of an educational model that spans disciplines along with the workarounds to current institutional barriers. This paper will also provide lessons learned related to 1) the potential pitfalls with educational programming becoming “un-disciplinary” rather than transdisciplinary, 2) ways in which to incentivize departments/faculty to engage in transdisciplinary efforts, and 3) new structures within higher education that can be used to help faculty/students/staff to more easily converge to increase access to learning across academic boundaries. 
    more » « less
  2. null (Ed.)
    In 2016, 10 universities launched a Networked Improvement Community (NIC) aimed at increasing the number of scholars from Alliances for Graduate Education and the Professoriate (AGEP) populations entering science, technology, engineering, and mathematics (STEM) faculty careers. NICs bring together stakeholders focused on a common goal to accelerate innovation through structured, ongoing intervention development, implementation, and refinement. We theorized a NIC organizational structure would aid understandings of a complex problem in different contexts and accelerate opportunities to develop and improve interventions to address the problem. A distinctive feature of this NIC is its diverse institutional composition of public and private, predominantly white institutions, a historically Black university, a Hispanic-serving institution, and land grant institutions located across eight states and Washington, DC, United States. NIC members hold different positions within their institutions and have access to varied levers of change. Among the many lessons learned through this community case study, analyzing and addressing failed strategies is as equally important to a healthy NIC as is sharing learning from successful interventions. We initially relied on pre-existing relationships and assumptions about how we would work together, rather than making explicit how the NIC would develop, establish norms, understand common processes, and manage changing relationships. We had varied understandings of the depth of campus differences, sometimes resulting in frustrations about the disparate progress on goals. NIC structures require significant engagement with the group, often more intensive than traditional multi-institution organizational structures. They require time to develop and ongoing maintenance in order to advance the work. We continue to reevaluate our model for leadership, climate, diversity, conflict resolution, engagement, decision-making, roles, and data, leading to increased investment in the success of all NIC institutions. Our NIC has evolved from the traditional NIC model to become the Center for the Integration of Research, Teaching and Learning (CIRTL) AGEP NIC model with five key characteristics: (1) A well-specified aim, (2) An understanding of systems, including a variety of contexts and different organizations, (3) A culture and practice of shared leadership and inclusivity, (4) The use of data reflecting different institutional contexts, and (5) The ability to accelerate infrastructure and interventions. We conclude with recommendations for those considering developing a NIC to promote diversity, equity, and inclusion efforts. 
    more » « less
  3. This research paper is a study of the support needs of nontraditional students in engineering (NTSE). Nontraditional students in engineering are one segment of the student body that has traditionally not been a part of the conversation in engineering education– those students who do not go through a typical four-year college degree largely at a residential campus. It is only by better understanding the range of issues that NTSE face that we will be able to design interventions and support systems that can assist them. Recent work in engineering education particularly argues that co-curricular support is a critical factor in student success as it effects curricular progress but there has been no work looking specifically at co-curricular support for NTSE and their retention and persistence. The population of NTSE is increasing across campuses as more students take on jobs to support their education and as those in the workforce return to complete their education. It is imperative that higher educational systems understand how to serve the needs of these students better. Although there are a range of ways in which nontraditional students (NTS) are defined, the NCES has proposed a comprehensive definition that includes enrollment criteria, financial and family status, and high school graduation status. Overall, the seven characteristics specifically associated with NTS are: (1) Delayed enrollment by a year or more after high school, (2) attended part-time, (3) having dependents, (4) being a single parent, (5) working full time while enrolled, (6) being financially independent from parents, and (7) did not receive a standard high school diploma. We ground our research in the Model of Co-Curricular Support (MCCS) which suggests it is the role of the institution to provide the necessary support for integration. If students are aware and have access to resources, which lead to their success, then they will integrate into the university environment at higher rates than those students who are not aware and have access to those resources. This research study focuses on answering one research question: How do NTSE engage with co-curricular supports as they progress through their degree programs? To answer this question, we recruited 11 NTSE with a range of nontraditional characteristics to complete prompted reflective journaling assignments five times throughout the Fall 2021 semester. Qualitative results showcase the nuanced lives of NTSE as they pursue their engineering degrees. In particular, results indicate students interact with faculty, classmates, and friends/peers the most, and only interact with advising when required. Students rarely reach out to larger student support for help or are involved with campus or other events happening. Classmate and friend/peer interactions are the most positive, while interactions with faculty had the largest negative outcomes. 
    more » « less
  4. A 2019 report from the National Academies on Minority Serving Institutions (MSIs) concluded that MSIs need to change their culture to successfully serve students with marginalized racial and/or ethnic identities. The report recommends institutional responsiveness to meet students “where they are,” metaphorically, creating supportive campus environments and providing tailored academic and social support structures. In recent years, the faculty, staff, and administrators at California State University, Los Angeles have made significant efforts to enhance student success through multiple initiatives including a summer bridge program, first-year in engineering program, etc. However, it has become clear that more profound changes are needed to create a culture that meets students “where they are.” In 2020, we were awarded NSF support for Eco-STEM, an initiative designed to change a system that demands "college-ready" students into one that is "student-ready." Aimed at shifting the deficit mindset prevailing in engineering education, the Eco-STEM project embraces an asset-based ecosystem model that thinks of education as cultivation, and ideas as seeds we are planting, rather than a system of standards and quality checks. This significant paradigm and culture transformation is accomplished through: 1) The Eco-STEM Faculty Fellows’ Community of Practice (CoP), which employs critically reflective dialogue[ ][ ] to enhance the learning environment using asset-based learner-centered instructional approaches; 2) A Leadership CoP with department chairs and program directors that guides cultural change at the department/program level; 3) A Facilitators’ CoP that prepares facilitators to lead, sustain, update, and expand the Faculty and Leadership CoPs; 4) Reform of the teaching evaluation system to sustain the cultural changes. This paper presents the progress and preliminary findings of the Eco-STEM project. During the first project year, the project team formulated the curriculum for the Faculty CoP with a focus on inclusive pedagogy, community cultural wealth, and community building, developed a classroom peer observation tool to provide formative data for teaching reflection, and designed research inquiry tools. The latter investigates the following research questions: 1) To what extent do the Eco-STEM CoPs effectively shift the mental models of participants from a factory-like model to an ecosystem model of education? 2) To what extent does this shift support an emphasis on the assets of our students, faculty, and staff members and, in turn, allow for enhanced motivation, excellence and success? 3) To what extent do new faculty assessment tools designed to provide feedback that reflects ecosystem-centric principles and values allow for individuals within the system to thrive? In Fall 2021, the first cohort of Eco-STEM Faculty Fellows were recruited, and rich conversations and in-depth reflections in our CoP meetings indicated Fellows’ positive responses to both the CoP curriculum and facilitation practices. This paper offers a work-in-progress introduction to the Eco-STEM project, including the Faculty CoP, the classroom peer observation tool, and the proposed research instruments. We hope this work will cultivate broader conversations within the engineering education research community about cultural change in engineering education and methods towards its implementation. 
    more » « less
  5. Obeid, Iyad ; Selesnick, Ivan ; Picone, Joseph (Ed.)
    The Temple University Hospital Seizure Detection Corpus (TUSZ) [1] has been in distribution since April 2017. It is a subset of the TUH EEG Corpus (TUEG) [2] and the most frequently requested corpus from our 3,000+ subscribers. It was recently featured as the challenge task in the Neureka 2020 Epilepsy Challenge [3]. A summary of the development of the corpus is shown below in Table 1. The TUSZ Corpus is a fully annotated corpus, which means every seizure event that occurs within its files has been annotated. The data is selected from TUEG using a screening process that identifies files most likely to contain seizures [1]. Approximately 7% of the TUEG data contains a seizure event, so it is important we triage TUEG for high yield data. One hour of EEG data requires approximately one hour of human labor to complete annotation using the pipeline described below, so it is important from a financial standpoint that we accurately triage data. A summary of the labels being used to annotate the data is shown in Table 2. Certain standards are put into place to optimize the annotation process while not sacrificing consistency. Due to the nature of EEG recordings, some records start off with a segment of calibration. This portion of the EEG is instantly recognizable and transitions from what resembles lead artifact to a flat line on all the channels. For the sake of seizure annotation, the calibration is ignored, and no time is wasted on it. During the identification of seizure events, a hard “3 second rule” is used to determine whether two events should be combined into a single larger event. This greatly reduces the time that it takes to annotate a file with multiple events occurring in succession. In addition to the required minimum 3 second gap between seizures, part of our standard dictates that no seizure less than 3 seconds be annotated. Although there is no universally accepted definition for how long a seizure must be, we find that it is difficult to discern with confidence between burst suppression or other morphologically similar impressions when the event is only a couple seconds long. This is due to several reasons, the most notable being the lack of evolution which is oftentimes crucial for the determination of a seizure. After the EEG files have been triaged, a team of annotators at NEDC is provided with the files to begin data annotation. An example of an annotation is shown in Figure 1. A summary of the workflow for our annotation process is shown in Figure 2. Several passes are performed over the data to ensure the annotations are accurate. Each file undergoes three passes to ensure that no seizures were missed or misidentified. The first pass of TUSZ involves identifying which files contain seizures and annotating them using our annotation tool. The time it takes to fully annotate a file can vary drastically depending on the specific characteristics of each file; however, on average a file containing multiple seizures takes 7 minutes to fully annotate. This includes the time that it takes to read the patient report as well as traverse through the entire file. Once an event has been identified, the start and stop time for the seizure is stored in our annotation tool. This is done on a channel by channel basis resulting in an accurate representation of the seizure spreading across different parts of the brain. Files that do not contain any seizures take approximately 3 minutes to complete. Even though there is no annotation being made, the file is still carefully examined to make sure that nothing was overlooked. In addition to solely scrolling through a file from start to finish, a file is often examined through different lenses. Depending on the situation, low pass filters are used, as well as increasing the amplitude of certain channels. These techniques are never used in isolation and are meant to further increase our confidence that nothing was missed. Once each file in a given set has been looked at once, the annotators start the review process. The reviewer checks a file and comments any changes that they recommend. This takes about 3 minutes per seizure containing file, which is significantly less time than the first pass. After each file has been commented on, the third pass commences. This step takes about 5 minutes per seizure file and requires the reviewer to accept or reject the changes that the second reviewer suggested. Since tangible changes are made to the annotation using the annotation tool, this step takes a bit longer than the previous one. Assuming 18% of the files contain seizures, a set of 1,000 files takes roughly 127 work hours to annotate. Before an annotator contributes to the data interpretation pipeline, they are trained for several weeks on previous datasets. A new annotator is able to be trained using data that resembles what they would see under normal circumstances. An additional benefit of using released data to train is that it serves as a means of constantly checking our work. If a trainee stumbles across an event that was not previously annotated, it is promptly added, and the data release is updated. It takes about three months to train an annotator to a point where their annotations can be trusted. Even though we carefully screen potential annotators during the hiring process, only about 25% of the annotators we hire survive more than one year doing this work. To ensure that the annotators are consistent in their annotations, the team conducts an interrater agreement evaluation periodically to ensure that there is a consensus within the team. The annotation standards are discussed in Ochal et al. [4]. An extended discussion of interrater agreement can be found in Shah et al. [5]. The most recent release of TUSZ, v1.5.2, represents our efforts to review the quality of the annotations for two upcoming challenges we hosted: an internal deep learning challenge at IBM [6] and the Neureka 2020 Epilepsy Challenge [3]. One of the biggest changes that was made to the annotations was the imposition of a stricter standard for determining the start and stop time of a seizure. Although evolution is still included in the annotations, the start times were altered to start when the spike-wave pattern becomes distinct as opposed to merely when the signal starts to shift from background. This cuts down on background that was mislabeled as a seizure. For seizure end times, all post ictal slowing that was included was removed. The recent release of v1.5.2 did not include any additional data files. Two EEG files had been added because, originally, they were corrupted in v1.5.1 but were able to be retrieved and added for the latest release. The progression from v1.5.0 to v1.5.1 and later to v1.5.2, included the re-annotation of all of the EEG files in order to develop a confident dataset regarding seizure identification. Starting with v1.4.0, we have also developed a blind evaluation set that is withheld for use in competitions. The annotation team is currently working on the next release for TUSZ, v1.6.0, which is expected to occur in August 2020. It will include new data from 2016 to mid-2019. This release will contain 2,296 files from 2016 as well as several thousand files representing the remaining data through mid-2019. In addition to files that were obtained with our standard triaging process, a part of this release consists of EEG files that do not have associated patient reports. Since actual seizure events are in short supply, we are mining a large chunk of data for which we have EEG recordings but no reports. Some of this data contains interesting seizure events collected during long-term EEG sessions or data collected from patients with a history of frequent seizures. It is being mined to increase the number of files in the corpus that have at least one seizure event. We expect v1.6.0 to be released before IEEE SPMB 2020. The TUAR Corpus is an open-source database that is currently available for use by any registered member of our consortium. To register and receive access, please follow the instructions provided at this web page: https://www.isip.piconepress.com/projects/tuh_eeg/html/downloads.shtml. The data is located here: https://www.isip.piconepress.com/projects/tuh_eeg/downloads/tuh_eeg_artifact/v2.0.0/. 
    more » « less