skip to main content


Title: Analysis of click-stream data to predict stem careers from student usage of an intelligent tutoring system
In recent years, we have seen the continuous and rapid increase of job openings in Science, Technology, Engineering and Math (STEM)-related fields. Unfortunately, these positions are not met with an equal number of workers ready to fill them. Efforts are being made to find durable solutions for this phenomena, and they start by encouraging young students to enroll in STEM college majors. However, enrolling in a STEM major requires specific skills in math and science that are learned in schools. Hopefully, institutions are adopting educational software that collects data from the students' usage. This gathered data will serve to conduct analysis and detect students' behaviors, predict their performances and their eventual college enrollment. As we will outline in this paper, we used data collected from the students' usage of an Intelligent Tutoring System to predict whether they would pursue a career in STEM-related fields. We conducted different types of analysis called "problem-based approach" and "skill-based approach". The problem- based approach focused on evaluating students' actions based on the problems they solved. Likewise, in the skill-based approach we evaluated their usage based on the skills they had practiced. Furthermore, we investigated whether comparing students' features with those of their peer schoolmates can improve the prediction models in both the skill-based and the problem-based approaches. The experimental re- sults showed that the skill-based approach with school aggregation achieved the best results with regard to a combination of two metrics which are the Area Under the Receiver Operating Characteristic Curve (AUC) and the Root Mean Squared Error (RMSE).  more » « less
Award ID(s):
1636847 1661987
NSF-PAR ID:
10190351
Author(s) / Creator(s):
;
Date Published:
Journal Name:
Journal of educational data mining
Volume:
12
Issue:
2
ISSN:
2157-2100
Page Range / eLocation ID:
1-18
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. To remain competitive in the global economy, the United States needs skilled technical workers in occupations requiring a high level of domain-specific technical knowledge to meet the country’s anticipated shortage of 5 million technically-credentialed workers. The changing demographics of the country are of increasing importance to addressing this workforce challenge. According to federal data, half the students earning a certificate in 2016-17 received credentials from community colleges where the percent enrollment of Latinx (a gender-neutral term referencing Latin American cultural or racial identity) students (56%) exceeds that of other post-secondary sectors. If this enrollment rate persists, then by 2050 over 25% of all students enrolled in higher education will be Latinx. Hispanic Serving Institutions (HSIs) are essential points of access as they enroll 64% of all Latinx college students, and nearly 50% of all HSIs are 2-year institutions. Census estimates predict Latinxs are the fastest-growing segment reaching 30% of the U.S. population while becoming the youngest group comprising 33.5% of those under 18 years by 2060. The demand for skilled workers in STEM fields will be met when workers reflect the diversity of the population, therefore more students—of all ages and backgrounds—must be brought into community colleges and supported through graduation: a central focus of community colleges everywhere. While Latinx students of color are as likely as white students to major in STEM, their completion numbers drop dramatically: Latinx students often have distinct needs that evolved from a history of discrimination in the educational system. HSI ATE Hub is a three-year collaborative research project funded by the National Science Foundation Advanced Technological Education Program (NSF ATE) being implemented by Florence Darlington Technical College and Science Foundation Arizona Center for STEM at Arizona State University to address the imperative that 2-year Hispanic Serving Institutions (HSIs) develop and improve engineering technology and related technician education programs in a way that is culturally inclusive. Interventions focus on strengthening grant-writing skills among CC HSIs to fund advancements in technician education and connecting 2-year HSIs with resources for faculty development and program improvement. A mixed methods approach will explore the following research questions: 1) What are the unique barriers and challenges for 2-year HSIs related to STEM program development and grant-writing endeavors? 2) How do we build capacity at 2-year HSIs to address these barriers and challenges? 3) How do mentoring efforts/styles need to differ? 4) How do existing ATE resources need to be augmented to better serve 2-year HSIs? 5) How do proposal submission and success rates compare for 2-year HSIs that have gone through the KS STEM planning process but not M-C, through the M-C cohort mentoring process but not KS, and through both interventions? The project will identify HSI-relevant resources, augment existing ATE resources, and create new ones to support 2-year HSI faculty as potential ATE grantees. To address the distinct needs of Latinx students in STEM, resources representing best practices and frameworks for cultural inclusivity, as well as faculty development will be included. Throughout, the community-based tradition of the ATE Program is being fostered with particular emphasis on forming, nurturing, and serving participating 2-year HSIs. This paper will discuss the need, baseline data, and early results for the three-year program, setting the stage for a series of annual papers that report new findings. 
    more » « less
  2. To remain competitive in the global economy, the United States needs skilled technical workers in occupations requiring a high level of domain-specific technical knowledge to meet the country’s anticipated shortage of 5 million technically-credentialed workers. The changing demographics of the country are of increasing importance to addressing this workforce challenge. According to federal data, half the students earning a certificate in 2016-17 received credentials from community colleges where the percent enrollment of Latinx (a gender-neutral term referencing Latin American cultural or racial identity) students (56%) exceeds that of other post-secondary sectors. If this enrollment rate persists, then by 2050 over 25% of all students enrolled in higher education will be Latinx. Hispanic Serving Institutions (HSIs) are essential points of access as they enroll 64% of all Latinx college students, and nearly 50% of all HSIs are 2-year institutions. Census estimates predict Latinxs are the fastest-growing segment reaching 30% of the U.S. population while becoming the youngest group comprising 33.5% of those under 18 years by 2060. The demand for skilled workers in STEM fields will be met when workers reflect the diversity of the population, therefore more students—of all ages and backgrounds—must be brought into community colleges and supported through graduation: a central focus of community colleges everywhere. While Latinx students of color are as likely as white students to major in STEM, their completion numbers drop dramatically: Latinx students often have distinct needs that evolved from a history of discrimination in the educational system. HSI ATE Hub is a three-year collaborative research project funded by the National Science Foundation Advanced Technological Education Program (NSF ATE) being implemented by Florence Darlington Technical College and Science Foundation Arizona Center for STEM at Arizona State University to address the imperative that 2-year Hispanic Serving Institutions (HSIs) develop and improve engineering technology and related technician education programs in a way that is culturally inclusive. Interventions focus on strengthening grant-writing skills among CC HSIs to fund advancements in technician education and connecting 2-year HSIs with resources for faculty development and program improvement. A mixed methods approach will explore the following research questions: 1) What are the unique barriers and challenges for 2-year HSIs related to STEM program development and grant-writing endeavors? 2) How do we build capacity at 2-year HSIs to address these barriers and challenges? 3) How do mentoring efforts/styles need to differ? 4) How do existing ATE resources need to be augmented to better serve 2-year HSIs? 5) How do proposal submission and success rates compare for 2-year HSIs that have gone through the KS STEM planning process but not M-C, through the M-C cohort mentoring process but not KS, and through both interventions? The project will identify HSI-relevant resources, augment existing ATE resources, and create new ones to support 2-year HSI faculty as potential ATE grantees. To address the distinct needs of Latinx students in STEM, resources representing best practices and frameworks for cultural inclusivity, as well as faculty development will be included. Throughout, the community-based tradition of the ATE Program is being fostered with particular emphasis on forming, nurturing, and serving participating 2-year HSIs. This paper will discuss the need, baseline data, and early results for the three-year program, setting the stage for a series of annual papers that report new findings. 
    more » « less
  3. There is little research or understanding of curricular differences between two- and four-year programs, career development of engineering technology (ET) students, and professional preparation for ET early career professionals [1]. Yet, ET credentials (including certificates, two-, and four-year degrees) represent over half of all engineering credentials awarded in the U.S [2]. ET professionals are important hands-on members of engineering teams who have specialized knowledge of components and engineering systems. This research study focuses on how career orientations affect engineering formation of ET students educated at two-year colleges. The theoretical framework guiding this study is Social Cognitive Career Theory (SCCT). SCCT is a theory which situates attitudes, interests, and experiences and links self-efficacy beliefs, outcome expectations, and personal goals to educational and career decisions and outcomes [3]. Student knowledge of attitudes toward and motivation to pursue STEM and engineering education can impact academic performance and indicate future career interest and participation in the STEM workforce [4]. This knowledge may be measured through career orientations or career anchors. A career anchor is a combination of self-concept characteristics which includes talents, skills, abilities, motives, needs, attitudes, and values. Career anchors can develop over time and aid in shaping personal and career identity [6]. The purpose of this quantitative research study is to identify dimensions of career orientations and anchors at various educational stages to map to ET career pathways. The research question this study aims to answer is: For students educated in two-year college ET programs, how do the different dimensions of career orientations, at various phases of professional preparation, impact experiences and development of professional profiles and pathways? The participants (n=308) in this study represent three different groups: (1) students in engineering technology related programs from a medium rural-serving technical college (n=136), (2) students in engineering technology related programs from a large urban-serving technical college (n=52), and (3) engineering students at a medium Research 1 university who have transferred from a two-year college (n=120). All participants completed Schein’s Career Anchor Inventory [5]. This instrument contains 40 six-point Likert-scale items with eight subscales which correlate to the eight different career anchors. Additional demographic questions were also included. The data analysis includes graphical displays for data visualization and exploration, descriptive statistics for summarizing trends in the sample data, and then inferential statistics for determining statistical significance. This analysis examines career anchor results across groups by institution, major, demographics, types of educational experiences, types of work experiences, and career influences. This cross-group analysis aids in the development of profiles of values, talents, abilities, and motives to support customized career development tailored specifically for ET students. These findings contribute research to a gap in ET and two-year college engineering education research. Practical implications include use of findings to create career pathways mapped to career anchors, integration of career development tools into two-year college curricula and programs, greater support for career counselors, and creation of alternate and more diverse pathways into engineering. Words: 489 References [1] National Academy of Engineering. (2016). Engineering technology education in the United States. Washington, DC: The National Academies Press. [2] The Integrated Postsecondary Education Data System, (IPEDS). (2014). Data on engineering technology degrees. [3] Lent, R.W., & Brown, S.B. (1996). Social cognitive approach to career development: An overivew. Career Development Quarterly, 44, 310-321. [4] Unfried, A., Faber, M., Stanhope, D.S., Wiebe, E. (2015). The development and validation of a measure of student attitudes toward science, technology, engineeirng, and math (S-STEM). Journal of Psychoeducational Assessment, 33(7), 622-639. [5] Schein, E. (1996). Career anchors revisited: Implications for career development in the 21st century. Academy of Management Executive, 10(4), 80-88. [6] Schein, E.H., & Van Maanen, J. (2013). Career Anchors, 4th ed. San Francisco: Wiley. 
    more » « less
  4. Need/Motivation (e.g., goals, gaps in knowledge) The ESTEEM implemented a STEM building capacity project through students’ early access to a sustainable and innovative STEM Stepping Stones, called Micro-Internships (MI). The goal is to reap key benefits of a full-length internship and undergraduate research experiences in an abbreviated format, including access, success, degree completion, transfer, and recruiting and retaining more Latinx and underrepresented students into the STEM workforce. The MIs are designed with the goals to provide opportunities for students at a community college and HSI, with authentic STEM research and applied learning experiences (ALE), support for appropriate STEM pathway/career, preparation and confidence to succeed in STEM and engage in summer long REUs, and with improved outcomes. The MI projects are accessible early to more students and build momentum to better overcome critical obstacles to success. The MIs are shorter, flexibly scheduled throughout the year, easily accessible, and participation in multiple MI is encouraged. ESTEEM also establishes a sustainable and collaborative model, working with partners from BSCS Science Education, for MI’s mentor, training, compliance, and building capacity, with shared values and practices to maximize the improvement of student outcomes. New Knowledge (e.g., hypothesis, research questions) Research indicates that REU/internship experiences can be particularly powerful for students from Latinx and underrepresented groups in STEM. However, those experiences are difficult to access for many HSI-community college students (85% of our students hold off-campus jobs), and lack of confidence is a barrier for a majority of our students. The gap between those who can and those who cannot is the “internship access gap.” This project is at a central California Community College (CCC) and HSI, the only affordable post-secondary option in a region serving a historically underrepresented population in STEM, including 75% Hispanic, and 87% have not completed college. MI is designed to reduce inequalities inherent in the internship paradigm by providing access to professional and research skills for those underserved students. The MI has been designed to reduce barriers by offering: shorter duration (25 contact hours); flexible timing (one week to once a week over many weeks); open access/large group; and proximal location (on-campus). MI mentors participate in week-long summer workshops and ongoing monthly community of practice with the goal of co-constructing a shared vision, engaging in conversations about pedagogy and learning, and sustaining the MI program going forward. Approach (e.g., objectives/specific aims, research methodologies, and analysis) Research Question and Methodology: We want to know: How does participation in a micro-internship affect students’ interest and confidence to pursue STEM? We used a mixed-methods design triangulating quantitative Likert-style survey data with interpretive coding of open-responses to reveal themes in students’ motivations, attitudes toward STEM, and confidence. Participants: The study sampled students enrolled either part-time or full-time at the community college. Although each MI was classified within STEM, they were open to any interested student in any major. Demographically, participants self-identified as 70% Hispanic/Latinx, 13% Mixed-Race, and 42 female. Instrument: Student surveys were developed from two previously validated instruments that examine the impact of the MI intervention on student interest in STEM careers and pursuing internships/REUs. Also, the pre- and post (every e months to assess longitudinal outcomes) -surveys included relevant open response prompts. The surveys collected students’ demographics; interest, confidence, and motivation in pursuing a career in STEM; perceived obstacles; and past experiences with internships and MIs. 171 students responded to the pre-survey at the time of submission. Outcomes (e.g., preliminary findings, accomplishments to date) Because we just finished year 1, we lack at this time longitudinal data to reveal if student confidence is maintained over time and whether or not students are more likely to (i) enroll in more internships, (ii) transfer to a four-year university, or (iii) shorten the time it takes for degree attainment. For short term outcomes, students significantly Increased their confidence to continue pursuing opportunities to develop within the STEM pipeline, including full-length internships, completing STEM degrees, and applying for jobs in STEM. For example, using a 2-tailed t-test we compared means before and after the MI experience. 15 out of 16 questions that showed improvement in scores were related to student confidence to pursue STEM or perceived enjoyment of a STEM career. Finding from the free-response questions, showed that the majority of students reported enrolling in the MI to gain knowledge and experience. After the MI, 66% of students reported having gained valuable knowledge and experience, and 35% of students spoke about gaining confidence and/or momentum to pursue STEM as a career. Broader Impacts (e.g., the participation of underrepresented minorities in STEM; development of a diverse STEM workforce, enhanced infrastructure for research and education) The ESTEEM project has the potential for a transformational impact on STEM undergraduate education’s access and success for underrepresented and Latinx community college students, as well as for STEM capacity building at Hartnell College, a CCC and HSI, for students, faculty, professionals, and processes that foster research in STEM and education. Through sharing and transfer abilities of the ESTEEM model to similar institutions, the project has the potential to change the way students are served at an early and critical stage of their higher education experience at CCC, where one in every five community college student in the nation attends a CCC, over 67% of CCC students identify themselves with ethnic backgrounds that are not White, and 40 to 50% of University of California and California State University graduates in STEM started at a CCC, thus making it a key leverage point for recruiting and retaining a more diverse STEM workforce. 
    more » « less
  5. Graduate training often takes a monodisciplinary approach that is not informed by best practices, ignores the needs and preferences of students, and overlooks the increasingly interdisciplinary and international nature of research. This is unfortunate, particularly since graduate education that is fully integrated with interdisciplinary research can help students become part of a trained and diverse workforce equipped to meet society’s many challenges. Against this backdrop, a National Science Foundation Research Traineeship (NRT) program is being established at the University of Kentucky leveraging the most effective instruments for the training of STEM professionals, such as network-based graduate student mentoring and career preparation encompassing both technical and professional skillsets. Briefly, the training graduate students will receive – in a way that is fully integrated with the research they perform – includes: 1) tools such as individual development plans and developmental network maps; 2) a multi-departmental and interdisciplinary course on research-related content; 3) a seminar course on transferrable skills (ethics, research, communication, teaching, mentoring, entrepreneurship, teamwork, management, leadership, outreach, etc.); 4) a certificate to be awarded once students complete the two courses above and garner additional credits from an interdisciplinary curriculum of research-related courses; 5) summer internships at other departments and at external institutions (other universities, industry, national laboratories) nationwide or abroad; 6) an annual research-related symposium including all elements of a scientific conference; 7) internal collaborative research grants for participants to fund and pursue their own ideas; 8) fields trips to facilities related to the research; and 9) coaching on job hunting as well as résumé, motivation letter and interview preparation. Since a workforce equipped to meet society’s challenges must be both well trained and diverse, multiple initiatives will ensure that this NRT will broaden participation in STEM. Recruitment-wise, close collaboration with a number of entities will provide this NRT with a broad recruitment pool of talented and diverse students. Moreover, collaboration with these entities will provide trainees with ample opportunities to acquire, practice and refine their professional skills, as trainees present their results and recruit in conferences, meetings and outreach events organized by these entities, become members and/or join their leadership, and expand their professional and mentoring network in the process. In addition, minority trainees will be surveyed periodically to probe their feelings of well-being, preparation, acceptance, belonging and distress, as well as their perception of how well structured their departments and programs are. According to recent literature, these factors determine whether or not they perform (i.e., publish) at rates comparable to their male majority peers. Saliently, the evaluation of the educational model employed will afford a comprehensive understanding not only of the academy components that were more utilized and impactful, but will reveal the individual mentoring and skill-building facets of the program driving its successful implementation. The evaluation plan includes outcomes, performance measures, an evaluation timetable, benchmarks and a description of how formative evaluation will improve practice, the evaluation process also extending to research activities. 
    more » « less