Search for: All records

Learning outcomes are clear and concise statements that describe what students should be able to do or know at the end of a particular course. These statements are crucial in instructional planning, curriculum development, and assessment of student progress and learning. Although there is no universal guidance on how to develop learning outcomes, Bloom’s taxonomy is one widely used framework that helps instructors develop outcomes that reflect different levels of thinking, from basic remembering to creative problem-solving. This study investigates the potential of generative AI, specifically GPT-4, in classifying course learning outcomes according to their respective cognitive levels within the revised Bloom’s taxonomy. To assess the effectiveness of GenAI, we conducted a comparative study using a dataset of 1000 annotated learning outcomes. We tested multiple prompt engineering strategies, including zero-shot, few-shot, chain-of-thought, rhetorical situation, and multiple binary questions, leveraging GPT-4. Classification performance was evaluated using accuracy, Cohen’s κ, and F1-score. The results indicate that the prompt incorporating rhetorical context and domain-specific knowledge achieved the highest classification performance, while the multiple binary question approach underperformed even compared to the zero-shot method. Furthermore, we compared the best-performing prompting strategy with a state-of-the-art classification model, BERT. Although the fine-tuned BERT model showed superior performance, prompt-based classification exhibited moderate to substantial agreement with expert annotations. Overall, this article demonstrates the potential of leveraging large language models to advance both theoretical understanding and practical application within the field of education and natural language processing. 

Abstract The recent development and use of generative AI (GenAI) has signaled a significant shift in research activities such as brainstorming, proposal writing, dissemination, and even reviewing. This has raised questions about how to balance the seemingly productive uses of GenAI with ethical concerns such as authorship and copyright issues, use of biased training data, lack of transparency, and impact on user privacy. To address these concerns, many Higher Education Institutions (HEIs) have released institutional guidance for researchers. To better understand the guidance that is being provided we report findings from a thematic analysis of guidelines from thirty HEIs in the United States that are classified as R1 or “very high research activity.” We found that guidance provided to researchers: (1) asks them to refer to external sources of information such as funding agencies and publishers to keep updated and use institutional resources for training and education; (2) asks them to understand and learn about specific GenAI attributes that shape research such as predictive modeling, knowledge cutoff date, data provenance, and model limitations, and educate themselves about ethical concerns such as authorship, attribution, privacy, and intellectual property issues; and (3) includes instructions on how to acknowledge sources and disclose the use of GenAI, how to communicate effectively about their GenAI use, and alerts researchers to long term implications such as over reliance on GenAI, legal consequences, and risks to their institutions from GenAI use. Overall, guidance places the onus of compliance on individual researchers making them accountable for any lapses, thereby increasing their responsibility. 

The release of ChatGPT in November 2022 prompted a massive uptake of generative artificial intelligence (GenAI) across higher education institutions (HEIs). In response, HEIs focused on regulating its use, particularly among students, before shifting towards advocating for its productive integration within teaching and learning. Since then, many HEIs have increasingly provided policies and guidelines to direct GenAI. This paper presents an analysis of documents produced by 116 US universities classified as as high research activity or R1 institutions providing a comprehensive examination of the advice and guidance offered by institutional stakeholders about GenAI. Through an extensive analysis, we found a majority of universities (N = 73, 63%) encourage the use of GenAI, with many offering detailed guidance for its use in the classroom (N = 48, 41%). Over half the institutions provided sample syllabi (N = 65, 56%) and half (N = 58, 50%) provided sample GenAI curriculum and activities that would help instructors integrate and leverage GenAI in their teaching. Notably, the majority of guidance focused on writing activities focused on writing, whereas references to code and STEM-related activities were infrequent, and often vague, even when mentioned (N = 58, 50%). Based on our findings we caution that guidance for faculty can become burdensome as policies suggest or imply substantial revisions to existing pedagogical practices. 

Since the release of ChatGPT in 2022, Generative AI (GenAI) is increasingly being used in higher education computing classrooms across the United States. While scholars have looked at overall institutional guidance for the use of GenAI and reports have documented the response from schools in the form of broad guidance to instructors, we do not know what policies and practices instructors are actually adopting and how they are being communicated to students through course syllabi. To study instructors' policy guidance, we collected 98 computing course syllabi from 54 R1 institutions in the U.S. and studied the GenAI policies they adopted and the surrounding discourse. Our analysis shows that 1) most instructions related to GenAI use were as part of the academic integrity policy for the course and 2) most syllabi prohibited or restricted GenAI use, often warning students about the broader implications of using GenAI, e.g. lack of veracity, privacy risks, and hindering learning. Beyond this, there was wide variation in how instructors approached GenAI including a focus on how to cite GenAI use, conceptualizing GenAI as an assistant, often in an anthropomorphic manner, and mentioning specific GenAI tools for use. We discuss the implications of our findings and conclude with current best practices for instructors. 

The term “nontraditional students” (NTS) is widely used in higher education research, but its definition varies across studies. Objectives This systematic literature review aims to examine how researchers define NTS in U.S.-based studies and identify potential definitional issues. Methods We conducted a systematic review following PRISMA guidelines, searching EBSCO databases (Education Research Complete, Education Full Text, and ERIC) for peer-reviewed articles published between 2018 and 2022. We analyzed 65 papers that met our inclusion criteria to assess the definitions used for NTS. In this systematic literature review we focus on the definitional issues related to how researchers use the term nontraditional students in US-based studies. We review 65 papers from search results containing 432 papers to understand how researchers define nontraditional students. Of the 65 papers reviewed fully, 33 papers included a specific definition of nontraditional students, 15 included an unspecified definition of nontraditional students, and 17 papers did not include a clear definition at all. Our work suggests that researchers use a clearer definition, such as from the NCES, to define nontraditional students and focus their attention on the seven categories given by NCES. 

Case studies are among the most popular and effective pedagogical techniques in ethics education. In this paper, we present a framework to develop and effectively use one type of case study: role-plays. We argue that role-plays are particularly effective for allowing students to think through complex problems and bridge multi-level issues, a core concern of ethics education. The fictional case implemented in the study presented here focuses on the use of algorithms for making lending decisions. The case narrative and its associated roles highlight and emphasize the interdependent and intertwined individual and societal perspectives. Thirty-six students consented to the research study in the course where the role-plays were implemented. Student responses related to their engagement with the role were analyzed. We found that participants moved between the multi-level perspectives in the case, identified ethical principles at each level, and connected case examples to real-world occurrences. Overall, using role-plays strongly encouraged students to appreciate the complexity of technology. This work is part of a larger project on using role-play case studies, and in our conclusions, we draw implications from our overall findings. 

The omnipresence of software systems across all aspects of society has necessitated that future technology professionals are aware of ethical concerns raised by the design and development of software and are trained to minimize harm by undertaking responsible engineering. This need has become even more urgent with artificial intelligence (AI) driven software deployment. In this paper we present a study of an interactive pedagogical intervention – role-play case studies – designed to teach undergraduate technology students about ethics with a focus on software systems. Drawing on the situated learning perspective from the Learning Sciences, we created case studies, associated stakeholder roles, discussion scripts, and pre and post discussion assignments to guide students’ learning. Open-ended data was collected from thirty-nine students and analyzed qualitatively. Findings from the study show that by taking on different perspectives on a problem, students were able to identify a range of ethical issues and understand the role of the software system process holistically, taking context, complexity, and trade-offs into account. In their discussion and reflections, students deliberated the role of software in society and the role of humans in automation. The curricula, including case studies, are publicly available for implementation. 

The explosion of AI across all facets of society has given rise to the need for AI education across domains and levels. AI literacy has become an important concept in the current technological landscape, emphasizing the need for individuals to acquire the necessary knowledge and skills to engage with AI systems. This systematic review examined 47 articles published between 2019 and 2023, focusing on recent work to capture new insights and initiatives given the burgeoning of the literature on this topic. In the initial stage, we explored the dataset to identify the themes covered by the selected papers and the target population for AI literacy efforts. We identified that the articles broadly contributed to one of the following themes: a) conceptualizing AI literacy, b) prompting AI literacy efforts, and c) developing AI literacy assessment instruments. We also found that a range of populations, from pre-K students to adults in the workforce, were targeted. In the second stage, we conducted a thorough content analysis to synthesize six key constructs of AI literacy: Recognize, Know and Understand, Use and Apply, Evaluate, Create, and Navigate Ethically. We then applied this framework to categorize a range of empirical studies and identify the prevalence of each construct across the studies. We subsequently review assessment instruments developed for AI literacy and discuss them. The findings of this systematic review are relevant for formal education and workforce preparation and advancement, empowering individuals to leverage AI and drive innovation.