An official website of the United States government
Novice programming students frequently engage in help-seeking to find information and learn about programming concepts. Among the available resources, generative AI (GenAI) chatbots appear resourceful, widely accessible, and less intimidating than human tutors. Programming instructors are actively integrating these tools into classrooms. However, our understanding of how novice programming students trust GenAI chatbots-and the factors influencing their usage-remains limited. To address this gap, we investigated the learning resource selection process of 20 novice programming students tasked with studying a programming topic. We split our participants into two groups: one using ChatGPT (n=10) and the other using a human tutor via Discord (n=10). We found that participants held strong positive perceptions of ChatGPT's speed and convenience but were wary of its inconsistent accuracy, making them reluctant to rely on it for learning entirely new topics. Accordingly, they generally preferred more trustworthy resources for learning (e.g., instructors, tutors), preferring ChatGPT for low-stakes situations or more introductory and common topics. We conclude by offering guidance to instructors on integrating LLM-based chatbots into their curricula-emphasizing verification and situational use-and to developers on designing chatbots that better address novices' trust and reliability concerns.
more »
« less
This content will become publicly available on July 7, 2026
Outcomes, Perceptions, and Interaction Strategies of Novice Programmers Studying with ChatGPT
Large Language Model (LLM) conversational agents are increasingly used in programming education, yet we still lack insight into how novices engage with them for conceptual learning compared with human tutoring. This mixed‑methods study compared learning outcomes and interaction strategies of novices using ChatGPT or human tutors. A controlled lab study with 20 students enrolled in introductory programming courses revealed that students employ markedly different interaction strategies with AI versus human tutors: ChatGPT users relied on brief, zero‑shot prompts and received lengthy, context‑rich responses but showed minimal prompt refinement, while those working with human tutors provided more contextual information and received targeted explanations. Although students distrusted ChatGPT’s accuracy, they paradoxically preferred it for basic conceptual questions due to reduced social anxiety. We offer empirically grounded recommendations for developing AI literacy in computer science education and designing learning‑focused conversational agents that balance trust‑building with maintaining the social safety that facilitates uninhibited inquiry.
more »
« less
- PAR ID:
- 10649531
- Publisher / Repository:
- ACM
- Date Published:
- Page Range / eLocation ID:
- 1 to 15
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This study highlights how middle schoolers discuss the benefits and drawbacks of AI-driven conversational agents in learning. Using thematic analysis of focus groups, we identified five themes in students’ views of AI applications in education. Students recognized the benefits of AI in making learning more engaging and providing personalized, adaptable scaffolding. They emphasized that AI use in education needs to be safe and equitable. Students identified the potential of AI in supporting teachers and noted that AI educational agents fall short when compared to emotionally and intellectually complex humans. Overall, we argue that even without technical expertise, middle schoolers can articulate deep, multifaceted understandings of the possibilities and pitfalls of AI in education. Centering student voices in AI design can also provide learners with much-desired agency over their future learning experiences.more » « less
-
This research-to-practice paper presents a curriculum, “AI Literacy for All,” to promote an interdisciplinary under-standing of AI, its socio-technical implications, and its practical applications for all levels of education. With the rapid evolution of artificial intelligence (AI), there is a need for AI literacy that goes beyond the traditional AI education curriculum. AI literacy has been conceptualized in various ways, including public literacy, competency building for designers, conceptual understanding of AI concepts, and domain-specific upskilling. Most of these conceptualizations were established before the public release of Generative AI (Gen-AI) tools such as ChatGPT. AI education has focused on the principles and applications of AI through a technical lens that emphasizes the mastery of AI principles, the mathematical foundations underlying these technologies, and the programming and mathematical skills necessary to implement AI solutions. The non-technical component of AI literacy has often been limited to social and ethical implications, privacy and security issues, or the experience of interacting with AI. In AI Literacy for all, we emphasize a balanced curriculum that includes technical as well as non-technical learning outcomes to enable a conceptual understanding and critical evaluation of AI technologies in an interdisciplinary socio-technical context. The paper presents four pillars of AI literacy: understanding the scope and technical dimensions of AI, learning how to interact with Gen-AI in an informed and responsible way, the socio-technical issues of ethical and responsible AI, and the social and future implications of AI. While it is important to include all learning outcomes for AI education in a Computer Science major, the learning outcomes can be adjusted for other learning contexts, including, non-CS majors, high school summer camps, the adult workforce, and the public. This paper advocates for a shift in AI literacy education to offer a more interdisciplinary socio-technical approach as a pathway to broaden participation in AI. This approach not only broadens students' perspectives but also prepares them to think critically about integrating AI into their future professional and personal lives.more » « less
-
This paper examines the design and evaluation of Large Language Model (LLM) tutors for Python programming, focusing on personalization that accommodates diverse student backgrounds. It highlights the challenges faced by socioeconomically disadvantaged students in computing courses and proposes LLM tutors as a solution to provide inclusive educational support. The study explores two LLM tutors, Khanmigo and CS50.ai, assessing their ability to offer personalized learning experiences. By employing a focus group methodology at a public minority-serving institution, the research evaluates how these tutors meet varied educational goals and adapt to students’ diverse needs. The findings underscore the importance of advanced techniques to tailor interactions and integrate programming tools based on students' progress. This research contributes to the understanding of educational technologies in computing education and provides insights into the design and implementation of LLM tutors that effectively support equitable student success.more » « less
-
Conversational AIs such as Alexa and ChatGPT are increasingly ubiquitous in young people’s lives, but these young users are often not afforded the opportunity to learn about the inner workings of these technologies. One of the most powerful ways to foster this learning is to empower youth to create AI that is personally and socially meaningful to them. We have built a novel development environment, AMBY–‘‘AI Made By You’’–for youth to create conversational agents. AMBY was iteratively designed with and for youth aged 12–13 through contextual inquiry and usability studies. AMBY is designed to foster AI learning with features that enable users to generate training datasets and visualize conversational flow. We report on results from a two-week summer camp deployment, and contribute design implications for conversational AI authoring tools that empower AI learning for youth.more » « less
