Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


  • NSF Public Access
  • Search Results
  • This Paper Was Written with the Help of ChatGPT: Exploring the Consequences of AI-Driven Academic Writing on Scholarly Practices
Title: This Paper Was Written with the Help of ChatGPT: Exploring the Consequences of AI-Driven Academic Writing on Scholarly Practices
This paper was written with the help of ChatGPT. Recent advancements in the development and deployment of large generative language models to power generative AI tools, including OpenAIż˝fs ChatGPT, have led to their broad usage across virtually all fields of study. While the tools have been trained to generate human-like-dialogue in response to questions or prompts, they are similarly used to compose larger, more complex artifacts, including social media posts, essays, and even research articles. Although this abstract has been written entirely by a human without any input, consultation, or revision from a generative language model, it would likely be difficult to discern any difference as a reader. In light of this, there is growing debate and concern regarding using these models to aid the writing process, particularly concerning publication. Aside from some notable risks, including the unintentional generation of false information, citation of non-existing research articles, or plagiarism by generating text that is sampled from another source without proper citation, there are additional questions pertaining to the originality of ideas expressed in a work has been partially-written or revised by a generative language model. We present this paper as both a case study into the usage of generative models to aid in the writing of academic research articles but also as an example of how transparency and open science practices may help in addressing several issues that have been raised in other contexts and communities. While this paper neither attempts to promote nor contest the use of these language models in any writing task, it is the goal of this work to provide insight and potential guidance into the ethical and effective usage of these models within this domain.  more » « less
Award ID(s):
2331379
PAR ID:
10531980
Author(s) / Creator(s):
; ;
Editor(s):
Benjamin, Paaßen; Carrie, Demmans Epp
Publisher / Repository:
International Educational Data Mining Society
Date Published:
Format(s):
Medium: X
Right(s):
Creative Commons Attribution 4.0 International
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; (, Applied Sciences)
    Recently, there have been significant advances and wide-scale use of generative AI in natural language generation. Models such as OpenAI’s GPT3 and Meta’s LLaMA are widely used in chatbots, to summarize documents, and to generate creative content. These advances raise concerns about abuses of these models, especially in social media settings, such as large-scale generation of disinformation, manipulation campaigns that use AI-generated content, and personalized scams. We used stylometry (the analysis of style in natural language text) to analyze the style of AI-generated text. Specifically, we applied an existing authorship verification (AV) model that can predict if two documents are written by the same author on texts generated by GPT2, GPT3, ChatGPT and LLaMA. Our AV model was trained only on human-written text and was effectively used in social media settings to analyze cases of abuse. We generated texts by providing the language models with fanfiction snippets and prompting them to complete the rest of it in the same writing style as the original snippet. We then applied the AV model across the texts generated by the language models and the human written texts to analyze the similarity of the writing styles between these texts. We found that texts generated with GPT2 had the highest similarity to the human texts. Texts generated by GPT3 and ChatGPT were very different from the human snippet, and were similar to each other. LLaMA-generated texts had some similarity to the original snippet but also has similarities with other LLaMA-generated texts and texts from other models. We then conducted a feature analysis to identify the features that drive these similarity scores. This analysis helped us answer questions like which features distinguish the language style of language models and humans, which features are different across different models, and how these linguistic features change over different language model versions. The dataset and the source code used in this analysis have been made public to allow for further analysis of new language models. 
    more » « less
  2. ; ; ; ; ; (, Journal of Chemical Education)
    Generative artificial intelligence (AI) is and will continue to be an impactful resource in chemical education. Contrasting numerous positive applications are the concerns about its use for cheating. Regarding the latter, it has been demonstrated that generative AI usage can be detected, even on multiple-choice chemistry exams. But this raises the questions “Why is there a performance difference?” and “How does ChatGPT ‘think’ differently than students?” Here, binomial regression analysis was performed on student and ChatGPT question (i.e., item) success rates on multiple-choice general chemistry II exams with respect to item variables including: format/components (e.g., containing “not”, fill in the blank, capitalization, etc.), topical content (i.e., book sections/ chapters), knowledge type, cognitive process, etc. There were some similar trends in performance; for example, every doubling of the word count resulted in a 10−20% decrease in the odds of both ChatGPT and students answering the question correctly. There were also contrasting outcomes, like with questions containing “not,” where ChatGPT was much more likely to answer correctly as compared with students. On the other hand, students performed better on questions requiring interpretation and classification subcategories of the “understanding” knowledge type. Given the rapid evolution and diversity of generative AI models, the specific outcomes of this analysis will quickly become outdated, but this report demonstrates item variable categorization and regression analysis as an effective tool for highlighting the performance differences between generative AI and students. Furthermore, these insights may help guide item design where the goal is to mitigate the impacts of cheating via generative AI usage or in designing online surveys where obtaining meaningful results requires one to differentiate human and generative AI responses. 
    more » « less
  3. ; ; (, Journal of China Computer-Assisted Language Learning)
    Abstract The rapid proliferation of ChatGPT has incited debates regarding its impact on human writing. Amid concerns about declining writing standards, this study investigates the role of ChatGPT in facilitating writing, especially among language learners. Using a case study approach, this study examines the experiences of Kailing, a doctoral student, who integrates ChatGPT throughout their writing process. The study employs activity theory as a lens for understanding writing with generative AI tools and data analyzed includes semi-structured interviews, writing samples, and GPT logs. Results indicate that Kailing effectively collaborates with ChatGPT across various writing stages while preserving her distinct authorial voice and agency. This underscores the potential of AI tools such as ChatGPT to enhance writing for language learners without overshadowing individual authenticity. This case study offers a critical exploration of how ChatGPT is utilized in the writing process and the preservation of a student’s authentic voice when engaging with the tool. 
    more » « less
  4. ; ; ; ; ; (, ACM)
    Generative AI tools, particularly those utilizing large language models (LLMs), are increasingly used in everyday contexts. While these tools enhance productivity and accessibility, little is known about how Deaf and Hard of Hearing (DHH) individuals engage with them or the challenges they face when using them. This paper presents a mixed-method study exploring how the DHH community uses Text AI tools like ChatGPT to reduce communication barriers and enhance information access. We surveyed 80 DHH participants and conducted interviews with 9 participants. Our findings reveal important benefits, such as eased communication and bridging Deaf and hearing cultures, alongside challenges like lack of American Sign Language (ASL) support and Deaf cultural understanding. We highlight unique usage patterns, propose inclusive design recommendations, and outline future research directions to improve Text AI accessibility for the DHH community. 
    more » « less
  5. ; ; ; (, ACM)
    The advancement of generative AI, involving the utilization of large language models (LLMs) like ChatGPT to assess public opinion and sentiment, has become increasingly prevalent. However, this upsurge in usage raises significant questions about the transparency and interpretability of the predictions made by these LLM Models. Hence, this paper explores the imperative of ensuring transparency in the application of ChatGPT for public sentiment analysis. To tackle these challenges, we propose using a lexicon-based model as a surrogate to approximate both global and local predictions. Through case studies, we demonstrate how transparency mechanisms, bolstered by the lexicon-based model, can be seamlessly integrated into ChatGPT’s deployment for sentiment analysis. Drawing on the results of our study, we further discuss the implications for future research involving the utilization of LLMs in governmental functions, policymaking, and public engagement. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Text Encoded Conversion
  • XML
  • Save / Share this Record
  • Send to Email