More Like this

1. Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study

   https://doi.org/10.2196/46939  

   Nov, Oded; Singh, Nina; Mann, Devin (January 2023, JMIR Medical Education)

   Background Chatbots are being piloted to draft responses to patient questions, but patients’ ability to distinguish between provider and chatbot responses and patients’ trust in chatbots’ functions are not well established. Objective This study aimed to assess the feasibility of using ChatGPT (Chat Generative Pre-trained Transformer) or a similar artificial intelligence–based chatbot for patient-provider communication. Methods A survey study was conducted in January 2023. Ten representative, nonadministrative patient-provider interactions were extracted from the electronic health record. Patients’ questions were entered into ChatGPT with a request for the chatbot to respond using approximately the same word count as the human provider’s response. In the survey, each patient question was followed by a provider- or ChatGPT-generated response. Participants were informed that 5 responses were provider generated and 5 were chatbot generated. Participants were asked—and incentivized financially—to correctly identify the response source. Participants were also asked about their trust in chatbots’ functions in patient-provider communication, using a Likert scale from 1-5. Results A US-representative sample of 430 study participants aged 18 and older were recruited on Prolific, a crowdsourcing platform for academic studies. In all, 426 participants filled out the full survey. After removing participants who spent less than 3 minutes on the survey, 392 respondents remained. Overall, 53.3% (209/392) of respondents analyzed were women, and the average age was 47.1 (range 18-91) years. The correct classification of responses ranged between 49% (192/392) to 85.7% (336/392) for different questions. On average, chatbot responses were identified correctly in 65.5% (1284/1960) of the cases, and human provider responses were identified correctly in 65.1% (1276/1960) of the cases. On average, responses toward patients’ trust in chatbots’ functions were weakly positive (mean Likert score 3.4 out of 5), with lower trust as the health-related complexity of the task in the questions increased. Conclusions ChatGPT responses to patient questions were weakly distinguishable from provider responses. Laypeople appear to trust the use of chatbots to answer lower-risk health questions. It is important to continue studying patient-chatbot interaction as chatbots move from administrative to more clinical roles in health care. 

   more » « less
2. Critical Assessment of Large Language Models’ (ChatGPT) Performance in Data Extraction for Systematic Reviews: Exploratory Study

   https://doi.org/10.2196/68097  

   Mahmoudi, Hesam; Chang, Doris; Lee, Hannah; Ghaffarzadegan, Navid; Jalali, Mohammad S (September 2025, JMIR AI)

   Abstract BackgroundSystematic literature reviews (SLRs) are foundational for synthesizing evidence across diverse fields and are especially important in guiding research and practice in health and biomedical sciences. However, they are labor intensive due to manual data extraction from multiple studies. As large language models (LLMs) gain attention for their potential to automate research tasks and extract basic information, understanding their ability to accurately extract explicit data from academic papers is critical for advancing SLRs. ObjectiveOur study aimed to explore the capability of LLMs to extract both explicitly outlined study characteristics and deeper, more contextual information requiring nuanced evaluations, using ChatGPT (GPT-4). MethodsWe screened the full text of a sample of COVID-19 modeling studies and analyzed three basic measures of study settings (ie, analysis location, modeling approach, and analyzed interventions) and three complex measures of behavioral components in models (ie, mobility, risk perception, and compliance). To extract data on these measures, two researchers independently extracted 60 data elements using manual coding and compared them with the responses from ChatGPT to 420 queries spanning 7 iterations. ResultsChatGPT’s accuracy improved as prompts were refined, showing improvements of 33% and 23% between the initial and final iterations for extracting study settings and behavioral components, respectively. In the initial prompts, 26 (43.3%) of 60 ChatGPT responses were correct. However, in the final iteration, ChatGPT extracted 43 (71.7%) of the 60 data elements, showing better performance in extracting explicitly stated study settings (28/30, 93.3%) than in extracting subjective behavioral components (15/30, 50%). Nonetheless, the varying accuracy across measures highlighted its limitations. ConclusionsOur findings underscore LLMs’ utility in extracting basic as well as explicit data in SLRs by using effective prompts. However, the results reveal significant limitations in handling nuanced, subjective criteria, emphasizing the necessity for human oversight. 

   more » « less
3. Designing for diverse museum visitors’ identity exploration around inventiveness

   https://doi.org/10.3389/feduc.2023.1078001  

   Kaplan, Avi; Garner, Joanna K; Rush, Amani; Smith, Monica (November 2023, Frontiers in Education)

   IntroductionThis paper describes an initial phase of research to inform the design of Change YOUR Game—an exhibition at the Smithsonian National Museum of American History (NMAH) that aims to promote visitors’ inventiveness in STEM and in other life domains. The exhibition’s content focused on the history of inventions and innovations in sports. The research was framed by the Dynamic Systems Model of Role Identity (DSMRI) and by a set of design principles for promoting visitors’ agentic identity exploration: The PRESS Principles. MethodsFive data collection sessions engaged participants in a virtual, simulated visit to an early rendition of the exhibition. Diverse groups of participants were prompted to consider the self-relevance of the exhibition content to their identities and how they have been and can be inventive in their lives. ResultsFour themes captured participants’ museum visitor role identities, varied readiness to engage in identity exploration of their inventiveness, and how these provided affordances and hindrances to participants’ engagement in identity exploration in the context of the simulated visit. The themes served as bases for design recommendations. DiscussionThe study highlights the potential of the DSMRI and PRESS design principles to address conceptual and methodological challenges of research that aims to inform environmental design of a context that is not yet in existence and that will be relatively fixed. The findings suggest recommendations for design as well as theoretical insights about museum visitor role identities and the contexts that may promote visitors’ active and agentic engagement in exploring their identities. 

   more » « less
4. Automated patient-robot assignment for a robotic rehabilitation gym: a simplified simulation model

   https://doi.org/10.1186/s12984-022-01105-4  

   Miller, Benjamin A.; Adhikari, Bikranta; Jiang, Chao; Novak, Vesna D. (November 2022, Journal of NeuroEngineering and Rehabilitation)

   Abstract BackgroundA robotic rehabilitation gym can be defined as multiple patients training with multiple robots or passive sensorized devices in a group setting. Recent work with such gyms has shown positive rehabilitation outcomes; furthermore, such gyms allow a single therapist to supervise more than one patient, increasing cost-effectiveness. To allow more effective multipatient supervision in future robotic rehabilitation gyms, we propose an automated system that could dynamically assign patients to different robots within a session in order to optimize rehabilitation outcome. MethodsAs a first step toward implementing a practical patient-robot assignment system, we present a simplified mathematical model of a robotic rehabilitation gym. Mixed-integer nonlinear programming algorithms are used to find effective assignment and training solutions for multiple evaluation scenarios involving different numbers of patients and robots (5 patients and 5 robots, 6 patients and 5 robots, 5 patients and 7 robots), different training durations (7 or 12 time steps) and different complexity levels (whether different patients have different skill acquisition curves, whether robots have exit times associated with them). In all cases, the goal is to maximize total skill gain across all patients and skills within a session. ResultsAnalyses of variance across different scenarios show that disjunctive and time-indexed optimization models significantly outperform two baseline schedules: staying on one robot throughout a session and switching robots halfway through a session. The disjunctive model results in higher skill gain than the time-indexed model in the given scenarios, and the optimization duration increases as the number of patients, robots and time steps increases. Additionally, we discuss how different model simplifications (e.g., perfectly known and predictable patient skill level) could be addressed in the future and how such software may eventually be used in practice. ConclusionsThough it involves unrealistically simple scenarios, our study shows that intelligently moving patients between different rehabilitation robots can improve overall skill acquisition in a multi-patient multi-robot environment. While robotic rehabilitation gyms are not yet commonplace in clinical practice, prototypes of them already exist, and our study presents a way to use intelligent decision support to potentially enable more efficient delivery of technologically aided rehabilitation. 

   more » « less
5. A qualitative exploration of parents and their children's uses and gratifications of ChatGPT

   https://doi.org/10.1111/fare.13171  

   Zhang, Shirley; Li, Jennica; Cagiltay, Bengisu; Kirkorian, Heather; Mutlu, Bilge; Fawaz, Kassem (March 2025, Family Relations)

   Abstract ObjectiveThis Emerging Ideas report explores families' (parents and their children) uses and gratification for ChatGPT. BackgroundGenerative artificial intelligence–based conversational agents, such as ChatGPT, can be used to accomplish a variety of tasks, yet little is known about how and why parents and their children may use these technologies. MethodsWe conducted semistructured qualitative and exploratory interviews with 12 U.S.‐based families that had experience sharing a ChatGPT account. Families were recruited using social media advertisements, and at least one child and one parent joined the interview. We asked families about what they used ChatGPT for and why they used the platform. ResultsFamilies reported four main motivators for using ChatGPT: (a) information seeking, (b) enhancing productivity, (c) entertainment, and (d) social bonding. Potential barriers to use included concerns about (a) ChatGPT's credibility and capabilities, (b) being less familiar with using ChatGPT, (c) the platform's ethical implications, and (d) possible privacy risks. ConclusionFamilies use ChatGPT for various purposes, but their uses and gratifications sometimes may differ depending on their perceptions of and experiences with the platform. ImplicationsOur findings suggest that with some improvements, ChatGPT has the potential to be a useful tool for both individual and shared use in families. 

   more » « less