New graduates' insufficient practice readiness persists even as the demand for nurses is growing in the United States [1]. This is a multifaceted challenge since nursing educators are faced with several paradigm shifts and competing gaps in preparing pre-licensure students for the profession. For instance, with the recent release of The Essentials: Core Competencies for Professional Nursing Education [2] and National Council of State Boards of Nursing Clinical Judgment Measurement Model (NCJMM) [3], nursing regulatory bodies have placed competency-based education at the forefront, prompting programs to transform teaching, learning and assessment practices. In addition, the Future of Nursing 2020-2030 report [4] has underscored the need for new nurses to be prepared to (a) treat patients that reflect diversity in social determinants of health and (b) promote health equity across communities. Furthermore, U.S nursing schools are having to turn away thousands of qualified applicants due to shortages of clinical sites, faculty, and resource constraints [5]. Lastly, nursing leaders are foreseeing a continued trend toward online/remote education. This pedagogical movement along with rapid digital transformation is likely to create new opportunities and challenges for nursing programs and regulations [6].
We believe that screen-based virtual simulations have the potential to cultivate students' practice readiness and aid nursing educators in addressing the aforementioned shifts and gaps in the discipline. Foronda and colleagues [7] concluded that utilizing virtual patient simulations (VPS)-a type of screen-based simulation-had a positive effect on multiple learning outcomes for nursing students. Eighty-six percent of studies in their review demonstrated that VPS were efficient at enhancing nursing students' knowledge acquisition, skill development, critical thinking, self-assurance, and satisfaction with learning. Recently, Cole [8] urged researchers to investigate learner performance as a direction for advancing simulation use for competency-based education.
Quantitative ethnography (QE) has enabled researchers and practitioners to investigate and illustrate complex patterns in human behavior in several domains. We build upon extant QE research, especially in nursing education, and apply Transmodal Analysis (TMA) for investigating students' interaction patterns in multimodal learning activities in Shadow Health® Digital Clinical Experiences TM (DCE). DCE is a type of VPS designed to cultivate nursing students' knowledge, skills, and attitudes for providing comprehensive and compassionate care to digitally standardized patients.
In what follows, we describe learning activities typically afforded by DCE, and provide an overview of DCE scenarios designed for gerontology, mental health, and community health content areas. Next, we describe the theoretical framework of this study, The Essentials [2], and delineate its application in this study. This is followed by a justification for using TMA to advance QE research on simulations in nursing education. Thereafter, we de-scribe our methodological procedures and modeling decisions. This is followed by a report of findings for Rose and Roshni (pseudonyms) who completed two focused exams (End of Life scenario in Gerontology, Bipolar Disorder scenario in Mental Health) and one contact tracing assignment (HIV Diagnosis and with Contact Tracing scenario in Community Health) in DCE from 2022-2023. We conclude this paper by discussing the study findings and outlining implications for future research.
The DCE provides an array of standardized clinical scenarios across a comprehensive range of courses in undergraduate nursing education to guide learners in developing the knowledge, skills, and attitudes needed to care for diverse patients in a safe environment. Typically in a DCE scenario, learners can interact with virtual patients and ask questions to explore their medical and sociocultural backgrounds. Learners can also perform physical exams and document their findings in a simulated electronic health record (EHR). During the virtual exam, learners can also express empathy when the virtual patient shares emotional, physical, or experiential difficulties, and offer educational statements when the patient reveals gaps in their understanding of relevant medical topics. The virtual patients are programmed to recognize and respond to thousands of questions and statements related to the learning objectives covered in each scenario, making the conversation feel natural and realistic. As such, learners engage in a clinical reasoning process by completing patient care activities such as collections of history and physical examination data, therapeutic communication skills, and creations of care plans.
DCE simulations have been successful at increasing critical thinking, confidence, and satisfaction among undergraduate nursing students [9]. Students as early as in their first year of nursing education have demonstrated significant efficiency gains when it comes to gathering patient data, applying therapeutic communication, and creating care plans using DCE [10]. In addition, recent studies have shown that DCE scenarios can be an effective means of teaching nursing students about patient care issues that they may not encounter as part of their clinical education, such as fostering cultural competence and sensitivity when caring for transgender patients [11].
For this study, we focus on DCE scenarios from the following content areas in undergraduate nursing education: Gerontology, Mental Health, and Community Health. In Gerontology scenarios, students interact with a diverse range of older adult patients, gathering data to assess risk for geriatric syndromes and medication contraindications using Beers Criteria. They take complete health histories, perform problem-focused physical assessments and construct care plans. In Mental Health scenarios, students engage with a set of patients who are experiencing a variety of mental health conditions. They take complete health histories, perform mental status assessments, conduct problem-focused physical exams, and complete either care plans or Situation Background Assessment Recommendation (SBAR) handoffs. In Community Health scenarios, students explore a systems-view ap-proach to healthcare-assessing community strengths and weaknesses, tracing the spread of disease, advocating for vulnerable populations, treating individual patients, and creating a care plan for the community.
The Essentials [2] provides a competency-based education framework for guiding the development and revision of nursing curricula to prepare entry-level and advanced-level nurses. In addition, it outlines programmatic expectations for teaching, learning, and assessment at both levels. Competencies and sub-competencies are organized within 10 domains. These are applicable across all healthcare areas and diversity of patient populations. At the entrylevel, learners should demonstrate attainment and integration of level 1 sub-competencies. Eight concepts (Clinical Judgment; Communication; Compassionate Care; Diversity, Equity, and Inclusion; Ethics; Evidence-based Practice; Health Policy; Social Determinants of Health) are central to professional nursing practice, integrated across the domains and competencies and included in The Essentials.
In this study, we examined competencies characterizing Domains 1 (Knowledge of Nursing Practice), 2 (Person-Centered Care), and 9 (Professionalism) described for entrylevel programs. These included: Demonstrating clinical judgment founded on a broad knowledge base, Engaging with the individual in establishing a caring relationship, Communicating effectively with individuals, Integrating assessment skills in practice, Diagnosing actual or potential health problems and needs, Promoting self-care management, Providing care coordination, and Employing participatory approach to nursing care. For each of these competencies, we used select level 1 sub-competencies to guide our operationalization of the theoretical constructs we examined in this study (see Table 1 for codebook). We also used NCSBN Clinical Judgment Measurement Model Layer 3 [3] to guide our examination of learner performance in DCE.
At the entry-level, it is important for nursing students to be exposed to varied experiences in four spheres of care (Disease Prevention/Promotion of Health and Well Being, Chronic Disease Care, Regenerative / Hospice / Restorative Care, and Hospice / Restorative Care Palliative Care) with diverse populations and ages [2]. These guidelines along with the core concepts informed our choice of specific scenarios from the three content areas and two types of assignments (focused exam and contact tracing). We examined transcripts of students' interactions with three virtual patients; namely, Regina Walker from gerontology, Lucas Callahan from mental health, and Quan Tran from community health. Regina is a 69-year-old Black/African American, cisgender, and heterosexual woman. She is a retired family coach and program director at a non-profit. Regina is experiencing increased pain and decreased activity due to metastatic cancer. She needs recommendations for and discussion on hospice care. Lucas is a 25-year-old White cisgender and heteroflexible man who is currently unemployed. He is at risk of intentions to harm himself or others. Lucas needs education on symptoms common with a hypomanic state including lack of sleep. He also needs a care plan and recommendation for outpatient therapy. Quan Tran is a 52-yearold Vietnamese American man. He chooses not to disclose his sexual orientation and is employed as a manager at a trading company. Quan was recently diagnosed with HIV at a community center. The possible contacts of this communicable disease need to be traced using evidence-based guidelines. Quan and the contacts also need to be educated and cared for with empathy. Through these scenarios, students were exposed to patients needing Hospice/Palliative Care (i.e., Regina-End of Life), Chronic Disease Care (i.e., Lucas-Bipolar Disorder), and Prevention/promotion of Health and Wellbeing (i.e., Quan-HIV Diagnosis and Contact Tracing).
Quantitative Ethnography (QE) is an emerging field for understanding complex processes and discovering meaningful patterns in various disciplines such as education [12], and policy [13]. As a unified approach of qualitative and quantitative analyses, QE provides both thick descriptions and statistical warrants on a given analytic claim [14]. Recent studies in nursing education have applied QE methods to examine alignment of curricular content [15], trace student learning trajectories [16], model instructor facilitation and classroom interaction across pre-briefing, simulation, and debriefing phases for scenarios in fundamentals of nursing [16] [17].
Nursing education by nature involves multiple modalities such as dialog, physical examinations, and documentation. However, existing work about nursing education in QE mainly relies on unimodal data. For example, Shah and colleagues [18] adopted epistemic network analysis (ENA) to investigate student learning trajectories based on discourse data collected from virtual reality simulation sessions. According to this study, ENA represented connections made among constructs derived from frameworks such as NCJMM and Quality and Safety Education for Nurses (QSEN). This study had two limitations: (1) ENA did not provide ordered relationships for the connections between any pair of constructs or selfreferences and, (2) ENA was not initially designed to analyze multimodal data and often requires a great deal of additional model parameterization. In ENA, it is not possible to assign different window sizes for different modalities, and as a result researchers often need to account for differences in the temporal influence of different data streams by manual adjustment in their model.
Current advancements in QE methods can help address these limitations. Specifically, Ordered Network Analysis enables researchers to represent self-references and ordered relationships of connections made during a learning activity such as a nursing simulation. Additionally, Transmodal Analysis enables researchers to model multimodal data by (a) specifying a function or functions that describe, for each data modality, how events interact and (b) using those functions to include multiple modalities in the same model [19]. As such, in this study, we implemented TMA combined with Ordered Network Analysis (T/ONA) [20] by specifying a different window size for each data modality. As described in further detail below, data was collected from students engaged in DCE through conversations, virtual exams, and documentations. These modalities (dialog, click, documentation) are highly interactive and interwoven during the learning processes in DCE. Hence, we chose to use TMA to model the cross-modality interactions in complex thinking and activity in virtual patient simulations.
Purposive sampling was applied to identify two Bachelor of Science in Nursing (BSN) students from the same cohort at a public university in south central United States. The students were selected because they had completed the three scenarios at the time of the study. Rose and Roshni (pseudonyms) were enrolled in the Adaptation in Aging and Psychiatric-Mental Health Nursing courses in Fall 2022 where they completed the scenarios for Regina Walker and Lucas Callahan respectively. They completed Quan Tran's scenario in Spring 2023 as part of their Community Health Nursing course.
In this study, the interview guide was fully enabled by the instructor for all three assignments in DCE. The interview guide is meant to scaffold students' DCE as they engage in subjective data collection. The interview guide shows students the high-level outline for each section of the patient interview they will need to collect. Faculty may choose to (partially or fully) enable or disable the interview guide when assigning assignments to students. The full option allowed Rose and Roshni to see explicitly what subjective findings were scored in the interview (e.g. asked about chief complaint) before they uncovered them. This was the typical preference for faculty at the institution where the two students were enrolled.
We organized and examined a total of 1760 lines of timestamped utterances for the two students' (Rose and Roshni) transcripts from three scenarios (Regina, Lucas, Quan). The utterances included a variety of interactions (answer, clarification, exam action, feedback, greet, prompt, response, statement) logged between participants (student, patient, the system, and other virtual characters). These interactions characterized the nature of specific learning activities (e.g., objective data collection, subjective data collection, education, and empathy) students typically engage in DCE across three modalities (click, dialog, documentation) and distinct phases (assessment, care plan, contact tracing) in a scenario.
The nested nature of each utterance provided insight into the overall pedagogical structure of the simulation experience. For instance, the dialog data involved a conversation with the virtual patient for subjective data collection related to but not limited to the history of the patient's present illness, past medical history, review of systems, and social history. Dialog data also captured students' empathizing and educating their patients. The click data involved examining the patient, performing physical assessments, interpreting observations for any abnormalities, and practicing contact tracing. The documentation data involved the student summarizing and/or interpreting the state of the patient throughout the scenario. The temporal structure of the utterances was meaningful to understanding the sequence each student followed in a specific scenario and the amount of time they spent in each learning activity. We applied a combination of manual and automated approaches to code the dataset using the codebook above (See Table 1). Automated coding allows researchers to operate automatically and minimize human efforts; however, it is challenging to obtain high accuracy or other model evaluation (i.e. recall, Kappa, F-score, etc) when coding affect-intensive complex constructs [21] [22] and domain-specific jargons and terms [23] [21]. In our study, considering the affordances and constraints of each coding method and our grounded understanding of the dataset used, we manually coded the constructs of Therapeutic Communication (TC), Subjective Assessment (SA), and Care-Management and Coordination (CMC) under social moderation [24]. These codes manifested in unique ways in students' transcripts based on the context of each scenario. Whereas, automation was applied to code students' actions for recognizing cues (RC), analyzing cues (AC), diagnosing and prioritizing hypotheses (DPH), generating solutions (GS) and evaluating outcomes (EO) because the conceptual definitions of these codes were consistent with procedural definitions (keyword matching) as captured by the DCE system. For example, logs were coded for an occurrence of RC when the student performed physical assessments using the exam action (e.g., assessed vitals). If the student correctly interpreted the result of their exam action (e.g., normothermic), this was coded as an occurrence of AC.
We applied T/ONA to represent patterns of student performance across the three scenarios in DCE. In particular, we specified different window lengths for learning events in different modalities. This is important to note because specifying different window lengths is an attempt to account for uneven data sizes and their varying temporal impacts in multimodal learning analytics. For instance, in a hypothetical context of collaborative problem solving where a student is involved in peer discussion and has access to resources in a system, chats may have a shorter window of impact on future learning events compared to searching and comprehending resources. That is, topics can rapidly change in a discussion, but engaging with a resource may have a longer influence on a student's connection-making compared to chatting.
In this study, we defined the smallest unit of analysis as students within learning activities. For each unit, ONA calculated and accumulated connections across eight codes (See Table 1) within recent temporal contexts. With learning events collected from three modalities, we operationalize recent temporal contexts for clicks, dialog, and documents respectively. That is, we configured unique window sizes for different modalities. Both patient interactions (dialog data) and exam actions (click data) have a strong temporal dependency and continuity due to interactivity between learners and the system. According to our qualitative analysis, we specified a window of five learning events for these two modalities. However, we selected a shorter window for the documentation modality, the length of two learning events, because each documentation is usually connected with the next action due to system design.
Using the methodological procedures and coding decisions described above, we developed and interpreted a combination of TMA and ONA (T/ONA) graphs to answer the two research questions (RQs):
1. How do students engage in DCE scenarios? We examined the alignment between learning patterns and guided instructional design across the three scenarios.
We compared the patterns of connection-making across the focused exam and contact tracing scenarios.
6 Results
To answer each research, we describe the connection patterns visually and statistically and interpret the dimensions based on the node positions in respective T/ONA graphs. These are supported by qualitative examples from students' performance in DCE scenarios.
6.1 RQ1: How do students engage in DCE scenarios?
Figure 1 represents the grand mean connections (strength, self-referencing, direction) of learning patterns for Rose and Roshni across all scenarios. The thickness of edges indicates the strength of connections; arrows on edges indicate the main direction of connections; and the radius of the outer contour of a node indicates the total receiving degree, while the radius of the inner contour of a node indicates the degree of self-references. According to the plot, students made self-references while practicing subjective assessment. That is, students spent a prolonged period performing a holistic assessment and obtaining a complete history of the patient. Self-referencing was also observed in students' practices of analyzing cues across the three scenarios. Students spent the bulk of their time in the scenarios interpreting patients' cues to relevant medical conditions/health problems. These activities preceded students' decision-making to foster patients' well-being. Therapeutic communication was central to Rose and Roshni's participation in the three scenarios. This allowed them to establish relationship-centered care, demonstrate empathy, practice humility and cultural sensitivity especially while engaging in subjective assessment and generating solutions for three different patients. Another pattern worth highlighting is students' practice of care management and coordination as a response to subjective assessment and generating solutions. Students were observed educating patients, promoting self-care, and facilitating continuity of care through coordination with family members and the healthcare team. Across the three scenarios, there was no significant difference between ONA scores for both students (t(74.424) = 1.692, p = 0.095, Cohen's d = 0.378). That is, both students practiced competencies related to clinical judgment, person centered-care, and professionalism in a similar manner as a result of using DCE. Below, we illustrate these connections to clinical competencies by drawing examples from Rose and Roshni's engagement in the gerontology scenario.
Students engaged in two phases in the end-of-life focused exam for virtual patient Regina Walker: assessment and care plan. During the assessment phase, Rose and Roshni inquired about major health deviations. This included interacting with Regina about her chief complaint (increased pain), history of present illness (onset, duration, location, characteristics, aggravating factors, relieving factors, and severity of pain), past medical history (existing health conditions, general and medication allergies), social history (substance use), review of systems (head, eyes, ears, nose, throat, respiratory, cardiovascular, gastrointestinal, neurological, psychological). A big portion of their dialogue centered around Regina's functional status and geriatric syndromes for which older adults may be at risk. This included questions about depression, living environment, feeling safe at home, bathing, dressing, toileting, transferring, continence, ability to eat independently, sleeping habits, confusion, evidence of falls, gait and ambulatory aids, skin breakdowns, weight changes, weight loss, oral or dental problems, appetite changes, health, and social activity, perception of health, fatigue, ER visits or hospitalizations). Both students complemented this subjective data collection by performing exam actions and noting their observations for any abnormalities. This included (a) assessing vitals, IV bags, IV pump, IV site, and urine quality; (b) inspecting eyes, mouth, and skin; (c) auscultating carotids, breath sounds, heart sounds, and bowel sounds; (d) palpating abdomen, bladder; (e) testing cognition, skin turgor, and capillary refill. In a relatively brief but important part of this phase, Rose and Roshni discussed Regina's comfort and preference for hospice care, the family's need for health care services, and implications for health policy, financing, and service availability.
Their care plan focused on symptom management and advocacy for appropriate palliative/hospice care. Specifically, Rose and Roshni indicated their diagnosis for Regina (readiness for effective coping), identified signs and symptoms (interest in hospice care, increased home care needs outlined short-term goals), and recommended interventions (e.g., evaluate the patient's current understanding of coping strategies supporting their transition to hospice care), discuss the plan with the patient, and evaluate their own understanding of hospice care as a result of their simulated experience with Regina.
There were subtle differences in students' participation patterns. For instance, Rose engaged in a dialogue with Regina before performing exam actions. Roshni chose the op-posite approach. Rose was also more compassionate in her communication during both phases; she (a) offered information about what to expect during the assessment phase, (b) provided reassurance during sensitive discussions, (c) expressed consideration for family and Regina while discussing an end-of-life transition plan. The following statements are an example:
Rose: Ms. Walker, we are going to work on ways for you to cope with the upcoming changes you are about to face. This will help the transition be less demanding on you as well as your family. I know this is going to be hard on you all, so I am here for any questions or concerns. I want to know how much education you have on coping strategies, give you some supplemental strategies, and also give you referrals.
Roshini: "I will provide you and your family members a referral to the funeral service, spiritual support and financial assistance if that is ok with you. That is, the type of assignment in DCE participated in influenced the clinical competencies students practiced applying during learning activities in a scenario. For connections made in focused exam scenarios, both care management, coordination, diagnosing, and prioritizing hypotheses were observed as common responses to analyzing cues. That is, after analyzing cues from the virtual patients, students tended to rank signs and symptoms, diagnose conditions, and provide care advice. Additionally, students made more connections from generating solutions to care management and coordination in focused exams. In other words, after generating solutions related to the virtual patients' conditions, Rose and Roshni offered corresponding care-management advice and coordination such as scheduling follow-up appointments. For the contact tracing scenario, students made more (1) self-references within subjective assessments, (2) self-references with therapeutic communication, (3) connections from subjective assessment to therapeutic communication, and (4) no connections to the clinical judgment codes (recognizing cues, analyzing cues, diagnosing and prioritizing hypothesis, generating solutions and evaluating outcomes) because there was no objective data collection and care planning in this scenario. Self-references and connections were influenced by a focus on conducting a comprehensive patient assessment, facilitating health literacy, preventing disease, and promoting well-being. Below, we illustrate Rose and Roshni's engagement in the community health scenario.
Students engaged in two phases of a contact tracing assignment for the virtual patient Quan Tran: assessment and contact tracing. During assessment, Rose and Roshni inquired about Quan's chief complaint (i.e., HIV diagnosis), history of present illness (testing history, prodrome), past medical history (vaccinations, allergies, past hospitalizations), social history (home life, support system, substance use, typical diet), medication (herbal supplements, antiretroviral prescription), sexual history (sexual partners), review of relevant systems (constitutional and mental health; integumentary, respiratory, and cardiovascular system), patient needs (goals and priorities), and social determinants of health (employment, health insurance, education). Once again there were subtle differences in Rose and Roshni's participation. For instance, Rose followed up with Quan when he reported not understanding how HIV infection is transmitted, taking an herbal supplement consisting of echinacea and goldenseal, and wanting to keep his diagnosis hidden from coworkers and family members. Roshni did not respond to Quan's lack of understanding of HIV transmission; she followed up on his use of herbal supplements and the reaction of others to HIV status. She also educated him when Quan reported unfamiliarity with his antiretroviral medication, and feeling anxious about his diagnosis.
Students were prompted and guided by the system during the second phase of the sce-nario to model the process of tracing contacts and notifying partners. This included asking contacts about the results of their sexually transmitted infection (STI) panel or offering assistance to identify a clinic that performs STI testing, underscoring the importance of treatment from an HIV specialist and an appropriate medication regimen, encouraging regular tracking of viral load, maintaining confidentiality, and promoting safe sex practices. [25]. Virtual patient simulations provide a beneficial modality in which learners apply and practice their clinical reasoning and critical thinking abilities be-fore interacting with real patients [26]. However, current research on simulations heavily relies on self-evaluations [8].
In this study, we examined two undergraduate students' performance in Elsevier's Shadow Health® Digital Clinical Experiences TM (DCE) and provided evidence for their practice of clinical competencies characterized in Domains 1 (Knowledge of Nursing Practice), 2 (Person-Centered Care), and 9 (Professionalism) for entry-level programs in The Essentials [2] and Layer 3 of NCSBN Clinical Judgment Measurement Model [3]. We constructed a grand mean plot of two students' multimodal learning activities across three virtual patient scenarios and a subtraction plot to compare connection-making between focused exams and contact tracing assignments in DCE. The grand mean plot (Figure 1) indicated that students practiced competencies in clinical judgment, person-centered care, and professionalism across the patient diversity and care needs represented in gerontology, mental health, and community health contexts. Additionally, the subtracted plot (Figure 2) between focused exam and contact tracing indicated that the two assignment types afforded students to foreground specific competencies more than others.
There is a growing body of research in health care and health professions education that applies Quantitative Ethnography to investigate complex questions about professional enculturation and practice [27] [28]. However, few studies have used multimodal data and fewer exist in the nursing context [29]. This paper applied TMA to examine data obtained from three types of modalities (click, dialog, documentation) to make sense of students' engagement in collecting and interpreting patient data, synthesizing evidence, and promoting care that is suitable for each patient's condition (i.e., Disease Prevention/Promotion of Health and Well-Being, Chronic Disease Care, Hospice /Palliative Care). TMA allowed us to set and account for the impact of different window sizes for a variety of data types. For a complete discussion of both the mechanisms of TMA and its conceptual and theoretical underpinnings, please refer to a forthcoming paper [19].
In a previous study on DCE, researchers observed students practicing recognizing cues for a prolonged period and demonstrated the ways in which this clinical skill manifested in a pediatric and a geriatric patient scenario in the health assessment content area [30]. However, this self-referencing phenomenon could not be modeled using Epistemic Network Analysis (ENA). In this study, Ordered Network Analysis allowed us to not only capture the strength of connections among multiple clinical competencies that ENA typically allows, but also illustrate ordered relationship and self-referencing (Figures 1 &2) across two assignment types (focused exam and contact tracing) in three content areas (gerontology, mental health, community health).
Nursing programs are likely to continue their expanded use of virtual simulations even after the peak of the COVID-19 pandemic [31]. At the same time, extant QE research is providing valuable insight into the design and enactment of simulation-based learning in undergraduate nursing education (manikin-based, virtual reality, digital standardized patients) [15] [18] [30]. This study provides additional impetus for continuing the application of QE research methods in this discipline. In the future, researchers should broaden exam-inations using virtual patient simulations like DCE to include additional (a) foundational and specialty content areas in nursing, (b) clinical competencies, and (c) participants and sample sizes. Researchers should also consider deepening their examinations by investigating (a) students' performance in DCE scenarios where the interview guide is partially enabled or turned off, and (b) the impact of interaction types (e.g., questions, clarification) between participants (e.g., student, patient, system). Findings from these studies can yield recommendations on how nursing faculty and administrators can use virtual simulations systematically for fostering students' practice readiness.