Audible alerts are frequently used in critical care settings to notify providers of abnormal patient values or deteriorating patient status. Recent work has examined how clinical decision support systems (CDSSs) can advance beyond issuing alerts about changes in patient status to also providing decision support about suggested interventions [12,13,32]. Prior research has also suggested integrating decision support into the existing systems rather than developing stand-alone CDSSs, which interrupt clinical workflows and patient care [12,27,32]. Clinicians responding to emergency patient scenarios in critical care settings often use digital cognitive aids-tools such as checklists and flowcharts that aid in tracking task completion [17]. Past studies on cognitive aids examined design issues [14,16], compliance rates [15,26], and impact on patient care [7,9,28]. These studies have found that cognitive aids reduce errors [28], lead to faster task completion [7], and influence team communication [9]. Despite these efforts, research on alerts design for cognitive aids is limited. A recent study found that using alerts on a digital checklist significantly increased documentation [19].
Less is known, however, about decision support alert design for cognitive aids used in team-based medical settings. Decision support alerts have been previously studied in association with electronic health records (EHRs) in primary care and inpatient hospital settings [6,8,23,30]. These studies found mixed results, with some alerts improving clinical outcomes [8] and others being frequently overridden and not affecting clinical outcomes [6,23]. These results highlight the importance of studying the design of alerts because poor user interfaces and poor presentation of recommendations can lead to alert fatigue, where users frequently ignore or override alerts [22]. For example, the design of decision Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. support alerts on an EHR often requires scrolling through multiple alerts that lack the information needed for assessing their accuracy [30]. In this late-breaking work, we explore how decision support alerts could be designed for cognitive aids in critical care settings by using a digital checklist for pediatric trauma resuscitations as a design probe.
Pediatric trauma resuscitations are time-critical events, where a team of providers cares for a severely injured child.
CDSSs have been proposed to aid trauma teams' situation awareness because the intense time pressure often leads to attention tunneling, where clinicians focus on obvious injuries while missing less noticeable but potentially more critical injuries [31]. In this paper, we interviewed 11 trauma team leaders to better understand how decision support alerts should be designed for cognitive aids in this domain. We used four alert mockups (pre-arrival alert, banner alert, insights screen, and pop-up alert) as design prompts and elicited feedback on the alerts design for the checklist. Our findings suggested three implications for designing cognitive aid alerts: (1) balance noticeability and distractibility, (2) consider relationship between patterns of cognitive aid use and alert design, and (3) include appropriate information for the user and team. We conclude by discussing how these findings inform our future work.
We interviewed 11 trauma team leaders at a level one trauma center between January and April 2021. The hospital's Institutional Review Board approved the study and participants were offered monetary compensation for their time.
Our participants were surgical residents and fellows (n=4, median years of experience=6) and ED physicians (n=7, median years of experience=12), who regularly lead trauma resuscitations at the trauma center. The trauma center is located in the U.S. Mid-Atlantic region and treats approximately 400 patients each year. The surgical resident or fellow and ED physician co-lead trauma teams following the Advanced Trauma Life Support (ATLS) protocol [5]. The ED physician is a permanent hospital physician with extensive pediatric experience but less training in trauma care. The surgical resident or fellow rotates at the hospital while being trained in trauma care but often has less experience in treating pediatric patients. Other team members include a junior resident performing the exam and bedside nurses assisting with tasks such as obtaining vital sign values and administering treatments. Since 2017, the surgical team leaders have been using a digital checklist during the resuscitation. The checklist is implemented on a Samsung Galaxy tablet and contains four tabs or sections, each corresponding to a phase of the resuscitation. The pre-arrival section lists the preparatory tasks that should be performed before patient arrival. The primary survey section contains tasks related to evaluating the major physiological systems for critical injuries, including the airway, breathing, and circulation. The secondary survey section has tasks for examining the patient from head to toe for other injuries. Finally, the prepare for travel section contains items that should be completed when preparing to transport the patient to their next hospital destination. Towards the top of the screen is a section to record the vital sign values and a place to take handwritten notes. When users start a new checklist, a pre-hospital form appears on the screen, allowing leaders to document pre-arrival context information about the patient. The checklist currently has one set of alerts which are triggered if the vital sign values remain undocumented for a specific period of time. The missing vital sign items begin to pulse and a dropdown alert appears informing the users of the undocumented vital signs.
During the interviews, we presented mockups of four different alerts on the digital checklist (Figure 1). Mockups can be used in exploratory design research to prompt discussion and collaboration between researchers and participants with clinicians. First, we aimed to design alerts that would not distract from patient care, an important criterion that emerged from our earlier user sessions where we asked clinicians about their experiences with the checklist alerts about undocumented vital signs. Second, we aimed to create alternative designs to interruptive modal dialogs (e.g., pop-up alerts). A recent study explored the design of alternative decision support features in electronic health records, finding that the alternative designs led to more appropriate medication prescribing [11]. Another study found that the alerts blocking users from using the system until the alert is resolved can reduce system adoption, even though they may be more effective than the non-blocking alerts [25]. Third, we aimed to create alerts that would incorporate the context variables informing the system's decision to trigger the alert. Clinicians distrust decision support alerts when they cannot evaluate the information behind the alert [13]. Alert designs should balance transparency and complexity, providing enough information for the clinician to evaluate the alert without slowing their decision making [12].
The first three mockups were alternatives to the more traditional pop-up alert. In the pre-arrival alert design (Figure 1.A), alerts appear on the top of the pre-hospital form as users document patient context information (e.g., age, injury type) before the patient's arrival. In the banner alert design (Figure 1.B), a banner containing the alert text appears in yellow at the top of the screen. In the insights screen design (Figure 1.C), an additional tab lists the alerts triggered throughout the case. The variables behind each alert are shown, along with a button to dismiss the alert. A button to access this insights screen is placed below the buttons to switch between the checklist tabs. When new alerts are issued, a red circle with the number of alerts appears next to this button. The final and fourth mockup is the pop-up alert design (Figure 1.D), which also contains the variables informing the alert and a button to dismiss the alert.
The interviews were thirty-minute long and conducted remotely over Zoom, a video-conferencing platform. We began by discussing the types of decision support alerts that should be included in this setting. Next, we asked participants to describe how they envisioned the design of alerts. We then gave an overview of the digital checklist. During the main part of the interview, we presented each alert mockup, informing participants that the design could be used for Woodstock '18, June 03-05, 2018, Woodstock, NY Anon.
any type of decision support alert. After showing each mockup, we asked the participant to describe their thoughts about the alert design, discuss any perceived issues, and suggest changes. For the banner alert design, we also asked the participant if the most recent alert or the highest priority alert should be displayed. At the end of the interview, we displayed all four mockups on one screen and asked participants to discuss their preferences between the four designs.
All participants consented to being recorded in the interviews. The audio recordings were automatically transcribed by Zoom, with two researchers manually correcting the transcripts. Next, two researchers analyzed the data using an inductive qualitative content analysis approach [3], focusing on how the interview data could inform the design of alerts for cognitive aids. The researchers independently open-coded the transcripts using NVivo, a qualitative data analysis software. The researchers then met to discuss the codes and iteratively perform axial coding to identify themes.
For each of the four alert designs, we present our key findings about participants' reactions and suggested changes.
We identified two themes in participants' reactions to the pre-arrival alert that highlighted the advantages of this design:
(1) presentation of alerts without obstruction of checklist, and ( 2 Participants also discussed how alerts triggered before the patient arrival could help with decisions and preparation: "...This kind of reminder is really, really useful. You know, to say 'Oh, this is a penetrating trauma with altered vital signs and an altered GCS [Glasgow Coma Score indicating the patient's neurological status],
I should be getting the blood transfusion protocol ready. '" [ED Leader (EDL) #6] It was also noted that it is easier to manage and discuss alerts before the team must shift their attention to the patient:
"If you're putting in the pre-arrival information, I think that alert makes sense because you're not distracted yet, like you don't have a patient...in that moment, that's all you're focused on..
We also identified one theme that highlighted a drawback for this alert design: potential for missing information. Four participants [SL#2, EDL#1,2&6] explained how the pre-arrival alert design may be less effective when the information that triggers the alert is unknown, inaccurate, or documented with a delay. The emergency medical services (EMS) team transporting the patient usually calls the hospital to announce the incoming patient. This call is then used to activate the trauma team by sending them a page with the available information about the case. Our participants described how vital signs values, age, and measures of consciousness were often unknown or inaccurate before the patients arrived.
The participants discussed how this missing information might affect the alerts and suggested using categories or ranges when exact numbers were not known:
"For GCS, sometimes we just have 'altered. ' One participant suggested making the entire top section red to be more noticeable: "Maybe make the very top layer red... so it's a little more visually jarring" [SL#3]. Another participant [EDL#3] wanted the alert information to give a recommendation (e.g., "provide blood") instead of a prediction (e.g., "increased risk of needing blood").
We identified three themes in the participant reactions to the banner alert: (1) the unobtrusiveness of this design
Two themes emerged in participant reactions to this separate screen synthesizing the alerts. The first theme highlighted the screen's ability to serve as a reference point. Four participants [SL#1,3,4, EDL#1] considered the insights screen as a good reference during the case and appreciated the variables that triggered the alerts, describing how this information improved transparency and helped them interpret the alerts: "It's an unobtrusive way to hide it, it's a good way to catalog and make sure. And then in those three minutes when you're spinning your wheels and not much is happening, a good place to check. The participants also discussed the timing of when they would check this screen throughout the case. Two participants [SL#2, EDL#1] thought they would check it at the beginning, when preparing for the patient. Two participants [EDL# 1,2] discussed checking it between the primary and secondary survey, both believing that this timing would depend on the situation. Three participants [SL#1,2,4] were concerned that they would not check it until the end of the resuscitation:
"I'd be worried that it would just be another tab that would get lost. Because you already have a lot of tabs and if it's at the bottom, it would be something that I wouldn't check until I'd gone through everything else, because your natural flow is to go down the side. That would be my only concern. " [SL#4] Moving the insights screen button up somewhere along the four major tabs was suggested, giving consideration to the tendency for checking the alerts during pauses. Two participants [EDL#4,5] also recommended a combination of insights screen with other alert types, especially with a banner alert that could be clicked to open the insights screen.
The most polarized reactions were expressed for the pop-up alert design. We identified three themes in these reactions:
(1) the ability of the pop-up alerts to attract attention (advantage), ( 2 This participant later described how they would not want the alert to block their documentation on the checklist, but would like the alert to remain on the screen to assist with team discussions: "It would be nice to have that up in case you announced it to the team, like the shock index is above the threshold and these are the reasons why, in case someone asked" [SL #4]. Two participants [SL#3, EDL#6] also mentioned that pop-up alerts should not block checklist use, suggesting to place the alerts in the handwritten note area to avoid obscuring the checklist. One ED physician worried that the pop-up alert design, especially when blocking the checklist, could distract from clinical thinking and lead to alert fatigue: Another ED leader discussed how the accuracy of the alert could impact whether the user is required to dismiss it: "If you can set the threshold to be reasonable, like the positive predictive value is 50%, so I'm only falsely alarmed half the time, then making it required to dismiss it is okay. " [EDL #5] Other suggestions for improving this alert design involved making it more visually obvious (e.g., using a "a thick border of red" [EL#6]), keeping a repository for all alerts including dismissed ones, and displaying the normal range next to the values used in the alert. One participant [SL#3] described how having the normal ranges would help them interpret the values because they are less familiar with pediatric patients.
The interviews with clinicians highlighted advantages and disadvantages of different alert designs while emphasizing the challenges of designing alerts in time-critical, team-based environments. From our findings, we discuss three implications for designing alerts for cognitive aids used in these settings: (1) balance noticeability and distractibility, (2) consider relationship between cognitive aid use and alert design, and (3) include appropriate information on the alert.
One of our design criteria was to create alerts that would not distract from patient care. In these initial designs, however, we may have reduced alert noticeability in an attempt to reduce its distractibility. Feedback from our study participants highlighted that noticeability is an important design criteria for alerts. Participants frequently referenced noticeability when discussing alert design and suggested changes to make all four designs more noticeable. Similar to findings on alert designs in EHRs [25], our participants considered alerts to be distracting and obtrusive if they prevented their use of the cognitive aid. Because all suggested improvements concerned the visual aspects of the alerts (e.g., using brighter colors, changing the positioning of alerts, and incorporating thicker borders), one potential way of balancing noticeability and distractibility is to improve alerts' visual noticeability without blocking the use of the cognitive aid.
In addition to noticing individual alerts, participants also discussed the importance of noticing when alerts update and managing multiple alerts. In time-critical events, multiple alerts may be triggered during a short period. Many of our participants stressed the importance of avoiding hard-stop alerts, which block system use until the alert is resolved. One participant discussed how the alert accuracy should determine if a user is required to dismiss it. Another study similarly suggested that accuracy of an alert should impact its intrusiveness [21]. If users can continue using the system without resolving alerts, multiple alerts may accumulate on the interface. To avoid having many alerts on the screen at once, we need to determine the approaches for dismissing the alerts. Our interviews highlighted three potential approaches: (1) allowing users to dismiss alerts at any time, (2) automatically dismissing alerts after they have been displayed for a specific amount of time, and (3) displaying a set number of recent alerts on the screen. Removing alerts from the interface only after they are dismissed by the user could help ensure that the alerts are noticed. This approach, however, requires additional user effort and many alerts may stay on the interface if the user does not dismiss them. Dismissing alerts without user intervention (either after a set amount of time or when the number of alerts is exceeded) could reduce this additional work and risk, but could also lead to alerts disappearing before being noticed. A hybrid combination of approaches could also be used. For example, allowing users to dismiss alerts and automatically removing them after a specified time if they had not been dismissed by the user. The notion of balancing noticeability and distractibility is also reflected in a prior study of critical care alarms, where participants ranked light pattern designs based on noticeability and level of distraction [4]. In our future work, we will evaluate how different alert designs and different approaches to dismissing alerts affect the noticeability and distractibility factors.
Cognitive aids are designed to be used throughout an event, aiding the user in tracking completion of important tasks [18]. Our findings showed that the stage of progress in the event can affect the alert design because both the user tolerance for distraction and availability of information required for triggering the alerts will vary between stages.
Patterns of cognitive aid use can also influence the alerts design. Depending on the domain and context, checklists may be used in a linear manner, where users advance through the items in order, or a sample manner, where users jump between different sections and items throughout the event [2]. Some participants in our study described using the checklist in a linear manner, discussing how they might only go to a separate screen for alerts when preparing to move to the next checklist section. For checklists used in a sample manner, users may be more inclined to visit a separate screen with alerts since they are already frequently moving between different sections. Our future work will further explore how alert design should differ between checklists used in a linear manner and checklist used in a sample manner. Our study participants also discussed how they had to balance the documentation on the checklist and viewing the alerts. Prior work has proposed integrating cognitive aids with other systems in the environment to reduce the documentation burden [24,33]. If this burden is reduced, alert designs requiring more attention may be more feasible.
Similar to findings in prior work [12,13], participants discussed the importance of displaying the variables used to trigger an alert to assess its accuracy and severity. One participant in our study also suggested including the normal ranges next to the values to help less experienced clinicians with the alert interpretation. Because designs should balance transparency and information complexity [12] and not all users may need the normal ranges, this information could be designed for on-demand viewing. For example, users could click an icon next to the values to see the normal ranges, hiding this extra information from users who do not need it. One participant in our study described how they might share the information included in the alert when discussing it with the team. For cognitive aids used in team-based settings, it may be important to consider not only how the user of the cognitive aid will interpret the information in the alert, but also how they can be supported in sharing that information with the wider team.
Another participant in our study preferred action statements (e.g., perform this intervention) over predictions (e.g., increased risk of needing this intervention), a sentiment also found in prior literature [32]. Although some clinicians may find action statements easier to interpret than predictions, others may think they infringe on their autonomy.
Concerns that decision support systems will intrude on autonomy and agency are one reason why clinicians are hesitant to use these systems [29]. Additionally, systems developers may be concerned that providing action statements instead of predictions will have ethical and legal ramifications. Our future research will explore how switching from predictions to action statements influences user autonomy and accountability.
By interviewing 11 clinicians with experience in leading trauma resuscitations, we identified three design implications for developing alerts for cognitive aids used in team-based, time-critical medical events. The participants highlighted the importance of noticing both individual alerts and changes in alerts, suggesting visual design changes to increase alert noticeability. Based on this feedback, we developed three approaches to managing multiple alerts on the cognitive aid interface. The interviews also highlighted the need to consider the relationship between the patterns of cognitive aid use and the alert design, as well as the information to include on the alert for both the user and team. Finally, we discussed opportunities for future work, which include studying approaches to dismissing alerts, understanding how patterns of checklist use influence alert design, and exploring how action statements and predictions affect user autonomy and accountability.