skip to main content


Title: Multimodal Cue Combinations: A Possible Approach to Designing In-Vehicle Takeover Requests for Semi-autonomous Driving
The rapid growth of autonomous vehicles is expected to improve roadway safety. However, certain levels of vehicle automation will still require drivers to ‘takeover’ during abnormal situations, which may lead to breakdowns in driver-vehicle interactions. To date, there is no agreement on how to best support drivers in accomplishing a takeover task. Therefore, the goal of this study was to investigate the effectiveness of multimodal alerts as a feasible approach. In particular, we examined the effects of uni-, bi-, and trimodal combinations of visual, auditory, and tactile cues on response times to takeover alerts. Sixteen participants were asked to detect 7 multimodal signals (i.e., visual, auditory, tactile, visual-auditory, visual-tactile, auditory-tactile, and visual-auditory-tactile) while driving under two conditions: with SAE Level 3 automation only or with SAE Level 3 automation in addition to performing a road sign detection task. Performance on the signal and road sign detection tasks, pupil size, and perceived workload were measured. Findings indicate that trimodal combinations result in the shortest response time. Also, response times were longer and perceived workload was higher when participants were engaged in a secondary task. Findings may contribute to the development of theory regarding the design of takeover request alert systems within (semi) autonomous vehicles.  more » « less
Award ID(s):
1755746
NSF-PAR ID:
10171814
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
Proceedings of the Human Factors and Ergonomics Society Annual Meeting
Volume:
63
Issue:
1
ISSN:
2169-5067
Page Range / eLocation ID:
1739 to 1743
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Adults aged 65 years and older are the fastest growing age group worldwide. Future autonomous vehicles may help to support the mobility of older individuals; however, these cars will not be widely available for several decades and current semi-autonomous vehicles often require manual takeover in unusual driving conditions. In these situations, the vehicle issues a takeover request in any uni-, bi- or trimodal combination of visual, auditory, or tactile alerts to signify the need for manual intervention. However, to date, it is not clear whether age-related differences exist in the perceived ease of detecting these alerts. Also, the extent to which engagement in non-driving-related tasks affects this perception in younger and older drivers is not known. Therefore, the goal of this study was to examine the effects of age on the ease of perceiving takeover requests in different sensory channels and on attention allocation during conditional driving automation. Twenty-four younger and 24 older adults drove a simulated SAE Level 3 vehicle under three conditions: baseline, while performing a non-driving-related task, and while engaged in a driving-related task, and were asked to rate the ease of detecting uni-, bi- or trimodal combinations of visual, auditory, or tactile signals. Both age groups found the trimodal alert to be the easiest to detect. Also, older adults focused more on the road than the secondary task compared to younger drivers. Findings may inform the development of next-generation of autonomous vehicle systems to be safe for a wide range of age groups. 
    more » « less
  2. Background: Drivers gather most of the information they need to drive by looking at the world around them and at visual displays within the vehicle. Navigation systems automate the way drivers navigate. In using these systems, drivers offload both tactical (route following) and strategic aspects (route planning) of navigational tasks to the automated SatNav system, freeing up cognitive and attentional resources that can be used in other tasks (Burnett, 2009). Despite the potential benefits and opportunities that navigation systems provide, their use can also be problematic. For example, research suggests that drivers using SatNav do not develop as much environmental spatial knowledge as drivers using paper maps (Waters & Winter, 2011; Parush, Ahuvia, & Erev, 2007). With recent growth and advances of augmented reality (AR) head-up displays (HUDs), there are new opportunities to display navigation information directly within a driver’s forward field of view, allowing them to gather information needed to navigate without looking away from the road. While the technology is promising, the nuances of interface design and its impacts on drivers must be further understood before AR can be widely and safely incorporated into vehicles. Specifically, an impact that warrants investigation is the role of AR HUDS in spatial knowledge acquisition while driving. Acquiring high levels of spatial knowledge is crucial for navigation tasks because individuals who have greater levels of spatial knowledge acquisition are more capable of navigating based on their own internal knowledge (Bolton, Burnett, & Large, 2015). Moreover, the ability to develop an accurate and comprehensive cognitive map acts as a social function in which individuals are able to navigate for others, provide verbal directions and sketch direction maps (Hill, 1987). Given these points, the relationship between spatial knowledge acquisition and novel technologies such as AR HUDs in driving is a relevant topic for investigation. Objectives: This work explored whether providing conformal AR navigational cues improves spatial knowledge acquisition (as compared to traditional HUD visual cues) to assess the plausibility and justification for investment in generating larger FOV AR HUDs with potentially multiple focal planes. Methods: This study employed a 2x2 between-subjects design in which twenty-four participants were counterbalanced by gender. We used a fixed base, medium fidelity driving simulator for where participants drove while navigating with one of two possible HUD interface designs: a world-relative arrow post sign and a screen-relative traditional arrow. During the 10-15 minute drive, participants drove the route and were encouraged to verbally share feedback as they proceeded. After the drive, participants completed a NASA-TLX questionnaire to record their perceived workload. We measured spatial knowledge at two levels: landmark and route knowledge. Landmark knowledge was assessed using an iconic recognition task, while route knowledge was assessed using a scene ordering task. After completion of the study, individuals signed a post-trial consent form and were compensated $10 for their time. Results: NASA-TLX performance subscale ratings revealed that participants felt that they performed better during the world-relative condition but at a higher rate of perceived workload. However, in terms of perceived workload, results suggest there is no significant difference between interface design conditions. Landmark knowledge results suggest that the mean number of remembered scenes among both conditions is statistically similar, indicating participants using both interface designs remembered the same proportion of on-route scenes. Deviance analysis show that only maneuver direction had an influence on landmark knowledge testing performance. Route knowledge results suggest that the proportion of scenes on-route which were correctly sequenced by participants is similar under both conditions. Finally, participants exhibited poorer performance in the route knowledge task as compared to landmark knowledge task (independent of HUD interface design). Conclusions: This study described a driving simulator study which evaluated the head-up provision of two types of AR navigation interface designs. The world-relative condition placed an artificial post sign at the corner of an approaching intersection containing a real landmark. The screen-relative condition displayed turn directions using a screen-fixed traditional arrow located directly ahead of the participant on the right or left side on the HUD. Overall results of this initial study provide evidence that the use of both screen-relative and world-relative AR head-up display interfaces have similar impact on spatial knowledge acquisition and perceived workload while driving. These results contrast a common perspective in the AR community that conformal, world-relative graphics are inherently more effective. This study instead suggests that simple, screen-fixed designs may indeed be effective in certain contexts. 
    more » « less
  3. The introduction of advanced technologies has made driving a more automated activity. However, most vehicles are not designed with cybersecurity considerations and hence, they are susceptible to cyberattacks. When such incidents happen, it is critical for drivers to respond properly. The goal of this study was to observe drivers’ responses to unexpected vehicle cyberattacks while driving in a simulated environment and to gain deeper insights into their perceptions of vehicle cybersecurity. Ten participants completed the experiment and the results showed that they perceived and responded differently to each vehicle cyberattack. Participants correctly identified the cybersecurity issue and took according action when the issue caused a noticeable visual and auditory response. Participants preferred to be clearly informed about what happened and what to do through a combination of visual, tactile, and auditory warnings. The lack of knowledge of vehicle cybersecurity was obvious among participants. 
    more » « less
  4. Adaptive task allocation is used in many human-machine systems and has been proven to improve operators’ performance with automated systems. However, there has been limited knowledge surrounding the benefits of adaptive task allocation in automated vehicles. In this study, participants were presented with photos and videos depicting driving scenarios of low or high workloads at two levels of automation (SAE Levels 2 and 3). The participants reported which tasks they felt comfortable allocating to themselves or to the driving automation system (DAS) in each driving scenario, as well as whether they would conduct the task allocation manually or have the DAS automatically allocate the tasks. Our results showed that participants preferred conducting manual task allocation and preferred the system to complete more tasks when the perceived workload was high. There was no significant difference between the high and low workload scenarios in terms of whether participants chose to allocate tasks. 
    more » « less
  5. Trust calibration poses a significant challenge in the interaction between drivers and automated vehicles (AVs) in the context of human-automation collaboration. To effectively calibrate trust, it becomes crucial to accurately measure drivers’ trust levels in real time, allowing for timely interventions or adjustments in the automated driving. One viable approach involves employing machine learning models and physiological measures to model the dynamic changes in trust. This study introduces a technique that leverages machine learning models to predict drivers’ real-time dynamic trust in conditional AVs using physiological measurements. We conducted the study in a driving simulator where participants were requested to take over control from automated driving in three conditions that included a control condition, a false alarm condition, and a miss condition. Each condition had eight takeover requests (TORs) in different scenarios. Drivers’ physiological measures were recorded during the experiment, including galvanic skin response (GSR), heart rate (HR) indices, and eye-tracking metrics. Using five machine learning models, we found that eXtreme Gradient Boosting (XGBoost) performed the best and was able to predict drivers’ trust in real time with an f1-score of 89.1% compared to a baseline model of K -nearest neighbor classifier of 84.5%. Our findings provide good implications on how to design an in-vehicle trust monitoring system to calibrate drivers’ trust to facilitate interaction between the driver and the AV in real time. 
    more » « less