More Like this

1. Gaze and Head Rotation Analysis in a Triadic VR Job Interview Simulation

   https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00083  

   Artiran, Saygin ; Bedmutha, Poorva S. ; Li, Aaron ; Cosman, Pamela ( October 2023 , IEEE)
2. Dialogue Act Classification via Transfer Learning for Automated Labeling of Interviewee Responses in Virtual Reality Job Interview Training Platforms for Autistic Individuals

   https://doi.org/10.3390/signals4020019  

   Adiani, Deeksha ; Colopietro, Kelley ; Wade, Joshua ; Migovich, Miroslava ; Vogus, Timothy J. ; Sarkar, Nilanjan ( June 2023 , Signals)

   Computer-based job interview training, including virtual reality (VR) simulations, have gained popularity in recent years to support and aid autistic individuals, who face significant challenges and barriers in finding and maintaining employment. Although popular, these training systems often fail to resemble the complexity and dynamism of the employment interview, as the dialogue management for the virtual conversation agent either relies on choosing from a menu of prespecified answers, or dialogue processing is based on keyword extraction from the transcribed speech of the interviewee, which depends on the interview script. We address this limitation through automated dialogue act classification via transfer learning. This allows for recognizing intent from user speech, independent of the domain of the interview. We also redress the lack of training data for a domain general job interview dialogue act classifier by providing an original dataset with responses to interview questions within a virtual job interview platform from 22 autistic participants. Participants’ responses to a customized interview script were transcribed to text and annotated according to a custom 13-class dialogue act scheme. The best classifier was a fine-tuned bidirectional encoder representations from transformers (BERT) model, with an f1-score of 87%. 

   more » « less
3. HirePreter: A Framework for Providing Fine-grained Interpretation for Automated Job Interview Analysis

   https://doi.org/10.1109/ACIIW52867.2021.9666201  

   Rahman, Wasifur ; Mahbub, Sazan ; Salekin, Asif ; Hasan, Md Kamrul ; Hoque, Ehsan ( September 2021 , 2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW))

   There has been a rise in automated technologies to screen potential job applicants through affective signals captured from video-based interviews. These tools can make the interview process scalable and objective, but they often provide little to no information of how the machine learning model is making crucial decisions that impacts the livelihood of thousands of people. We built an ensemble model – by combining Multiple-Instance-Learning and Language-Modeling based models – that can predict whether an interviewee should be hired or not. Using both model-specific and model-agnostic interpretation techniques, we can decipher the most informative time-segments and features driving the model's decision making. Our analysis also shows that our models are significantly impacted by the beginning and ending portions of the video. Our model achieves 75.3% accuracy in predicting whether an interviewee should be hired on the ETS Job Interview dataset. Our approach can be extended to interpret other video-based affective computing tasks like analyzing sentiment, measuring credibility, or coaching individuals to collaborate more effectively in a team. 

   more » « less
4. 2D and 3D Visualization of Eye Gaze Patterns in a VR-Based Job Interview Simulator: Application in Educating Employers on the Gaze Patterns of Autistic Candidates

   https://doi.org/10.1007/978-3-030-78092-0_36  

   Breen, M ; McClarty, J ; Langley, C ; Farzidayeri, J ; Trevethan, K ; Swenson, B ; Sarkar, M ; Wade, J ; Sarkar, N. ( July 2021 , International Conference on Human-Computer Interaction (HCII))

   Antona, M ; null (Ed.)

   Employment of autistic individuals is strikingly low in relation to the skill level and capabilities of this population. Roughly 65% of autistic adults are either unemployed or underemployed relative to their abilities but there is increasing recognition that this number could be greatly improved through empowering autistic individuals while simultaneously providing a boost to the economy. Much of this disparity can be attributed in part to the lack of awareness and understanding among employers regarding behavior of autistic individuals during the hiring process. Most notably, the job interview—where strong eye contact is traditionally expected but can be extremely uncomfortable for autistic individuals—presents an unreasonable initial barrier to employment for many. The current work presents a data visualization dashboard that is populated with quantitative data (including eye tracking data) captured during simulated job interviews using a novel interview simulator called Career Interview Readiness in Virtual Reality (CIRVR). We conducted a brief series of case studies wherein autistic individuals who took part in a CIRVR interview and other key stakeholders provided lived experiences and qualitative insights into the most effective design and application of such data visualization dashboard. We conclude with a discussion of the role of information related to visual attention in job interviews with an emphasis on the importance of descriptive rather than prescriptive interpretation. 

   more » « less
5. Multimodal job interview simulator for training of autistic individuals

   https://doi.org/10.1080/10400435.2023.2188907  

   Adiani, Deeksha ; Breen, Michael ; Migovich, Miroslava ; Wade, Joshua ; Hunt, Spencer ; Tauseef, Mahrukh ; Khan, Nibraas ; Colopietro, Kelley ; Lanthier, Megan ; Swanson, Amy ; et al ( April 2023 , Assistive Technology)