An official website of the United States government
Computer-based job interview training, including virtual reality (VR) simulations, have gained popularity in recent years to support and aid autistic individuals, who face significant challenges and barriers in finding and maintaining employment. Although popular, these training systems often fail to resemble the complexity and dynamism of the employment interview, as the dialogue management for the virtual conversation agent either relies on choosing from a menu of prespecified answers, or dialogue processing is based on keyword extraction from the transcribed speech of the interviewee, which depends on the interview script. We address this limitation through automated dialogue act classification via transfer learning. This allows for recognizing intent from user speech, independent of the domain of the interview. We also redress the lack of training data for a domain general job interview dialogue act classifier by providing an original dataset with responses to interview questions within a virtual job interview platform from 22 autistic participants. Participants’ responses to a customized interview script were transcribed to text and annotated according to a custom 13-class dialogue act scheme. The best classifier was a fine-tuned bidirectional encoder representations from transformers (BERT) model, with an f1-score of 87%.
more »
« less
HirePreter: A Framework for Providing Fine-grained Interpretation for Automated Job Interview Analysis
There has been a rise in automated technologies to screen potential job applicants through affective signals captured from video-based interviews. These tools can make the interview process scalable and objective, but they often provide little to no information of how the machine learning model is making crucial decisions that impacts the livelihood of thousands of people. We built an ensemble model – by combining Multiple-Instance-Learning and Language-Modeling based models – that can predict whether an interviewee should be hired or not. Using both model-specific and model-agnostic interpretation techniques, we can decipher the most informative time-segments and features driving the model's decision making. Our analysis also shows that our models are significantly impacted by the beginning and ending portions of the video. Our model achieves 75.3% accuracy in predicting whether an interviewee should be hired on the ETS Job Interview dataset. Our approach can be extended to interpret other video-based affective computing tasks like analyzing sentiment, measuring credibility, or coaching individuals to collaborate more effectively in a team.
more »
« less
- Award ID(s):
- 1750380
- PAR ID:
- 10329010
- Date Published:
- Journal Name:
- 2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)
- Page Range / eLocation ID:
- 1 to 5
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Antona, M; null (Ed.)Studies show that young autistic adults are under- or unemployed, with almost half never holding a paying job in their 20’s. Unemployment within this population leads to decreased personal growth and increased dependence on caregivers. Research suggests that the interview process is one of the largest barriers to employment for this population. Autistic individuals often struggle with emotion regulation, which can be exacerbated by the interview process. To address this, we propose the use of a stress detection model in conjunction with a virtual reality interview simulator. This combination will allow for the interview to adapt to the state of the participant to improve the skills and engagement of the user and positively influence their comfort level. Data regarding negative affective responses to categories of questions can also be used to inform employers on better interviewing techniques. A model was designed using data obtained from neurotypical participants completing a modified Computerized Paced Serial Addition Task (PASAT-C) and evaluated on a dataset obtained from Autistic participants who took part in a simulated interview. Agreement between the model and ground truth was compared based on Pearson correlation coefficients. It was found that was r(289) = 0.28, which was statistically significant (p < .001; CI: 0.17 to 0.38). Our preliminary results provide evidence for the validity of observer-based labeling of data captured using a wrist-worn physiological sensor.more » « less
-
This study is a work-in-progress investigating the experiences most salient to newly hired engineers in an electric power utility as they began new jobs. The study is based on an inductive, qualitative design using semi-structured interviews with 12 newly hired engineers. It was the process of integrating and developing the individual’s competencies to better match the requirements of the job (i.e., socialization or onboarding) that was an indicator of job performance—and the focus of this study. The characteristics of this competency matching during the first year of their new employment was related to the newly hired engineers’ education (for the new graduates), and job experiences (for experienced hires)--both acquired from their schooling and previous work. The interviews of newly hired engineers provided in-depth reports of their experiences developing and refining their technical competencies, as well as their professional competencies within the organization. These experiences clearly portray the complexities of how the newcomers worked out the requirements of integrating into the organization. The new hires reported that many of the requirements of the job were not clearly defined or presented; rather, they often learned through trial-and-error. Initial findings indicate that the learning experiences of these newcomers cluster around four content areas of learning aided by facilitating processes in the development of competence.The findings of this study corroborate and elaborate on previous work done on the preparation and transition of engineering graduates from school-to-work, and provide new insights into the process of integrating individual competencies into job requirements. The contribution of this work highlights how newcomers learned about their new jobs and what competencies they drew upon from their education, as well as how they applied their competencies to the practice of engineering. This included various types of formal, informal, incidental and social learning building upon their previous educational and work experiences, self-directed learning on the job, and the mentoring obtained from more experienced insiders. The implications of this work inform the development of professional engineers broadly for STEM careers, and specifically for the energy industry, which is part of the increasing interdisciplinarity and convergence of a wide range of technical fields.more » « less
-
This study is a work-in-progress investigating the experiences most salient to newly hired engineers in an electric power utility as they began new jobs. The study is based on an inductive, qualitative design using semi-structured interviews with 12 newly hired engineers. It was the process of integrating and developing the individual’s competencies to better match the requirements of the job (i.e., socialization or onboarding) that was an indicator of job performance—and the focus of this study. The characteristics of this competency matching during the first year of their new employment was related to the newly hired engineers’ education (for the new graduates), and job experiences (for experienced hires)--both acquired from their schooling and previous work. The interviews of newly hired engineers provided in-depth reports of their experiences developing and refining their technical competencies, as well as their professional competencies within the organization. These experiences clearly portray the complexities of how the newcomers worked out the requirements of integrating into the organization. The new hires reported that many of the requirements of the job were not clearly defined or presented; rather, they often learned through trial-and-error. Initial findings indicate that the learning experiences of these newcomers cluster around four content areas of learning aided by facilitating processes in the development of competence.The findings of this study corroborate and elaborate on previous work done on the preparation and transition of engineering graduates from school-to-work, and provide new insights into the process of integrating individual competencies into job requirements. The contribution of this work highlights how newcomers learned about their new jobs and what competencies they drew upon from their education, as well as how they applied their competencies to the practice of engineering. This included various types of formal, informal, incidental and social learning building upon their previous educational and work experiences, self-directed learning on the job, and the mentoring obtained from more experienced insiders. The implications of this work inform the development of professional engineers broadly for STEM careers, and specifically for the energy industry, which is part of the increasing interdisciplinarity and convergence of a wide range of technical fields.more » « less
-
When designing communicative visualizations, we often focus on goals that seek to convey patterns, relations, or comparisons (cognitive learning objectives). We pay less attention to affective intents–those that seek to influence or leverage the audience's opinions, attitudes, or values in some way. Affective objectives may range in outcomes from making the viewer care about the subject, strengthening a stance on an opinion, or leading them to take further action. Because such goals are often considered a violation of perceived ‘neutrality’ or are ‘political,’ designers may resist or be unable to describe these intents, let alone formalize them as learning objectives. While there are notable exceptions–such as advocacy visualizations or persuasive cartography–we find that visualization designers rarely acknowledge or formalize affective objectives. Through interviews with visualization designers, we expand on prior work on using learning objectives as a framework for describing and assessing communicative intent. Specifically, we extend and revise the framework to include a set of affective learning objectives. This structured taxonomy can help designers identify and declare their goals and compare and assess designs in a more principled way. Additionally, the taxonomy can enable external critique and analysis of visualizations. We illustrate the use of the taxonomy with a critical analysis of an affective visualization.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/ACIIW52867.2021.9666201
