Chemical processing companies are increasingly dedicated to process safety due to the significant number of process safety failures that occur each year. For example, an explosion followed by a chemical fire killed four employees while injuring 32 employees, and 28 members of the public in 2007 at T2 Laboratories Inc. The explosion was a result of a runaway exothermic reaction that was not a recognized hazard from T2. 1 This incident led to the addition in 2012 of "consideration of hazards associated with the engineering application of basic sciences" to the program criteria for Chemical Engineering of Criteria for Accrediting Engineering Programs (ABET). 2,3 This change emphasized the importance of process safety education in the training of chemical engineering students. 2 Institutions approached the addition of safety material differently, ranging from new courses both within and outside of the department, to spring break programs where students traveled to facilities to conduct their own hazard safety analyses. 2 Teaching process safety in the classroom is recognized as an important need worldwide. In 2009, University of Melbourne students were given a safety case study, and were required to investigate and present their findings in a presentation. They were also required to write a critique of another group's presentation. 4 The University of Melbourne also used "safety shares" to teach students about process safety. A safety share was a 2-4 minute discussion given at the beginning of every lecture in a second year engineering class. The shares would cover general safety advice, the importance of situational awareness, and case studies. 3 Results from the Melbourne studies have shown that students found these methods to be effective, and showed that students agreed that graduating engineers need a stronger appreciation for process safety. 3,4 To ensure that departments are meeting the ABET specified process safety outcomes, departments are encouraged to track metrics such as exam scores, or a sub-metric of the ABET criteria. 2 One of the issues is that many of these assessment methods are not able to be applied across institutions and have not been validated. In response to this, the authors have created the Engineering Process Safety Research Instrument (EPSRI) to measure process safety decision making. This paper describes the creation and content validation of the EPSRI.
The EPSRI was modeled after the Defining Issues Test version 2 (DIT2) and the Engineering Ethical Reasoning Instrument (EERI). 5,6 In its original design, the EPSRI has 8 dilemmas, three decisions, and between 12-15 additional considerations for each dilemma that students must rate based on relative importance to their decision making process. 5,7 The dilemmas developed in the EPSRI are based on case studies and investigations from process safety failures to provide a realistic context for the decision making process. An example of a dilemma will be discussed as part of the presentation at the conference. Each author was responsible for creating two dilemmas. These dilemmas were then reviewed by all authors for clarity, grammar and spelling.
The considerations provided are meant to reflect pre-conventional, conventional, and postconventional decision making as described by Kohlberg's Moral Development Theory. 8 This theory represents the "transformations that occur in a person's form or structure of thought," (pg. 54) and occurs through six stages. 8 The first two stages are considered pre-conventional, where "right and wrong" are interpreted in terms of consequences to oneself. Decisions are "selfishly" made based on physical consequence and satisfying one's needs. The next two stages are considered conventional. At this level, decision making revolves around conformity and loyalty to personal expectations, behavior which may please or help others and authority, fixed rules, and the maintenance of social order. The final two stages are categorized as post-conventional. People who operate at this level show a clear effort to define moral values and principles. Decisions are based on general individual rights, and self-chosen ethical principles. 8 Twelve considerations were initially developed for each dilemma and an additional three considerations were added during the review process to provide sufficient choices for review by content experts. It is planned to include one nonsense consideration in its final state. Rest et. al. described that meaningless items, or M-items, are used to detect unreliable data. 9 M-items are items on the DIT2 that are written with similar complexity and vocabulary to the other considerations but are not relevant to the dilemma in question. 9 The nonsense items on the EPSRI will serve as Mitems, and will be used to detect unreliable data from the data pool in the further validation study that will be conducted.
The validation of the EPSRI followed the content validation process outlined by Devellis. 10 The process involves having people knowledgeable in the area of interest, in this case process safety and engineering education, review the items to determine the relevance of the items to what is intended to be measured. 10 Professionals should also review the content of the scale to ensure that constructs represent their prescribed definitions. 10 Lastly, the scale should then be reviewed to ensure that no content areas have been omitted. 10 In the case of the EPSRI, these experts included professionals in the chemical process industry, chemical engineering faculty members, and engineering education researchers. Content review was done through a survey instrument that was prepared in Qualtrics. Due to the length of the original instrument, two surveys were prepared that covered half of the content found in the original instrument. This approach made the survey completion time more reasonable for the content experts. Proper human subject approval was obtained prior to conducting the study.
Content experts from the chemical industry and individuals holding chemical engineering faculty positions were asked to rate the dilemmas' relevance to real-life engineering process safety situations. Content experts could rate the dilemma as not relevant (1), moderately relevant (2), or very relevant (3). Content experts were also asked to provide feedback on dilemmas they rated as not relevant, or moderately relevant. Once the surveys were completed, the researchers averaged the scores to determine whether or not a dilemma should be omitted from the instrument. Dilemmas with an average score under a 2 were omitted, and dilemmas with an average score of a 2 or above were kept. Dilemmas rated above a 2 with no suggestions from content experts were left the same. Dilemmas that had suggestions provided were revised if all team members agreed that the change would yield an improvement to the clarity of the dilemma.
Considerations were reviewed by all content experts to determine how well they aligned with provided definitions of pre-conventional, conventional and post-conventional thinking. Content experts could rate responses as low (1), moderate (2), or high (3). Content experts were also asked to give feedback on considerations they rated as low or moderate, as well as provide considerations that may have not been included in the original design of the instrument. The average of these scores was taken, to determine whether the consideration should be eliminated, revised, or kept. Considerations rated below a 2 were eliminated. Considerations rated between 2.0 and 2.5 were eliminated if there were sufficient considerations relative to that construct (i.e. at least 3 considerations for each of pre-conventional, conventional and post-conventional thinking for each dilemma) and if no suggestions were given. However, if helpful suggestions were given on a consideration, it was revised and kept. Considerations rated above a 2.5 were kept, and were only revised if team members agreed that the suggestions given were beneficial in providing additional context to the dilemma in question.
Seven content experts took the first survey, of which 14% were from chemical industry, 43% were chemical engineering faculty members, and 43% were from a learning science or engineering education background. Five content experts completed the second survey, of which 20% were from chemical industry, 40% were chemical engineering faculty members, and 40% were from a learning science or engineering education background.
Table 1 has a listing of the dilemmas' individual ratings and average scores. Based upon the screening process outlined in the Methods, Dilemma 1 was eliminated, and the seven remaining dilemmas were kept. The initial development of the instrument included 8 dilemmas as the authors anticipated some of the dilemmas may need to be removed. This allowed for leeway during the validation process with the knowledge that the final DIT2 instrument only contained 5 dilemmas.
An example of a revision performed was the dilemma where an individual had to choose between an inexpensive pipe material that needs to be replaced every month versus an expensive pipe material that needs to be replaced once a year. Content experts felt this situation didn't Table 1. Individual and average scores of the eight dilemmas . represent much of a dilemma, and that they would most likely choose the cheaper option since no safety risks were stated. Revisions made to this dilemma included adding a statement that the two options operated similarly under normal conditions, but the additional maintenance that accompanies the cheaper pipe poses a higher safety risk which increases the perception of a dilemma within the statement.
Table 2 provides the results that were obtained on the consideration review from the content experts. Each number represents the average rating obtained from all content experts in their review of that consideration number. The average rating for pre-conventional, conventional, and post-conventional considerations across all dilemmas was 2.7, 2.8 and 2.8 respectively. Overall, only 23% of the pre-conventional considerations, 11% of the conventional considerations and 11% of the post-conventional considerations were rated below a 2.5.
Content experts were provided with between 12-15 considerations per dilemma for a total of 107 considerations with the knowledge that some considerations may be eliminated through the validation process. By the end of the validation process, 11 of the original considerations had been removed, and one consideration had been added. The final EPSRI survey instrument will only have 12 considerations as modeled in the DIT2 instrument design but the final modifications to the instrument will not be performed until the full validation study is conducted in Spring 2018.
An example of a revision made to a post-conventional consideration was from Dilemma 6 when the consideration asked "is it ever right to have to regularly override equipment manually?" Content experts believed that manual overrides were common in practice, and suggested it be rephrased to include manually overriding the equipment in deviation of established operational practices. All team members agreed that the change was useful, and the revision was made.
There were also cases where the research team disagreed with a suggestion provided by the content experts, such as when a pre-conventional consideration ("Would your co-workers lose confidence in your abilities if you asked for assistance?") was rated a 2.6. Content experts commented that "not all engineers are good with tools," and asked why someone would be worried to ask for assistance. A revision to this consideration was not made on the basis that students face this issue, and based on prior discussions with employees at chemical companies, junior engineers or recent hires not realizing when they need to ask for help is a problem. Therefore it was agreed upon by the team to leave this consideration in the instrument.
The content validation for this instrument was based on a small sample of content experts. Seven content experts completed the first survey which included Dilemmas 5 through 8, and five content experts completed the second survey which included Dilemmas 1 through 4. For this reason, the content validation of this instrument may not have been exhaustive but the authors feel it is representative of key stakeholders within the chemical education field.
The aim of this study was to validate the EPSRI which will be used as a process safety decision making evaluation instrument for chemical engineering students. The initial version of the EPSRI had eight dilemmas with 12-15 considerations each. The content validation was completed by content experts who were from the chemical industry, chemical engineering faculty, or had a learning science/engineering education background. The results from the content validation process led to one dilemma and 11 original considerations being removed from the instrument, as well as the addition of one consideration.
Moving forward, the authors plan to run a think-aloud protocol with chemical engineering students in spring of 2018. This process will allow the researchers to gain insight into how students respond to the instrument and whether any clarifications need to be made. A full validation study will be subsequently run with senior chemical engineering students at three institutions representing diverse contexts. This large sample field test will allow determination whether the survey has adequate statistical properties to be a reliable and valid tool. 11