An official website of the United States government
For two days in February 2018, 17 cybersecurity ed- ucators and professionals from government and in- dustry met in a “hackathon” to refine existing draft multiple-choice test items, and to create new ones, for a Cybersecurity Concept Inventory (CCI) and Cyber- security Curriculum Assessment (CCA) being devel- oped as part of the Cybersecurity Assessment Tools (CATS) Project. We report on the results of the CATS Hackathon, discussing the methods we used to develop test items, highlighting the evolution of a sample test item through this process, and offer- ing suggestions to others who may wish to organize similar hackathons. Each test item embodies a scenario, question stem, and five answer choices. During the Hackathon, par- ticipants organized into teams to (1) Generate new scenarios and question stems, (2) Extend CCI items into CCA items, and generate new answer choices for new scenarios and stems, and (3) Review and refine draft CCA test items. The CATS Project provides rigorous evidence- based instruments for assessing and evaluating educa- tional practices; these instruments can help identify pedagogies and content that are effective in teach- ing cybersecurity. The CCI measures how well stu- dents understand basic concepts in cybersecurity— especially adversarial thinking—after a first course in the field. The CCA measures how well students understand core concepts after completing a full cy- bersecurity curriculum.
more »
« less
Development of the Hogan Personality Content Single-Items Inventory
The Hogan Personality Inventory (HPI) and Hogan Developmental Survey (HDS) are among the most widely used and extensively well-validated personality inventories for organizational applications; however, they are rarely used in basic research. We describe the Hogan Personality Content Single-Items (HPCS) inventory, an inventory designed to measure the 74 content subscales of the HPI and HDS via a single-item each. We provide evidence of the reliability and validity of the HPCS, including item-level retest reliability estimates, both self-other agreement and other-other (or observer) agreement, convergent correlations with the corresponding scales from the full HPI/HDS instruments, and analyze how similarly the HPCS and full HPI/HDS instruments relate to other variables. We discuss situations where administering the HPCS may have certain advantages and disadvantages relative to the full HPI and HDS. We also discuss how the current findings contribute to an emerging picture of best practices for the development and use of inventories consisting of single-item scales.
more »
« less
- Award ID(s):
- 2121275
- PAR ID:
- 10473953
- Publisher / Repository:
- SAGE Publications
- Date Published:
- Journal Name:
- Assessment
- Volume:
- 31
- Issue:
- 6
- ISSN:
- 1073-1911
- Format(s):
- Medium: X Size: p. 1233-1261
- Size(s):
- p. 1233-1261
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Reliable simulations of molecules in condensed phase require the combination of an accurate quantum mechanical method for the core region, and a realistic model to describe the interaction with the environment. Additionally, this combination should not significantly increase the computational cost of the calculation compared to the corresponding in vacuo case. In this review, we describe the combination of methods based on coupled cluster (CC) theory with polarizable classical models for the environment. We use the polarizable continuum model (PCM) of solvation to discuss the equations, but we also show how the same theoretical framework can be extended to polarizable force fields. The theory is developed within the perturbation theory energy and singles‐T density (PTES) scheme, where the environmental response is computed with the CC single excitation amplitudes as an approximation for the full one‐particle reduced density. The CC‐PTES combination provides the best compromise between accuracy and computational effort for CC calculations in condensed phase, because it includes the response of the environment to the correlation density at the same computational cost of in vacuo CC. We discuss a number of numerical applications for ground and excited state properties, based on the implementation of CC‐PTES with single and double excitations (CCSD‐PTES), which show the reliability and computational efficiency of the method in reproducing experimental or full‐CC data. This article is characterized under:Electronic Structure Theory > Ab Initio Electronic Structure MethodsElectronic Structure Theory > Combined QM/MM MethodsSoftware > Quantum Chemistrymore » « less
-
Assessment instruments that generate quantitative data on attributes (cognitive, affective, behavioral,etc.) of participants are commonly used in the chemistry education community to draw conclusions in research studies or inform practice. Recently, articles and editorials have stressed the importance of providing evidence for the validity and reliability of data collected with these instruments following guidance from the Standards for Educational and Psychological Testing. This study examines how quantitative instruments have been used in the journalChemistry Education Research and Practice(CERP) from 2010–2021. Of the 369 unique researcher-developed instruments used during this time frame, the majority only appeared in a single publication (89.7%) and were rarely reused. Cognitive topics were the most common target of the instruments (56.6%). Validity and/or reliability evidence was provided in 64.4% of instances where instruments were used inCERPpublications. The most frequently reported evidence was single administration reliability (e.g., coefficient alpha), appearing in 47.9% of instances. Only 37.2% of instances reported evidence of both validity and reliability. These results indicate that, as a field, opportunities exist to increase the amount of validity and reliability evidence available for data collected with instruments and that reusing instruments may be one method of increasing this type of data quality evidence for instruments used by the chemistry education community.more » « less
-
Automated hiring systems are among the fastest-developing of all high-stakes AI systems. Among these are algorithmic personality tests that use insights from psychometric testing, and promise to surface personality traits indicative of future success based on job seekers' resumes or social media profiles. We interrogate the reliability of such systems using stability of the outputs they produce, noting that reliability is a necessary, but not a sufficient, condition for validity. We develop a methodology for an external audit of stability of algorithmic personality tests, and instantiate this methodology in an audit of two systems, Humantic AI and Crystal. Rather than challenging or affirming the assumptions made in psychometric testing --- that personality traits are meaningful and measurable constructs, and that they are indicative of future success on the job --- we frame our methodology around testing the underlying assumptions made by the vendors of the algorithmic personality tests themselves. In our audit of Humantic AI and Crystal, we find that both systems show substantial instability on key facets of measurement, and so cannot be considered valid testing instruments. For example, Crystal frequently computes different personality scores if the same resume is given in PDF vs. in raw text, violating the assumption that the output of an algorithmic personality test is stable across job-irrelevant input variations. Among other notable findings is evidence of persistent --- and often incorrect --- data linkage by Humantic AI. An open-source implementation of our auditing methodology, and of the audits of Humantic AI and Crystal, is available at https://github.com/DataResponsibly/hiring-stability-audit.more » « less
-
Jointly Making Inventory Stocking and Assortment Offering Decisions Making the product assortment offer decisions for a customer requires keeping a balance between offering an assortment that will satisfy the current customer and reserving the products with scarce inventories for the customers who will arrive in the future. Therefore, whereas the assortment offer decisions should depend on the current inventories of the products, the stocking decisions should anticipate how the offered assortments will deplete the inventories, thereby creating a natural interaction between inventory stocking and assortment offer decisions. In “Coordinated Inventory Stocking and Assortment Customization,” Bai, El Housni, Rusmevichientong, and Topaloglu develop models that jointly make the assortment offer and inventory stocking decisions. Their models allocate the limited stocking capacity to the inventories for different products, while taking into consideration the assortment offer decisions that will be made for the customers arriving over a selling horizon.more » « less
