Science teacher knowledge for effective teaching consists of multiple knowledge bases, one of which includes science content knowledge and pedagogical knowledge. With the inclusion of science and engineering practices into the national science education standards in the US, teachers’ content knowledge goes beyond subject matter knowledge and into the realm of how scientists use practices for scientific inquiry. This study compares two approaches to constructing and validating two different versions of a survey that aims to measure the construct of teachers’ knowledge of models and modeling in science teaching. In the first version, a 24-item Likert scale survey containing content and pedagogical knowledge items was found to lack the ability to distinguish different knowledge levels for respondents, and validation through factor analysis indicated content and pedagogical knowledge items could not be separated. Findings from the validation results of the first survey influenced revisions to the second version of the survey, a 25-item multiple-choice instrument. The second survey employed a competence model framework for models and modeling for item specifications, and results from exploratory factor analysis revealed this approach to assessing the construct to be more appropriate. Recommendations for teacher assessment of science practices using competence models and points to consider in survey design, including norm-referenced or criterion-referenced tests, are discussed.
more »
« less
Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures
Abstract Topic models, as developed in computer science, are effective tools for exploring and summarizing large document collections. When applied in social science research, however, they are commonly used for measurement, a task that requires careful validation to ensure that the model outputs actually capture the desired concept of interest. In this paper, we review current practices for topic validation in the field and show that extensive model validation is increasingly rare, or at least not systematically reported in papers and appendices. To supplement current practices, we refine an existing crowd-sourcing method by Chang and coauthors for validating topic quality and go on to create new procedures for validating conceptual labels provided by the researcher. We illustrate our method with an analysis of Facebook posts by U.S. Senators and provide software and guidance for researchers wishing to validate their own topic models. While tailored, case-specific validation exercises will always be best, we aim to improve standard practices by providing a general-purpose tool to validate topics as measures.
more »
« less
- Award ID(s):
- 1738288
- PAR ID:
- 10427560
- Date Published:
- Journal Name:
- Political Analysis
- Volume:
- 30
- Issue:
- 4
- ISSN:
- 1047-1987
- Page Range / eLocation ID:
- 570 to 589
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Habitat suitability models have been used for decades to develop spatially explicit predictions of landscape capacity to support populations of target species. As high-resolution remote sensing data are increasingly included in habitat suitability models that inform spatial conservation and restoration decisions, it is essential to validate model predictions with independent, quantitative data collected over sustained time frames. Here, we used data collected from 12 reefs over a 14 yr sampling period to validate a recently developed physical habitat suitability model for intertidal oyster reefs in coastal Virginia, USA. The model used intertidal elevation, water residence time, and fetch to predict the likelihood of suitable conditions for eastern oysters Crassostrea virginica across a coastal landscape, and remotely sensed elevation was the most restrictive parameter in the model. Model validation revealed that adult oyster biomass was on average 1.5 times greater on oyster reefs located in predicted ‘suitable’ habitat relative to reefs located in predicted ‘less suitable’ habitat over the 14 yr sampling period. By validating this model with long-term population data, we highlight the importance of elevation as a driver of sustained intertidal oyster success. These findings extend the validation of habitat suitability models by quantitatively supporting the inclusion of remotely sensed data in habitat suitability models for intertidal species. Our results suggest that future oyster restoration and aquaculture projects could enhance oyster biomass by using habitat suitability models to select optimal site locations.more » « less
-
null (Ed.)Although strategies exist to measure actors' efforts to set policy, media, and lawmaking agendas, political scientists lack a method for identifying and accurately measuring another form of agenda setting that lies under the surface anytime two people talk. Within interactions, such as debates, deliberations, and discussions, actors can set the agenda by shifting others' attention to their preferred topics. In this article, I use a topic model that locates where topic shifts occur within an interaction in order to measure the relative agenda-setting power of actors. Validation exercises show that the model accurately identifies topic shifts and infers coherent topics. Three empirical applications also validate the agenda-setting measure within different political settings: U.S. presidential debates, in-person deliberations, and online discussions. These applications show that successfully setting the agenda can shape an interaction's outcomes, demonstrating the importance of continued research on this form of agenda setting.more » « less
-
Visualization and topic modeling are widely used approaches for text analysis. Traditional visualization methods find low-dimensional representations of documents in the visualization space (typically 2D or 3D) that can be displayed using a scatterplot. In contrast, topic modeling aims to discover topics from text, but for visualization, one needs to perform a post-hoc embedding using dimensionality reduction methods. Recent approaches propose using a generative model to jointly find topics and visualization, allowing the semantics to be infused in the visualization space for a meaningful interpretation. A major challenge that prevents these methods from being used practically is the scalability of their inference algorithms. We present, to the best of our knowledge, the first fast Auto-Encoding Variational Bayes based inference method for jointly inferring topics and visualization. Since our method is black box, it can handle model changes efficiently with little mathematical rederivation effort. We demonstrate the efficiency and effectiveness of our method on real-world large datasets and compare it with existing baselines.",more » « less
-
null (Ed.)Visualization and topic modeling are widely used approaches for text analysis. Traditional visualization methods find low-dimensional representations of documents in the visualization space (typically 2D or 3D) that can be displayed using a scatterplot. In contrast, topic modeling aims to discover topics from text, but for visualization, one needs to perform a post-hoc embedding using dimensionality reduction methods. Recent approaches propose using a generative model to jointly find topics and visualization, allowing the semantics to be infused in the visualization space for a meaningful interpretation. A major challenge that prevents these methods from being used practically is the scalability of their inference algorithms. We present, to the best of our knowledge, the first fast Auto-Encoding Variational Bayes based inference method for jointly inferring topics and visualization. Since our method is black box, it can handle model changes efficiently with little mathematical rederivation effort. We demonstrate the efficiency and effectiveness of our method on real-world large datasets and compare it with existing baselines.more » « less
An official website of the United States government

