skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Evaluating the Performance of Topic Modeling Techniques with Human Validation to Support Qualitative Analysis
Examining the effectiveness of machine learning techniques in analyzing engineering students’ decision-making processes through topic modeling during simulation-based design tasks is crucial for advancing educational methods and tools. Thus, this study presents a comparative analysis of different supervised and unsupervised machine learning techniques for topic modeling, along with human validation. Hence, this manuscript contributes by evaluating the effectiveness of these techniques in identifying nuanced topics within the argumentation framework and improving computational methods for assessing students’ abilities and performance levels based on their informed decisions. This study examined the decision-making processes of engineering students as they participated in a simulation-based design challenge. During this task, students were prompted to use an argumentation framework to articulate their claims, evidence, and reasoning, by recording their informed design decisions in a design journal. This study combined qualitative and computational methods to analyze the students’ design journals and ensured the accuracy of the findings through the researchers’ review and interpretations of the results. Different machine learning models, including random forest, SVM, and K-nearest neighbors (KNNs), were tested for multilabel regression, using preprocessing techniques such as TF-IDF, GloVe, and BERT embeddings. Additionally, hyperparameter optimization and model interpretability were explored, along with models like RNNs with LSTM, XGBoost, and LightGBM. The results demonstrate that both supervised and unsupervised machine learning models effectively identified nuanced topics within the argumentation framework used during the design challenge of designing a zero-energy home for a Midwestern city using a CAD/CAE simulation platform. Notably, XGBoost exhibited superior predictive accuracy in estimating topic proportions, highlighting its potential for broader application in engineering education.  more » « less
Award ID(s):
2406698 2219271
PAR ID:
10604000
Author(s) / Creator(s):
; ; ; ;
Editor(s):
Leung, Carson
Publisher / Repository:
MDPI
Date Published:
Journal Name:
Big Data and Cognitive Computing
Volume:
8
Issue:
10
ISSN:
2504-2289
Page Range / eLocation ID:
132
Subject(s) / Keyword(s):
argumentation framework topic modeling machine learning qualitative analysis natural language processing
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; (, Design and Technology Education: An International Journal)
    This research investigates students’ argumentation quality in engineering design thinking. We implemented Learning by Evaluating (LbE) using Adaptive Comparative Judgment (ACJ), where students assess pairs of items to determine the superior one. In ACJ, students provided rationales for their critiques, explaining their selections. Fifteen students participated in an LbE exercise before starting their backpack design projects, critically evaluating multiple backpack designs and producing 145 comments. Writing comments required students to discern and justify the superior design, fostering informed judgment and articulation of their reasoning. The study used the Claim, Evidence, and Reasoning (CER) framework, adapted for engineering design thinking, to analyse these critiques. The framework emphasized three aspects: Empathy (understanding user needs), Ideation (deriving design inspiration), and Insight (gaining valuable understanding from evaluated designs). We employed both deductive and inductive content analysis to evaluate the argumentation quality in students’ critiques. High-quality argumentation was identified based on six codes: user-focused empathy, design inspirations, logical rationalizations, multi-criteria evaluations, aesthetic considerations, and cultural awareness. Poor-quality argumentation lacked these elements and was characterized by vagueness, uncertainty, brevity, inappropriateness, irrelevance, gender bias, and cultural stereotyping. By identifying critical elements of effective argumentation and common challenges students may face, this study aims to enhance argumentation skills in engineering design thinking at the secondary education level. These insights are intended to help educators prepare students for insightful and successful argumentation in engineering design projects. 
    more » « less
  2. ; (, 2021 ASEE Virtual Annual Conference)
    Generally, the focus of undergraduate engineering programs is on the development of technical skills and how they can be applied to design and problem solving. However, research has shown that there is also a need to expose students to business and society factors that influence design in context. This technical bias is reinforced by the available tools for use in engineering education, which are highly focused on ensuring technical feasibility, and a corresponding lack of tools for engineers to explore other design needs. One important contextual area is market systems, where design decisions are made while considering factors such as consumer choice, competitor behavior, and pricing. This study examines student learning throughout a third-year design course that emphasizes market-driven design through project-based activities and assignments, including a custom-made, interactive market simulation tool. To bridge the gap between market-driven design and engineering education research, this paper explores how students think about and internally organize design concepts before and after learning and practicing market-driven design approaches and tools in the context of an engineering design course. The central research question is: In what ways do student conceptions of product design change after introducing a market-driven design curriculum? In line with the constructivism framework of learning, it is expected that student conceptions of design should evolve to include more market considerations as they learn about and apply market-driven design concepts and techniques to their term projects. Four different types of data instruments are included in the analysis: Concept maps generated by the students before and after the course, open-ended written reflection assignments at various points in the semester, surveys administered after learning the market simulation tool and at the end of the course, and final project reports in which student teams listed their top 3-5 lessons learned in the course. Using the changes between the pre- and post-course concept maps as the primary metric to represent evolving design conceptions, data from the reflections, surveys, and reports are evaluated to assess their influence on such learning. Because market-driven design is a multi-faceted topic, a market-driven design is hierarchically decomposed into specific sub-topics for these evaluations. These include profitability (which itself encompasses pricing and costs), modeling and simulation, and market research (which encompasses consumers and competition). For each topic, correlation analyses are performed and regression models are fit to assess the significance of different factors on learning. The findings provide evidence regarding the effectiveness of the course’s market-driven design curriculum and activities on influencing student conceptions of design. 
    more » « less
  3. (, American Society of Engineering Education)
    Generally, the focus of undergraduate engineering programs is on the development of technical skills and how they can be applied to design and problem solving. However, research has shown that there is also a need to expose students to business and society factors that influence design in context. This technical bias is reinforced by the available tools for use in engineering education, which are highly focused on ensuring technical feasibility, and a corresponding lack of tools for engineers to explore other design needs. One important contextual area is market systems, where design decisions are made while considering factors such as consumer choice, competitor behavior, and pricing. This study examines student learning throughout a third-year design course that emphasizes market-driven design through project-based activities and assignments, including a custom-made, interactive market simulation tool. To bridge the gap between market-driven design and engineering education research, this paper explores how students think about and internally organize design concepts before and after learning and practicing market-driven design approaches and tools in the context of an engineering design course. The central research question is: In what ways do student conceptions of product design change after introducing a market-driven design curriculum? In line with the constructivism framework of learning, it is expected that student conceptions of design should evolve to include more market considerations as they learn about and apply market-driven design concepts and techniques to their term projects. Four different types of data instruments are included in the analysis: Concept maps generated by the students before and after the course, open-ended written reflection assignments at various points in the semester, surveys administered after learning the market simulation tool and at the end of the course, and final project reports in which student teams listed their top 3-5 lessons learned in the course. Using the changes between the pre- and post-course concept maps as the primary metric to represent evolving design conceptions, data from the reflections, surveys, and reports are evaluated to assess their influence on such learning. Because market-driven design is a multi-faceted topic, a market-driven design is hierarchically decomposed into specific sub-topics for these evaluations. These include profitability (which itself encompasses pricing and costs), modeling and simulation, and market research (which encompasses consumers and competition). For each topic, correlation analyses are performed and regression models are fit to assess the significance of different factors on learning. The findings provide evidence regarding the effectiveness of the course’s market-driven design curriculum and activities on influencing student conceptions of design. 
    more » « less
  4. ; ; ; ; ; ; ; ; ; ; et al (, Intelligencebased medicine)
    Background and Objectives: Sepsis is a leading cause of mortality in intensive care units (ICUs). The development of a robust prognostic model utilizing patients’ clinical data could significantly enhance clinicians’ ability to make informed treatment decisions, potentially improving outcomes for septic patients. This study aims to create a novel machine-learning framework for constructing prognostic tools capable of predicting patient survival or mortality outcome. Methods: A novel dataset is created using concatenated triples of static data, temporal data, and clinical outcomes to expand data size. This structured input trains five machine learning classifiers (KNN, Logistic Regression, SVM, RF, and XGBoost) with advanced feature engineering. Models are evaluated on an independent cohort using AUROC and a new metric, 𝛾, which incorporates the F1 score, to assess discriminative power and generalizability. Results: We developed five prognostic models using the concatenated triple dataset with 10 dynamic features from patient medical records. Our analysis shows that the Extreme Gradient Boosting (XGBoost) model (AUROC = 0.777, F1 score = 0.694) and the Random Forest (RF) model (AUROC = 0.769, F1 score = 0.647), when paired with an ensemble under-sampling strategy, outperform other models. The RF model improves AUROC by 6.66% and reduces overfitting by 54.96%, while the XGBoost model shows a 0.52% increase in AUROC and a 77.72% reduction in overfitting. These results highlight our framework’s ability to enhance predictive accuracy and generalizability, particularly in sepsis prognosis. Conclusion: This study presents a novel modeling framework for predicting treatment outcomes in septic patients, designed for small, imbalanced, and high-dimensional datasets. By using temporal feature encoding, advanced sampling, and dimension reduction techniques, our approach enhances standard classifier performance. The resulting models show improved accuracy with limited data, offering valuable prognostic tools for sepsis management. This framework demonstrates the potential of machine learning in small medical datasets. 
    more » « less
  5. ; ; (, Proceedings of the International Conference on Software Engineering)
    Mobile application (app) reviews contain valuable information for app developers. A plethora of supervised and unsupervised techniques have been proposed in the literature to synthesize useful user feedback from app reviews. However, traditional supervised classification algorithms require extensive manual effort to label ground truth data, while unsupervised text mining techniques, such as topic models, often produce suboptimal results due to the sparsity of useful information in the reviews. To overcome these limitations, in this paper, we propose a fully automatic and unsupervised approach for extracting useful information from mobile app reviews. The proposed approach is based on keyATM, a keyword-assisted approach for generating topic models. keyATM overcomes the problem of data sparsity by using seeding keywords extracted directly from the review corpus. These keywords are then used to generate meaningful domain-specific topics. Our approach is evaluated over two datasets of mobile app reviews sampled from the domains of Investing and Food Delivery apps. The results show that our approach produces significantly more coherent topics than traditional topic modeling techniques. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email