An official website of the United States government
This study introduces the statistical theory of using the Standardized Root Mean Squared Error (SRMR) to test close fit in ordinal factor analysis. We also compare the accuracy of confidence intervals (CIs) and tests of close fit based on the Standardized Root Mean Squared Error (SRMR) with those obtained based on the Root Mean Squared Error of Approximation (RMSEA). We use Unweighted Least Squares (ULS) estimation with a mean and variance corrected test statistic. The current (biased) implementation for the RMSEA never rejects that a model fits closely when data are binary and almost invariably rejects the model in large samples if data consist of five categories. The unbiased RMSEA produces better rejection rates, but it is only accurate enough when the number of variables is small (e.g., p = 10) and the degree of misfit is small. In contrast, across all simulated conditions, the tests of close fit based on the SRMR yield acceptable type I error rates. SRMR tests of close fit are also more powerful than those using the unbiased RMSEA.
more »
« less
The Effect of Estimation Methods on SEM Fit Indices
We examined the effect of estimation methods, maximum likelihood (ML), unweighted least squares (ULS), and diagonally weighted least squares (DWLS), on three population SEM (structural equation modeling) fit indices: the root mean square error of approximation (RMSEA), the comparative fit index (CFI), and the standardized root mean square residual (SRMR). We considered different types and levels of misspecification in factor analysis models: misspecified dimensionality, omitting cross-loadings, and ignoring residual correlations. Estimation methods had substantial impacts on the RMSEA and CFI so that different cutoff values need to be employed for different estimators. In contrast, SRMR is robust to the method used to estimate the model parameters. The same criterion can be applied at the population level when using the SRMR to evaluate model fit, regardless of the choice of estimation method.
more »
« less
- Award ID(s):
- 1659936
- PAR ID:
- 10545810
- Publisher / Repository:
- SAGE Publications
- Date Published:
- Journal Name:
- Educational and Psychological Measurement
- Volume:
- 80
- Issue:
- 3
- ISSN:
- 0013-1644
- Format(s):
- Medium: X Size: p. 421-445
- Size(s):
- p. 421-445
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Ghenaiet, Adel (Ed.)In the design of a large deployable mesh reflector, high surface accuracy is one of ultimate goals since it directly determines overall performance of the reflector. Therefore, evaluation of surface accuracy is needed in many cases of design and analysis of large deployable mesh reflectors. The surface accuracy is usually specified as root-mean-square error, which measures deviation of a mesh geometry from a desired working surface. In this paper, methods of root-mean-square error calculation for large deployable mesh reflectors are reviewed. Concept of reflector gain, which describes reflector performance, and its relationship with the root-mean-square error is presented. Approaches to prediction or estimation of root-mean-square error in preliminary design of a large deployable mesh reflector are shown. Three methods of root-mean-square error calculation for large deployable mesh reflectors, namely, the nodal deviation root-mean-square error, the best-fit surface root-mean-square error, and the direct root-mean-square error, are presented. Concept of effective region is introduced. An adjusted calculation of root-mean-square error is suggested when the concept of effective region is involved. Finally, these reviewed methods of root-mean-square error calculation are applied to surface accuracy evaluation of a two-facet mesh geometry, a center-feed mesh reflector, and an offset-feed mesh reflector for demonstration and comparison.more » « less
-
This paper introduces the pilot implementation of the Evidence Based Personas survey instrument for assessing non-cognitive attributes of relevance from undergraduate students at different stages of their engineering degree for the purpose of informing proactive advising processes. The survey instrument was developed with two key objectives: first, to assess its potential for streamlining and shortening existing instruments, and second, to explore the possibility of consolidating items from different surveys that measure the same or closely related constructs. A proactive advising system is being developed that uses the Mediation Model of Research Experiences (MMRE) as a framework. Within this framework, participation in various educational activities is linked to increased Commitment to Engineering via three mediating parameters: Self-Efficacy, Teamwork/Leadership Self-Efficacy, and Engineering Identity. The existing, validated MMRE survey instrument was used as a starting point for development of the current instrument with a goal of streamlining / shortening the number of questions. Ultimately, we envision augmenting the shortened instrument with items related to broader non-cognitive and affective constructs from the SUCCESS instrument. Noting that both the MMRE and SUCCESS instruments include measures of Self-Efficacy and Engineering Identity, selected questions from both were included and compared. Data was collected from 395 total respondents, and subsequent data analysis was based on 337 valid participants. Factor Analysis techniques, both exploratory and confirmatory, were employed to uncover underlying or latent variables within the results, particularly in the areas of Self-Efficacy where the combined items of the SUCCESS instrument and the MMRE instrument were used. Cronbach’s alpha analysis was employed to assess the internal consistency of the survey instrument. The Teamwork, Engineering Identity, and Commitment to Engineering constructs all produced a Cronbach’s alpha value in excess of 0.80. The Self-Efficacy construct fell below the 0.80 threshold at 0.77 which is considered to be respectable but is indicative of some short comings compared to that of the other constructs. The results of the EFA four-factor pattern matrix show the SUCCESS instrument items breaking out into their own components while the MMRE items merge with some of the items from the Engineering Identity construct suggesting a distinction in the underlying concepts these items may be measuring. This finding is further supported in the CFA through an assessment of the Goodness of Fit (GFI), Tucker-Lewis Index (TLI), and Root Mean Square Error of Approximation (RMSEA) of these constructs. The initial groupings of the four constructs produced a robust CFI value of 0.853, robust TLI value of 0.838, and a robust RMSEA value of 0.075. Self-Efficacy is broken out into two sub-scales one defined by the three items from the SUCCESS instrument and the other defined by the four remaining items from the MMRE instrument. Engineering Identity was also broken into two sub-scales. The robust CFI and TLI report values of 0.928 and 0.919 respectively, and the robust RMSEA is reported to be 0.053. The findings of the factor analyses indicate that a shortened form of the MMRE survey instrument will provide reliable measures of the underlying constructs. Additionally, the results suggest that the self-efficacy as measured by items from the MMRE and from the SUCCESS instruments are related to two separate aspects of self-efficacy and do not load well into a single factor.more » « less
-
This study compares two missing data procedures in the context of ordinal factor analysis models: pairwise deletion (PD; the default setting in Mplus) and multiple imputation (MI). We examine which procedure demonstrates parameter estimates and model fit indices closer to those of complete data. The performance of PD and MI are compared under a wide range of conditions, including number of response categories, sample size, percent of missingness, and degree of model misfit. Results indicate that both PD and MI yield parameter estimates similar to those from analysis of complete data under conditions where the data are missing completely at random (MCAR). When the data are missing at random (MAR), PD parameter estimates are shown to be severely biased across parameter combinations in the study. When the percentage of missingness is less than 50%, MI yields parameter estimates that are similar to results from complete data. However, the fit indices (i.e., χ2, RMSEA, and WRMR) yield estimates that suggested a worse fit than results observed in complete data. We recommend that applied researchers use MI when fitting ordinal factor models with missing data. We further recommend interpreting model fit based on the TLI and CFI incremental fit indices.more » « less
-
The association between elevation (agro-climatic zones, ACZs) and the mean annual total rainfall (MATRF) is not straightforward in different parts of the world. This study sought to estimate the amount of MATRF across four elevation zones of Jema watershed, which is situated in the northwestern highlands of Ethiopia, by employing an appropriate interpolation method. The elevation of the watershed ranges from 1895 to 3518 m a.s.l. For the sake of this study, 34 sample MATRF data were extracted from satellite and nearby gauge stations that were recorded from 1983 to 2010. These data sources were reconstructed by International Research Institute for Climate and Society at Columbia University, USA, at a scale of 10 km by 10 km. An elevation data set generated from a digital elevation model with 30-m resolution (DEM 30 m) was considered as a covariable to estimate the MATRF. To identify the optimal interpolation model, mean errors were computed using cross-validation statistics. The root-mean-square error (RMSE) analysis showed that ordinary cokriging (OCK) was the most accurate model with a predictive power of 87.3%. The root-mean-square standardized (RMSSE) analysis showed that the best precision value (0.72) occurred in OCK. Stable and Gaussian trend lines together with local polynomial types of trend removal, and an elliptical neighborhood search function could perform best to maximize the accuracy and the precision of estimating MATRF. Elevation, as a covariable, enhanced the degree of accuracy and precision of estimation. The value of the trend line function (least square) between the MATRF and elevation was very weak (R2 = 0.07), whereas the value of trend line function (least square) between the MATRF and the longitude coordinates (east–west direction) was medium (R2 = 0.34). The estimated MATRF for the entire watershed under study ranged from 1228 to 1640 mm. To conclude, elevation could contribute to the estimation of the MATRF. The value of the MATRF showed a declining pattern from the lower to higher elevation areas of the watershed.more » « less
