INTRODUCTION

Statistics educators want their students to thrive in the data deluge and have continuously worked toward a better understanding of what causes students to persist in statistics. This study is expanding into the related field of data science, although research on data science education is in its infancy (Cassel et al., 2018). In statistics education, it is well-established that student attitudes toward statistics are an important factor in student learning (Bond et al., 2012;Budé et al., 2007;Schau & Emmioğlu, 2012). Research also shows that instructor attitudes influence student attitudes and outcomes (Lavy & Sand, 2015), but this relationship has not been studied in-depth specifically in statistics or data science. There is also a lack of research regarding how student and instructor attitudes interact with the learning environment to produce educational outcomes in statistics courses, and this type of work is almost non-existent so far in data science education. It is clear that attitudes matter in learning, but to what extent, and with what interactions, is unknown. To build a data-literate society with a workforce ready to tackle complex datasets, we must better understand the interrelationships of student attitudes, instructor attitudes, and the learning environment in both statistics and data science.

In recent decades, there have been renewed calls to develop better instruments to measure attitudes toward statistics (Gal & Ginsburg, 1994), including by researchers attending a retreat at the American Statistical Association, who identified the study of affective constructs as a priority research area in college-level statistics education (Pearl et al., 2012). The report from Pearl and colleagues identifies four priorities for the study of affective constructs, including how affective constructs can be measured, how affect relates to student success in statistics, and how to measure affective constructs of teachers to best understand the impact of teaching practices on student outcomes. The project discussed in this paper meets each of these objectives.

The calls to further study how to measure affect, more broadly referred to as attitudes, have been made even though some instruments designed to measure attitudes toward statistics already exist. The Survey of Attitudes Toward Statistics (SATS; Schau, 1992) is the most thoroughly validated and most commonly used, instrument for studying student statistics attitudes (Nolan et al., 2012). However, there are many documented challenges with the SATS, such as the rigid pre/post structure (Whitaker et al., 2022) and scales that exhibit a ceiling effect (Ramirez et al., 2012). SATS development was not guided by educational theory (Schau et al., 1995), making it difficult to align student responses with educational theory. Further, the factor structure of the SATS continues to be debated (Cashin & Elmore, 2005;Vanhoof et al., 2011). A new instrument measuring student attitudes toward statistics is needed.

Regarding data science, no validated instrument measuring either student or instructor attitudes toward data science currently exists. In statistics, the closest instrument matching this need is the Statistics Teaching Inventory (Zieffler et al., 2012), but it focuses on providing a snapshot of instructor practices in introductory statistics courses rather than attitudes toward teaching statistics. Environment inventories capturing modern statistics and data science classrooms must be established that align with any attitudinal instruments to develop this broad understanding of statistics and data science learning.

The Motivational Attitudes in Statistics and Data Science Education Research (MASDER) project was funded in 2020 by the United States National Science Foundation (DUE-2013392) to meet the need for validated attitudinal survey instruments. The goal of MASDER is to develop and validate six new instruments, three in statistics and three in data science (Table 1). In each field, the MASDER research team is developing a survey of student attitudes toward statistics (S-SOMAS; Student Survey of Motivational Attitudes toward Statistics) or data science (S-SOMADS); a survey of instructor attitudes toward teaching statistics (I-SOMAS; Instructor Survey of Motivational Attitudes toward Teaching Statistics) or data science (I-SOMADS); and learning environment inventories for both fields (E-SOMAS and E-SOMADS). The grant provides funding for data collection in the United States to build a representative sample of statistics and data science students and educators and supports the development of a website and infrastructure for public sharing of survey instruments, data, and findings. This is the first time a cohesive, synergistic set of instruments has been designed to work together to give broader understanding of the state of statistics and data science education. This paper (a) discusses the theoretical models guiding the development of the MASDER surveys, (b) gives details on the structure and intended uses of the surveys, and (c) discusses preliminary psychometric results from surveys developed thus far.

THEORETICAL MODELS

The design for three instruments within a given field was guided by the researchers' conceptualization of a meta-model (Figure 1), which describes how learning environments and instructors influence student motivation, which then influences student achievement. After literature review, the MASDER team found no established frameworks that appropriately modeled the connections between these areas. Areas have been studied individually or linked in pairs; for example, research shows that instructor attitudes influence student attitudes and outcomes (Lavy & Sand, 2015). However, a meta-model did not exist showing how all areas connect. MASDER therefore developed the meta-model based on the linkages that exist in the literature. The meta-model demonstrates the need to concurrently study the learning environment, instructors, and students to understand student achievement. The final outcome of student achievement is not necessarily grades. In future studies a measure of student conceptual understanding of statistics or data science such as the CAOS test (delMas, et al., 2007) may be used to assess achievement.

The MASDER team created more detailed models for each of the three instruments proposed in the meta-model. Student and instructor attitudinal models are based on Expectancy-Value Theory (EVT), a theory of motivation that explains students' achievement-related choices and behaviors (e.g., Eccles, 1983;Wigfield & Eccles, 2000). MASDER chose EVT as a model for attitudes because EVT has been in use as a framework for motivation for almost 40 years and is a widely adopted model for this purpose today (Wigfield & Eccles, 2020). While originally developed to model motivation of adolescents, EVT has been adopted for the study of adults as well (Wigfield & Eccles, 2020), and MASDER continues that tradition with the use of EVT for modeling college student and instructor motivation. The use of the phrase "motivational attitudes" in our SOMA survey titles intends to join common language from the fields of education research (motivation) and statistics education research (attitudes). Designed to mirror the original EVT model, Figure 2 displays the EVT model for the student surveys (S-SOMAS and S-SOMADS), and Figure 3 displays the parallel model for instructor surveys (I-SOMAS and I-SOMADS). Each blue or red cell, respectively, represents a construct intended to be measured by the survey; for further explanation of construct definitions, see Whitaker et al. (2018). The theoretical model for the environment inventories is shown in Figure 4, which was loosely based on the "framework for the analysis of teaching practices and beliefs" (Organisation for Economic Cooperation and Development, 2009) after no adequate existing model for the learning environment was found. The surveys are developed in a staggered process, rather than simultaneously developing all six instruments. This allows knowledge gained during the development of one survey to be used to accelerate the development of subsequent surveys. S-SOMAS was developed first because the most literature was available to inform survey development. It was followed by the development of S-SOMADS and I-SOMAS. I-SOMADS, E-SOMAS, and E-SOMADS are currently under development and will be administered in the 2022-23 academic year. Environment inventories follow a slightly different development pattern since items are not organized in scales intended to measure constructs.

DESCRIPTION OF SURVEY INSTRUMENTS

Student and instructor attitudinal surveys (S-SOMAS/DS and I-SOMAS/DS) are written to be answered using a seven-point Likert response scale. Multiple items are included for each construct that is intended to be measured according to the theoretical model. The goal is to have about four to six items per construct when the surveys are finalized (though this will depend on the psychometric properties of each scale). Table 2 displays several constructs from S-SOMAS along with an example item. E-SOMAS/DS items do not necessarily have a common response format; nor are they organized into scales that produce a score because items are more varied to capture various aspects of the learning environment. Each type of survey (S, I, and E) is custom designed for a specific need. S-SOMAS and S-SOMADS are intentionally designed to work longitudinally; rather than creating separate forms for pre/post data collection, the items are instead written to allow for longitudinal data collection that does not require students to be currently enrolled in a course (see Table 2 for examples). However, it is expected that many researchers and practitioners will use the instrument for pre/post course analysis, which is an appropriate use. I-SOMAS and I-SOMADS are specifically designed to measure attitudes toward teaching statistics or data science and are intended to be administered less frequently (perhaps once per year) than student surveys because instructor attitudes are not expected to change rapidly. Environment inventories measure institutional and course characteristics, the classroom learning and enacted classroom behaviors. Because most of these characteristics are specific to the class being taught, E-SOMAS and E-SOMADS are designed to be completed by the instructor for each individual course in which they are administering the student survey. All three data sources are intended to be combined to provide a rich picture of what motivates students to learn.

PSYCHOMETRIC PROPERTIES

The S-SOMAS survey is furthest along in the development process. A third pilot instrument containing 72 items measuring eight hypothesized constructs (see Figure 2) was administered to 2,707 college students in Spring 2022. Data was split into training and validation sets for analysis. Based on this pilot administration, subsets of items that are candidates to become the final instrument and exhibit strong psychometric properties have been identified. After also considering EFA and IRT results to identify problematic items, a CFA model with 38 items shows strong psychometric properties (CFI = 0.988, TLI = 0.986, RMSEA = 0.058, SRMR = 0.049) based on the criteria given by Hooper et al. (2008).

The I-SOMAS and S-SOMADS first pilot surveys were also administered in Spring 2022. I-SOMAS was completed by 168 statistics instructors. Due to the small sample size, results are preliminary. It is never expected that an initial pilot will result in perfect psychometric properties, hence the iterative development procedures. For I-SOMAS, based again on EFA, CFA, and IRT, the MASDER team found that a core set of items worked well for each hypothesized construct; some additional items should be written for most constructs to continue to refine the survey. The first S-SOMADS pilot was completed by 75 students. While not large enough to report full psychometric properties, the MASDER team is generally seeing core groups of items correlate for each hypothesized construct, which is a positive sign for continued administration and development. These surveys will continue to be administered in Fall 2022 to increase the sample size, allowing for more robust analysis.

CONCLUSION

The MASDER family of instruments will education researchers and practitioners to develop a richer understanding of influences on student learning in statistics and data science. Their development expands prior work that studied statistical affect and allows for novel research in the field of data science education. The research team looks forward to future collaborations to expand use of the instruments into other languages and countries. Further information is available at http://sdsattitudes.com/.