Which programmatic, educator, and setting characteristics are associated with better student learning outcomes in environmental education (EE)? Reviews of the last decade-plus of research in EE reveal that a large proportion of empirical studies have focused on summative evaluations of single programs in both EE and the larger informal science education field (NRC 2009;Ardoin, Biedenweg, and O'Connor 2015;Ardoin et al. 2018;Rickinson 2001;Stern, Powell, and Hill 2014). These studies have measured the influence of programs on multiple outcomes of interest and documented numerous positive influences on student learning and well-being. However, summative evaluations cannot identify the programmatic elements that are most responsible for reported outcomes. Empirical evidence of this nature can only emerge from either experimental or comparative work, where programs are the unit of analysis and different program characteristics are monitored and measured and linked with measured participant outcomes.
Unfortunately, a dearth of this type of study exists in both EE and the broader informal science education field (NRC 2009;Stern, Powell, and Hill 2014). So despite strong evidence that EE programs can achieve positive learning outcomes, research rarely focuses on which approaches are most consistently linked to better programmatic outcomes (Ardoin, Biedenweg, and O'Connor 2015;Bourke, Buskist, and Herron 2014;Stern, Powell, and Hill 2014).
To address this gap, we observed a sample of 299 day-long EE field trip programs occurring across the U.S.A. for youth grades 5-8 (ages 9 to 14) and tracked the extent of use and quality of implementation of 66 programmatic, educator, and setting characteristics. We measured a range of student outcomes immediately after the programs using a retrospective survey. This approach allowed us to begin to answer the following research question: Which characteristics and approaches are associated with better participant outcomes (i.e. learning, interest in learning, 21st century skills, meaning/self-identity, self-efficacy, place connection, environmental attitudes, environmental stewardship, collaboration, and school motivation)?
In the U.S., it is estimated that thousands of youth attend EE field trip programs with their schools annually (Thompson and Houseal 2020). EE field trip programs involve leaving the school grounds and traveling (Ardoin et al. 2018;Storksdieck 2006). These programs generally aim to enhance awareness and knowledge about the environment and its associated challenges as well as develop the skills, dispositions, and expertise to make informed decisions and take actions to address these challenges (e.g. Ardoin et al. 2018;UNESCO 1977). EE field trip programs also typically seek to meet educational standards (e.g. Powell et al. 2019), inspire place connection (Ardoin 2006;Gruenewald 2003), and improve social and emotional learning (Bowers et al. 2010;Garst, Browne, and Bialeschki 2011;Lerner et al. 2005). For this study, we developed and measured cross-cutting outcomes that reflect the broad goals of EE (10 subscales-See Powell et al. 2019 for more detail) and focused on students in grades 5-8 (ages 9-14), because research suggests that middle childhood is a period of rapid development of higher levels of moral reasoning (Kellert 2002;Eisenberg et al. 1987;Kohlberg 1971) and greater logical and abstract cognitive abilities (Kellert 2002;Piaget 1936). Thus, this age period is considered important for developing the higher-level skills needed to foster environmental literacy and develop a connection with nature (e.g. Kahn and Kellert 2002;Sobel 2008).
Although the content of EE programs can vary, a hallmark of EE is to provide immersive and direct contact with nature through hands-on and engaging techniques and approaches (e.g. Storksdieck 2006; North American Associate of Environmental Education (NAAEE) 2020). Many of the most promoted techniques and approaches are described in NAAEE's Guidelines for Excellence in Environmental Education (NAAEE 2017(NAAEE , 2020) ) and other recent studies and reviews (Stern, Powell, and Hill 2014;Sobel 2008;Krasny 2020). These characteristics can be organized into three broad categories: programmatic and pedagogical approaches; educator characteristics and practices; and setting characteristics.
Programmatic characteristics refer to how a program is structured and the types of activities undertaken. These techniques include specific pedagogies, such as investigation-focused, inquiry-based, issue-based, experiential, or place-based education (Woodhouse and Knapp 2000;Jose, Patrick, and Moseley 2017;Moseley et al. 2020;Gruenewald 2008;Stern, Powell, and Hill 2014;Stern and Powell 2020;NAAEE 2017NAAEE , 2020)). Other programmatic approaches reflect crosscutting effective communication practices, such as having an introduction, a clear theme or take-home message, transitions between programmatic elements, and a conclusion that provides opportunities for participants to reflect upon the take home message and their EE experience (e.g. Ham 1992Ham , 2013;;Stern and Powell 2013). Table 2 in the results section provides a description of 43 programmatic characteristics hypothesized to influence learning outcomes during EE field trips for adolescent youth.
Educator characteristics refer to attributes of the educator thought to enhance outcomes, as well as specific behaviors and ways environmental educators interact with their students to create the instructional environment. Attributes such as passion for the topic and program (Tilden 2009), sincere and authentic interactions with participants (Stern and Powell 2013), confidence, and apparent knowledge, all have the potential to impact outcomes in live educational programs (Stern and Powell 2013;Powell and Stern 2013). Providing emotional support to students and being responsive to participant needs have also been shown to enhance learning outcomes in formal classrooms (Hamre and Pianta 2005;Merritt et al. 2012;Pianta, La Paro, and Hamre 2008;Reyes et al. 2012;Rudasill, Gallagher, and White 2010) and more recently in informal and EE settings (e.g. O'Hare et al. 2020). Table 3 in the results section lists and describes 18 educator characteristics and behaviors hypothesized to influence learning outcomes during EE field trips for adolescent youth. We developed two indexes that characterize seven of the specifically observed characteristics (see Measurement), resulting in 13 variables to consider in relationship to outcomes.
Research also suggests that the educational environment, or the setting in which an informal or non-formal educational program takes place, may meaningfully influence participant outcomes (e.g. Kellert 2005;Powell et al. 2009;Archer and Wearing 2003). For example, the level of naturalness of a setting is thought to enhance mental and physical well-being (e.g. Kuo, Barnes, and Jordan 2019;Ryan et al. 2010) as well as enhance cognition and learning (Born et al. 2001;Kuo, Barnes, and Jordan 2019;Wells 2000;Wells and Evans 2003). The novelty of the setting, or the perception of something new, unique, or unfamiliar (Garst 2018), can inspire curiosity, learning, and collaborative and collective action (dale et al. 2020;deWitt and Storksdieck 2008;de Waal 2008;Keltner et al. 2014). The degree of beauty of the setting can also enhance creativity and imagination (Holton 1988), awareness of balance, symmetry, harmony and grace (e.g. Kellert 2008), motivation to participate in science (Chandrasekhar 1987), and connection to place (Gruenewald 2003(Gruenewald , 2008)). Similarly, the extent to which participants are physically immersed in a natural setting also appears theoretically important (e.g. Garst 2018;Kellert 2002Kellert , 2005)). Table 4 in the results section describes five setting characteristics hypothesized to influence learning outcomes during field trips for adolescent youth.
This study investigated EE field trip programs for adolescent youth (grades 5-8; ages 10-14) in the U. S. provided by 90 program providers in 24 states to examine 1) which programmatic, educator, and setting characteristics were used during these programs and 2) the relationship between programmatic, educator, and setting characteristics and student outcomes, which were assessed by surveying participating students immediately after the programs. Clemson University's Institutional Review Board reviewed all procedures described below prior to data collection and determined that procedures were Exempt. verbal informed consent was received from program providers, educators, and students prior to all data collection procedures per guidelines outlined under approved protocols (IRB00000481; FWA00004497).
Working with the North American Association of Environmental Education (NAAEE), the National Park Service (NPS), and the Association of Nature Center Administrators (ANCA), we identified over 300 potential organizations, including nature centers, botanical gardens, science museums, and national, state, and local parks, that appeared to offer single-day EE field trip programs for students in grades 5-8 (ages 10-14) across the United States during the period of research (i.e. January-June 2018).
We aimed to develop a sample that would represent a broad range of field trip programs in a wide variety of contexts that would be generally reflective of field trip programs across the United States. To do so, we relied on Ruggiero's (2016) evaluation of Environmental Literacy Plans in the US, which ranked states in terms of the status and quality of their statewide Environmental Literacy Plans, as a proxy for the general status of EE in each state. We divided the states into quartiles based on this evaluation and then systematically sought to observe programs of at least 10 program providers from states in each quartile (see dale et al. 2020 for more information). We then sought to maximize diversity in terms of both program types and socioeconomic context. Inclusion criteria for this study included: programs took place away from school; programs focused on EE (broadly defined); programs lasted a single day or less in duration; programs served grades 5-8; program providers expressed a willingness to participate, and program providers conducted multiple programs during the period of research.
After contacting each potential program provider, we identified clusters of willing providers in different regions of the country that offered single day field trip programs for youth in grades 5-8. Ultimately, we observed 345 programs provided by 90 unique organizations across 24 states and the district of Columbia: 18 providers from the first quartile, 39 providers from the second quartile, 19 providers from the third quartile, and 14 providers from the fourth quartile.
We developed and refined our observational techniques and data collection procedures based on prior research in formal education (e.g. Gage and Needels 1989;Pianta, La Paro, and Hamre 2008;Stronge, Ward, andGrant 2007, 2011), on interpretive programs in U.S. National Parks (Stern and Powell 2013), and at a residential EE center in Maryland, USA (Frensley, Stern, and Powell 2020;Frensley et al. 2022). We then undertook extensive pilot testing with the entire research team, comprised of eight field researchers and the principal investigators of the study. The research team observed multiple filmed programs followed by 17 in-person field trip programs for the target student population in diverse contexts. during these pilot studies, all members of the research team scored each program as individuals and then compared and discussed at length all discrepancies in scoring. This process was used iteratively to clarify and refine the operational definitions and/or measurement of each programmatic, educator, and setting characteristic under consideration. We used this process to develop consistent (within and across researchers), reliable (produces the same observational result under the same conditions within and across researchers), and valid (accurately measures the construct in question) scoring of all observations across the eight field researchers.
Four pairs of researchers visited and collected data at 345 EE field trip programs for 5th to 8th graders. A single pair of researchers visited each location for observation and data collection. Once onsite, the researchers introduced themselves at the outset of each program and then systematically monitored and recorded the extent and quality of programmatic, educator and setting characteristics on a predesigned observation sheet.
For the first two weeks of program observation, each pair of researchers observed programs together and completed initial scoring independently. Afterwards, each pair of researchers would discuss their observations and scores, which enabled each team to reach consensus on the measure of each indicator, ensuring reliability and consistency in scoring of observational variables. After roughly two weeks for each pair, discrepancies in scoring were rare. Researchers then began to occasionally observe programs individually at the same location. Throughout the 22-week field season, researchers periodically observed programs together again to ensure reliability and consistency in scoring of each variable. All team members (i.e. four pairs of researchers, PIs, and Co-PI) also met periodically online to clarify any questions about scoring. At three staggered points in time over the course of the study, the original pairs of researchers were purposefully intermingled to observe programs together to further enhance the reliability of observation measures.
Immediately following each program, all attending students were invited to complete a paper survey regarding their opinions of the program and its influence on them, which was used to assess the outcomes of the program. For all programs, we attempted a census of all eligible attendees. No time limit was given for the students to complete the survey. The average completion time was approximately 8 minutes. Overall, 5,317 surveys were collected from participants from 345 programs, and the average response rate was 81%.
data from the 5,317 surveys were entered into Microsoft Excel and then transferred to SPSS for screening and analysis. First, we dropped three programs (26 surveys) because response rates were below 50%. We then screened surveys for missing values and removed 210 surveys that were missing responses to more than 25% of the items. With these removals, one additional program dropped below a 50% response rate. It was removed entirely (8 additional surveys). We also screened for obvious patterns indicating invalid responses, such as no variability in answers, strings of consecutive numbers, or using one circle to indicate responses for multiple survey items. We identified and removed 94 surveys with these problems. One additional program dropped below 50% response rate following these removals. It was removed from the database along with seven additional surveys. data were then screened for multivariate outliers using Mahalanobis distance (MAH) (Tabachnick and Fidell 2018). A total of 563 respondents were removed for exceeding the criterion Mahalanobis distance value. Six more programs dropped below 50% response rate and as a result were removed from the database (dropping an additional 33 surveys). The resulting sample contained 4,376 individual surveys from 334 programs provided by 90 organizations in 24 states and Washington, dC.
Outcomes. Conducting a large-scale comparative study required a consistent tool for measuring outcomes across a wide range of programs that would be valid, reliable, and sensitive enough to vary depending on the quality of the programs. The process of developing this tool included 1) reviewing the literature, 2) involving stakeholders and program providers in workshops to define and refine crosscutting outcomes applicable to a range of EE field trip programs; 3) operationalizing the outcomes following recommended scale development procedures (e.g. devellis 2003), which included iterative stakeholder review to ensure external validity; and 4) conducting a series of pilot studies in diverse EE settings across the US to refine and cross-validate the scales using confirmatory factor analyses and multi-group invariance testing procedures (see Powell et al. 2019 for full description). This work resulted in a scale comprised of ten related outcomes, including Place Connection, Learning, Interest in Learning, 21st Century Skills, Meaning/Self-Identity, Self-Efficacy, Environmental Attitudes, Environmental Stewardship, Collaboration, and School Motivation (see Table 1). All outcomes were measured with multiple items that were measured on a scale of 0-10. Self-Efficacy and Environmental Attitudes were measured using retrospective pre/post questions asking students to reflect on how they felt about given statements before the program, and after as a result of the experience. The mean scores reported for these items represent the differences between pre and post scores. The overall outcome measure, EE21 (short for Environmental Education Outcomes for the 21st Century), is a single composite measure representing the mean of each subscale equally weighted and aggregated to the program level (see results Table 1). The EE21 composite index has been statistically validated through confirmatory factor analysis (Powell et al. 2019).
Because our research seeks to identify which programmatic approaches and characteristics positively influence EE21 irrespective of age and race, and because previous research (e.g. Browning and Rigolon 2019; Browning and Locke 2020) and our own analyses (Stern, Powell, and Frensley 2022) have demonstrated that grade level and race can have strong influences on learning outcomes, we controlled for their influence by group mean centering the EE21 scores (Tabachnick and Fidell 2018). This procedure resulted in us dropping 27 programs from the sample because participants were enrolled in multiple grades. We eliminated another eight programs from analysis because of missing data pertaining to race, which was based on both self-report from students and school data pertaining to racial majority (see Stern, Powell, and Frensley 2022). This resulted in a final sample of 299 programs included in these analyses.
Programmatic, educator and setting characteristics. At each EE field trip program, we recorded observations of 66 different programmatic/pedagogical, educator, and setting characteristics (see results Tables 234). Fifty-five constructs were scored on a 1-to-4 scale, two constructs on a 1-to-3 scale, and seven constructs with binary measurement (presence/absence); two were continuous variables. The 1-to-4 scales followed the logic of calibration, as discussed by Ragin (2008) and used in prior research (e.g. Stronge, Ward, andGrant 2007, 2011;Stern and Powell 2013;Frensley et al. 2022): 1 represented total absence; 2 represented minor presence; 3 represented moderate presence; and 4 represented that the characteristic was a dominant aspect of the program. Extensive pilot testing with the full research team revealed that these scales enabled easy categorization, especially for more ambiguous cases, by considering whether the observed program more or less reflected the characteristic in question (the difference between a 2 and 3 on the scale). It also maximized scale size, which is desirable for detecting meaningful differences between programs and their characteristics. Threepoint scales were used for constructs with less variability. One three-point scale measured staging, or the general state of the group upon arrival to the field trip. In this case, the scale represented (3) well-organized; (2) moderately organized (some chaos or confusion); (1) disorganized, frenzied, late, or generally negative. In the other case, the three-point scale represented the extent to which the educator advocated for a specific viewpoint or action. In this case, the scale reflected (3) advocacy was clearly present;
(2) vague or implied advocacy, unclear; and (1) not present. Binary variables indicate the simple presence of absence of a characteristic (see results Tables 2 and3).
Based on our prior analyses (O'Hare et al. 2020;Stern and Powell 2013) and theory (e.g. Pianta and Hamre 2009), we conducted exploratory factor analyses and reliability analyses on the educator characteristics prior to further analyses. We did not conduct confirmatory factor in this case because educator and program characteristics are formative variables that were observed and represent a specific practice or attribute that is thought to directly influence a dependent variable. This is opposed to reflective indicators, which are thought to represent a broader concept and are not directly observed (see Kline 2005;Jarvis et al. 2003;Podsakoff et al. 2003, for further explanation). Exploratory factor analyses and reliability analyses on program level data revealed the presence of two latent educator characteristics. We have named the two resulting educator factors 'clarity and comfort' and 'emotional support' leaving us with 61 variables to carry forward in subsequent analyses (see results Table 3).
To describe the programmatic, educator, and setting characteristics of our sample, we conducted frequency and central tendency analyses. Eleven variables demonstrated little to no variability and were thus dropped from further analysis (see results Table 2). This resulted in 50 characteristics included in our final analyses. To answer our research question, we conducted bivariate Pearson correlation analyses to explore linear relationships between each of the programmatic, educator, and setting characteristics and group-mean-centered (GMC) EE21. We conducted independent samples t-tests on the binary observational items to assess if their presence or absence was related to student learning outcomes. To account for the potential of Type 1 errors, we used a Bonferroni correction to identify the strongest statistically significant relationships (in this case, p ≤ .001 for 50 distinct analyses).
To further examine the characteristics associated with better learning outcomes, we compared programmatic, educator, and setting characteristics of the top performing quartile of programs versus the lowest performing quartile using independent samples t-tests and chi-squared tests. To account for the potential of Type 1 error, we again used a Bonferroni correction (p ≤ .001). We also computed Cohen's d effect sizes for the t-tests, which assess the meaningfulness each statistically significant difference between groups (Tabachnick and Fidell 2018). Cohen's d scores below 0.2 can be considered spurious, above 0.2 are considered small, those approaching 0.5 demonstrate a medium effect size, and those nearing or above 0.8 are considered as large effect sizes (Cohen 1992). We calculated phi as the effect size for chi-squared tests. Phi scores below 0.1 are negligible; up to 0.3 are small; between 0.3 and 0.5 are medium; and 0.5 and greater are large (Tabachnick and Fidell 2018).
Finally, we took the programmatic, educator, and setting characteristics that were significantly correlated at the Bonferroni correction threshold and conducted relative weight analysis (RWA) using RWA-Web (Tonidandel and LeBreton 2015), a form of dominance analysis, to examine their relative weight or influence on GMC EE21. RWA iteratively assesses the contribution of an independent variable (in this case, programmatic, educator, and setting characteristics) to the prediction of a dependent variable (in this case, GMC EE21) by itself and in combination with other predictor variables (Budescu 1993;Budescu and Azen 2004;Tonidandel andLeBreton 2011, 2015;Tonidandel, LeBreton, and Johnson 2009). This analysis is warranted when there are large numbers of independent variables that are correlated with each other (Nathans, Oswald, and Nimon 2012;Tonidandel and LeBreton 2011) and provides the relative strength or 'dominance' of an independent variable, as compared with the other independent variables across all potential combinations (Budescu 1993;Budescu and Azen 2004;Tonidandel andLeBreton 2011, 2014). We conducted RWA because many of the 11 predictor variables were correlated with each other, and because RWA provides not only relative weights, but also 95% confidence intervals for the individual relative weights (Johnson 2004) and corresponding significance tests based on bootstrapping with 10,000 iterations (Tonidandel, LeBreton, and Johnson 2009). The RWA also provides a measure of the overall variance in GMC EE21 explained by the dominant predictor variables.
All descriptive statistics reported are calculated from the 299 programs validated by data cleaning procedures and group-mean centering of the EE21 dependent variable. Forty-three percent of these programs served 5th graders, 32% served 6th graders, 19% served 7th graders, and 6% served 8th graders. Of these programs, 46% were composed of a majority of students who identified as White, 32% were composed of a majority of students who identified as Hispanic, 13% of programs had no racial majority, and 9% of programs were composed of a majority of students who identified as Black. Free and reduced lunch statistics were available for 275 of the visiting school groups. The proportion of students eligible for free or reduced price lunches ranged from 2% to 100%, with a mean of 57%, similar to the national average of 58% in 2018 (USdA Food and Nutrition Service 2020).
The mean program time was 186.6 minutes, with a standard deviation of 69.9 minutes. The mean group size was 16 students with a standard deviation of 7.3. This number reflects the group size that participated in a program with a unique educator or educators. Many times school groups were broken up into smaller and more manageable sub-groups for programs with a primary instructor. visiting school teachers were passive observers in 43% of programs. In the other 57%, they played a more active role by participating along with students, providing disciplinary support, and/or co-leading programs. The on-site environmental educators at each location were also asked prior to their programs to identify their desired outcome goals for participants: 68% sought to influence knowledge, 10% skills, 25% interest in learning, 34% attitudes, 34% place attachment, and 24% desired to influence behaviors.
Table 1 displays the mean and standard deviation for each item that composes EE21 as well as the grand mean and standard deviation for the EE21 composite scale from the 299 programs prior to group mean centering to control for the influence of grade and race (Table 1).
Programmatic characteristics:. To examine which techniques and approaches were used across the 299 EE field trip programs, we report the mean score and the frequency of use of the different program, educator, and setting characteristics. For 11 characteristics, one-half of the scale (2 points on the observation scale) contained less than 5% of the observations, indicating that we either nearly always or never observed that construct's presence across the sample (Table 2). due to their low variance, we removed them from further analysis. The programmatic approaches that were rarely observed included: individual and group reflection, group discussion, presenting multiple view-points, teaching 21st Century Skills, role playing, storytelling, and the experiential learning cycle. Other results of note include that observed programs were commonly primarily focused on conveying factual information (approximately 78% were moderate-to-dominantly fact-focused), provided few opportunities for free exploration (15% of programs provided moderate or high levels of free exploration), omitted a clear theme/message (34% provided a clear theme or message), and often lacked a conclusion (just over 50% had a conclusion). Approximately 55% of programs were largely place-based.
Educator characteristics. The majority of educators associated with these programs were highly responsive (83% scored a 3 or 4), were knowledgeable and comfortable leading the programs (over 94% scored a 3 or 4), provided clear instructions and content delivery (over 86% scored a 3 or 4), and provided substantial emotional support (Table 3).
Programs occurred mostly outside (84% scored a 3 or 4) with over half occurring in natural (60% scored a 3 or 4) and aesthetically pleasing settings (55% scored a 3 or 4). The results also suggest that approximately 70% of programs provided relatively low levels of physical immersion in the environment (Table 4).
To determine the relationship between each of the programmatic, educator, and setting characteristics and GMC EE21, we conducted bivariate Pearson correlation analyses. Additionally, we conducted independent samples t-tests on the binary observational items to assess if their presence or absence contributed to student learning outcomes. To account for the potential of Type 1 error, we used a Bonferroni correction to identify significant relationships (in this case p ≤ .001) in all analyses. While 15 programmatic characteristics were statistically correlated with EE21 (p ≤ .05), only six met the Bonferroni correction threshold (Table 5). These included group size (smaller groups are associated with better outcomes), place-based programs, higher levels of verbal engagement, high quality questions that led to provocation, the use of transitions between program elements, and the organization of the group upon arrival (staging). Three educator characteristics passed the Bonferroni correction threshold (p ≤ .001) (Table 5). These included high levels of responsiveness to students needs and inquiries; the comfort and clarity index, which includes clear delivery of instructions and program content as well as the educator's apparent confidence and comfort leading the program; and the emotional support index, which includes providing sincere, positive, and reassuring communications and affinity seeking behaviors. Two setting characteristics, naturalness and the novelty of the setting, were significantly correlated with EE21 at the Bonferroni correction threshold (p ≤ .001) (Table 5). None of the binary observational variables met the Bonferroni correction threshold (p ≤ .001) although overly slow-paced programs performed significantly worse than other programs (Table 2), and educators that were 'walking encyclopedias' performed worse than other educators (p = .009) (Table 3).
To explore whether there are distinguishing characteristics and approaches between top and bottom performing programs, we compared the programmatic, educator and setting characteristics associated with the highest performing quartile of programs vs. the lowest performing quartile. We first divided the 299 programs into quartiles based on their group mean centered EE21 score. Tables 6-9 share all statistically significant differences.
The highest performing quartile of programs had significantly smaller group sizes, a greater degree of place-based programming, and higher levels of verbal engagement than the lowest performing quartile of programs (Table 6). The use of transitions, lecture-based (negative), class management, and providing a conclusion were significantly different with medium effect sizes, although these variables did not pass the Bonferroni correction (Table 6). Slow-paced programs were also more common in the lowest performing quartile (p < .001), although the effect size was small in this respect (Table 7).
Educators associated with the highest performing quartile of programs were significantly more responsive, demonstrated higher levels of comfort and clarity, and provided greater emotional support than educators associated with the lowest performing quartile (Table 8). Moreover, programs with walking-encyclopedia-style educators were more commonly in the lowest performing quartile (p < .001) although the effect size is small (Table 7). The settings associated with the highest performing programs were significantly more natural and novel than the settings associated with the lowest performing quartile of programs (Table 9).
To understand the proportion of the total variance in GMC EE21 explained by the 11 programmatic, educator, and setting characteristics (Table 5) that were most powerfully related to GMC EE21 (p ≤ 0.001), we conducted a special type of multiple regression known as relative weight analysis (RWA) (Johnson 2000) using RWA-Web (Tonidandel and LeBreton 2015). Because RWA is a type of regression that tests linear relationships with a continuous dependent variable, we selected the eleven variables with the strongest bivariate relationships with GMC EE21 (rather than the top vs. bottom quartile comparisons). RWA also examines the relative importance of each predictor within the regression. The results indicated that a weighted linear combination of the 11 most predictive programmatic, educator, and setting variables explained 18% of the variance in EE21 (Table 10.). An examination of the relative weights revealed that group size, educator comfort and clarity, responsiveness, naturalness of setting, and novelty of setting explained a statistically unique and significant amount of variance in GMC EE21 (Table 10.). RWA identifies the incremental contribution of each variable in the presence of other correlated predictors (Tonidandel and LeBreton 2011). The relative weights (RW) suggest that group size (.044), naturalness of the setting (.024), novelty of the setting (.022), responsiveness (.0154), and educator comfort and clarity (.153) appear to explain unique incremental variance in GMC EE21, while the other six predictor variables do not explain unique variance because of their correlations with the other predictors (Table 10) (Tonidandel andLeBreton 2011, 2015).
Our study attempted to identify the techniques and approaches that most consistently lead to better outcomes in 299 diverse 5th-8th grade field trip programs across the United States. We first examined the relationship between different programmatic, educator, and setting characteristics with student outcomes. Second, we examined which of these characteristics best distinguished between the programs yielding the most positive and least positive student outcomes. Finally, we conducted RWA to determine the proportion of variance in the overall GMC EE21 outcome measure that could be explained by the characteristics with strongest predictive ability in our sample. We highlight key lessons that emerged from our observations and these statistical explorations below.
Our initial descriptive results suggest that EE field trip programs largely appear to be taught by educators with high levels of confidence in performing their duties and high degrees of emotional support for their students. The results also indicate that many of the most commonly promoted techniques in EE (see NAAEE 2017NAAEE , 2020) ) were rarely encountered in our sample of 299 single-day field trip programs, including individual and group reflection, incorporating multiple viewpoints, group discussion, and teaching 21st century skills. One potential explanation for this is that these field trip programs were focused on meeting educational standards, which can come at the expense of using some of the techniques and pedagogies promoted by the field. Some argue, including Gruenewald and Manteaw (2007) and Stevenson (2007), that the influence of 'No Child Left Behind Act' and the focus on meeting educational standards has eroded the practice of EE in the U.S. and its mission to enhance environmental literacy and ); ci-l = lower bound of confidence interval used to test the statistical significance of rW; ci-u = upper bound of confidence interval used to test the statistical significance of rW; rs-rW = relative weight rescaled as a percentage of predicted variance in the criterion variable attributed to each predictor (tonidandel and leBreton 2015). a significant at 95% (ci did not contain 0).
address key environmental issues. Research on teacher motivations for attending field trip programs supports this argument; most teachers desire to link activities to curriculum standards (e.g. Kisiel 2005;Storksdieck 2006). Our results also suggest that many EE field trips in the U.S. are strongly influenced by formal science education practices and some observed programs performed activities that were roughly equivalent to classroom science lab assignments, rather than providing fully immersive, place-based experiences espoused by EE experts and practitioners (Krasny 2020;Stern, Powell, and Hill 2014;Sobel 2008;NAAEE 2020). Our results suggest that these programs can still achieve positive learning outcomes. However, certain practices, as expected, were linked with better outcomes than others.
The series of complementary tests we conducted identified 11 programmatic, educator, and setting characteristics that were most powerfully and consistently associated with positive learning outcomes (Table 5). These characteristics were also highly related to each other. For example, educators that provide high degrees of responsiveness also tended to provide high levels of emotional support (Table 5). The RWA revealed that five of these 11 variables accounted for unique incremental variance; the other six did not account for additional unique variance and were largely accounted for by these five. Overall, the RWA suggests that these variables explain 18% of the variance in learning outcomes, while controlling for grade and race (GMC EE21).
In this study, smaller groups exhibited more positive outcomes. Prior research has commonly found that smaller class sizes are associated with increased academic achievement in formal education (Shin and Chung 2009;Bosworth 2014;Chingos and Whitehurst 2011). However, research on group size has been limited in the case of EE field trips (Bitgood 1989;Stern, Powell, and Ardoin 2008) and similar experiences (Powell, Kellert, and Ham 2009;Stern and Powell 2013) and findings have been less consistent. We know of no study that has examined the influence of group size on single-day EE field trips on student outcomes. Our study suggests that breaking larger groups into smaller, more intimate groups are likely beneficial for single-day EE field trip programs. Further explorations of our data did not yield an optimal group size. Rather, we found a roughly linear relationship.
Our results also suggest programs that occurred in more natural and more novel natural settings were associated with more positive learning outcomes. Natural settings can range from urban gardens to wilderness areas (e.g. dale et al. 2020). Novel natural settings may include unique natural phenomena or viewing and interacting with wildlife, particularly unique and charismatic fauna not commonly observed by the program participants otherwise (e.g. Skibins, Powell, and Hallo 2013). These results mirror prior research in formal (e.g. Kuo, Barnes, and Jordan 2019) and informal education (e.g. Browning and Rigolon 2019) and support widely held assumptions in the field of the importance of connecting participants with nature. However, these results do not suggest that programs in more urban locations cannot be effective. Rather, whether in an urban or rural environment, educators should consider how to incorporate natural settings and novelty in their programs to the greatest extent possible.
Related to these findings, our results also suggest that programs that focused on the unique natural, cultural, and social attributes of the location in their programming were associated with more positive learning outcomes. Highlighting the unique components of a location is a hallmark of place-based education and highly promoted in EE (NAAEE 2020;Krasny 2020;Stern, Powell, and Hill 2014;Sobel 2008;Gruenewald 2003Gruenewald , 2008)). Many programs in our sample largely ignored the unique attributes of the place and instead provided content and activities that could occur in any location. These programs that did not or minimally highlighted the attributes of the location, on average, yielded less positive student outcomes.
Our results also highlighted the important role that the educator plays in delivering a successful EE program. Specifically, our study found that educators that (1) clearly communicated both instructions and content to their students, (2) exuded comfort, confidence, and a good working knowledge of the location and content, and (3) created an emotionally supportive and responsive class environment produced more positive student outcomes. Research in both formal education (e.g. Stronge, Ward, andGrant 2007, 2011) as well as more informal settings such as interpretation in parks have found similar results (Stern and Powell 2013). In particular, acknowledging student success; affinity seeking behaviors, such as providing eye-contact, smiling, and listening; providing positive feedback; and responding to students' questions, interests, and non-verbal cues all build a positive learning environment and trust between students and an educator (e.g. Pianta, La Paro, and Hamre 2008, Pianta and Hamre;Rudasill, Gallagher, and White). Formal education research has extensively explored the importance of these relational and positive learning environment skills for teachers (e.g. Pianta and Hamre 2009;Rudasill, Gallagher, and White 2010;Reyes et al. 2012) and our work further reinforces this importance in more non-formal educational settings.
The results of this study also suggest the importance of using quality transitions to connect different concepts, programmatic elements/activities, and locations, as well as providing a conclusion at the end of the program that reinforces the take-home message. In the context of EE field trip programs, transitions can be used to maintain cognitive engagement by connecting content and concepts, by linking locations (e.g. 'students, I want you to pay attention to how the landscape changes as we walk to our next location. I'm going to ask you about what you notice when we arrive'), or by connecting different activities to help students to draw larger lessons. Providing a well-crafted conclusion at the end of a field trip program serves as a way to summarize the key themes and the learning that occurred, as well as offer opportunities for students to reflect upon the experience (Ham 2013(Ham , 1992)). Research in interpretation (Ham 2013;Powell and Stern 2013;Stern and Powell 2013) reinforces these results and speaks to the need to plan, organize, and deliver day-long EE field trip programs as a cohesive, well-organized, and integrated whole.
Our research suggests that verbal engagement and high-quality questions, in particular, also enhance student outcomes. Prior research and theory outline what constitutes a 'high quality' question: it is one that cannot be answered with yes or no; challenges students to apply new information: explores relationships and cause and effect; draw comparisons; elicits opinions and evaluation; focuses attention; and/or drives inference or problem solving (Ham 1992;NPS 2019). Our results suggest that any verbal engagement can enhance program outcomes for students, but high-quality questions during an EE field trip program raises the potential to increase cognitive engagement, critical thinking, and meta-cognition in students.
The state in which students arrive at a field trip program (staging) also influences learning outcomes. did the students arrive disorganized, late, or confused or were they well-prepared and ready to participate in the field trip? While being late can often be outside the control of visiting school groups or program providers, other forms of preparation can be addressed. Extensive research on field trip programs, including EE field trip programs, suggests that student preparation, including logistical preparation, before a program is important for setting the stage for effective student learning (e.g. Storksdieck 2006;deWitt and Storksdieck 2008;Lee, Stern, and Powell 2020). In some cases, students might be exposed to a pre-visit video about the site, which can focus on the right gear to bring, clarify expectations for the visit, introduce important subject matter, vocabulary, tools, or techniques that might help to prepare them for a more fruitful exhibit. Alternatively, teachers might cover these topics in the classroom, sometimes with the help of support materials designed by the field trip providers. Prior research suggests that teachers sometimes forgo preparing for a field trip due to a lack of familiarity with the site and/or a perceived lack of time in the classroom (e.g. Anderson, Kisiel, and Storksdieck 2006;Cox-Petersen and Pfaffinger 1998). developing stronger partnerships with visiting school teachers could potentially enhance preparation for both teachers and students (Lee, Stern and Powell 2020) and enhance staging.
Finally, several other programmatic and educator characteristics were also associated with higher performing programs and had mid-level effect sizes (above .5) (Cohen 1992). Effective class management was positively associated with outcomes. Meanwhile, three characteristics were negatively associated with outcomes: too slow of program pace, providing lecture-based programming, and educators that operated as walking encyclopedias regurgitating facts. Prior research in formal education supports these findings that effective class management improves academic performance (e.g. Pianta and Hamre 2009). Research in informal settings also supports the findings that an overreliance on didactic lecture based programming and fact-based approaches erode outcomes (e.g. Stern and Powell 2013).
Our study and analyses identified program characteristics that were most consistently linked to short-term student outcomes across a large and diverse sample of single-day EE field trip programs. The final RWA model accounted for 18% of the variance in the GMC EE21 outcome. Thus, although these 11 characteristics were consistently related to better student outcomes, approximately 80% of the variance was unexplained and likely attributable to other factors, such as pre-visit preparation (Lee, Stern and Powell 2020), students' backgrounds (Stern, Powell and Frensley 2022), measurement shortcomings, and other factors not measured in this study. Moreover, immediate post program measurement of student outcomes provided a consistent means for comparison across programs, however the study was unable to measure longer-term outcomes for participating students. Thus, the findings distinguish the approaches and characteristics that influence immediate outcomes, but we cannot necessarily extrapolate to what may influence longer-term impacts. While conducting a large-scale study of this nature using pre-experience, immediate post experience, and follow-up outcome measures would be a challenging logistical task, it could yield more definitive results regarding the program characteristics that lead to more lasting impacts. Moreover, these studies could better examine the influence of pre-experience preparation and follow-up activities (Lee, Stern and Powell 2020;Smith-Sebasto and Cavern 2006).
Because we so rarely observed some program characteristics, such as reflection, multiple viewpoints, storytelling, and others, our study was unable to assess their potential relationships with student outcomes. It may be that these approaches are particularly rare in the types of EE we observed-that is, single-day field trips for middle school students. Observing other types of EE programs, such as multi-day residential programs, might yield greater insights into these and other characteristics. We encourage future researchers to build off our work and the work of others to examine similar programs in other contexts, as well as virtual programs and multi-day residential programs in experimental or comparative ways to begin to better understand what works in these other EE contexts (for example, see Merritt et al. 2022;Frensley et al. 2022).
Because we aggregated the data to the program level, our study also did not isolate which program approaches are best for different audiences (specific grade/age, races and ethnicities, or sexual identity), but rather which practices showed strongest relationships with outcomes across all audiences. Because prior research suggests that EE is not one-size fits all, future research should examine what works best for different audiences and in what contexts. Aggregating the data to the program level precluded us from using some of the individual-student demographic data. Our aggregated sample size was also insufficient to examine which characteristics tended to function better or worse within meaningful aggregated subsamples. Using multi-level modeling may allow for identifying the relationship between different approaches and the outcomes for specific types of individuals. However, a larger and more diverse sample size would be necessary to enable such cross-context investigations. Moreover, our quantitative measures, while broadly descriptive, do not address the nuances that a more qualitative examination of fewer programs would be able to explore. We urge future researchers to pursue comparative approaches that combine qualitative observations of programs with quantitative outcome measures to provide holistic descriptions of extraordinary individual programs.
Our study observed a large sample of single-day field trip programs for youth in grades 5th-8th in the US in an attempt to identify the techniques and approaches that were most consistently associated with more positive learning outcomes. The results do not provide an exclusive list of approaches and characteristics that definitively enhance learning outcomes-as our study variables were limited to those we observed within our sample. However, as one of the first studies of this kind, we urge other researchers to build upon our research design to further ground the field of EE in evidence and better adapt to the changing demographics and lived experiences of youth worldwide.