There have been numerous efforts documenting the effects of open science in existing papers; however, these efforts typically only consider the author's analyses and supplemental materials from the papers. While understanding the current rate of open science adoption is important, it is also vital that we explore the factors that may encourage such adoption. One such factor may be publishing organizations setting open science requirements for submitted articles: encouraging researchers to adopt more rigorous reporting and research practices. For example, within the education technology discipline, theACM Conference on Learning @ Scale (L@S) has been promoting open science practices since 2018 through a Call For Papers statement. The purpose of this study was to replicate previous papers within the proceedings of L@S and compare the degree of open science adoption and robust reproducibility practices to other conferences in education technology without a statement on open science. Specifically, we examined 93 papers and documented the open science practices used. We then attempted to reproduce the results with invitation from authors to bolster the chance of success. Finally, we compared the overall adoption rates to those from other conferences in education technology. Although the overall responses to the survey were low, our cursory review suggests that researchers at L@S might be more familiar with open science practices compared to the researchers who published in the International Conference on Artificial Intelligence in Education (AIED) and the International Conference on Educational Data Mining (EDM): 13 of 28 AIED and EDM responses were unfamiliar with preregistrations and 7 unfamiliar with preprints, while only 2 of 7 L@S responses were unfamiliar with preregistrations and 0 with preprints. The overall adoption of open science practices at L@S was much lower with only 1% of papers providing open data, 5% providing open materials, and no papers had a preregistration.
All openly accessible work can be found in an Open Science Framework project.
more »
« less
Next directions in measurement of the home mathematics environment: An international and interdisciplinary perspective
This article synthesizes findings from an international virtual conference, funded by the United States National Science Foundation, focused on the home mathematics environment (HME). In light of inconsistencies and gaps in research investigating relations between the HME and children’s outcomes, the purpose of the conference was to discuss actionable steps and considerations for future work. The conference was composed of international researchers with a wide range of expertise and backgrounds. Presentations and discussions during the conference centered broadly on the need to better operationalize and measure the HME as a construct—focusing on issues related to child, family, and community factors, country and cultural factors, and the cognitive and affective characteristics of caregivers and children. Results of the conference and a subsequent writing workshop include a synthesis of core questions and key considerations for the field of research on the HME. Findings highlight the need for the field at large to use multi-method measurement approaches to capture nuances in the HME, and to do so with increased international and interdisciplinary collaboration, open science practices, and communication among scholars.
more »
« less
- PAR ID:
- 10316633
- Author(s) / Creator(s):
- ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; more »
- Date Published:
- Journal Name:
- Journal of Numerical Cognition
- Volume:
- 7
- Issue:
- 2
- ISSN:
- 2363-8761
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
There have been numerous efforts documenting the effects of open science in existing papers; however, these efforts typically only consider the author’s analyses and supplemental materials from the papers. While understanding the current rate of open science adoption is important, it is also vital that we explore the factors that may encourage such adoption. One such factor may be publishing organizations setting open science requirements of submitted arti- cles: encouraging researchers to adopt more rigorous reporting and research practices. For example, within the education technology discipline, the ACM Conference on Learning @ Scale (L@S) has been promoting open science practices since 2018 through a Call For Pa- pers statement. The purpose of this study was to replicate previous papers within the proceedings of L@S and compare the degree of open science adoption and robust reproducibility practices to other conferences in education technology without a statement on open science. Specifically, we examined 93 papers and documented the open science practices used. We then attempted to reproduce the results with intervention from authors to bolster the chance of suc- cess. Finally, we compared the overall adoption rates to those from other conferences in education technology. Our cursory review sug- gests that researchers at L@S were more knowledgeable in open science practices, such as preregistration or preprints, compared to the researchers who published in International Conference on Artificial Intelligence in Education and the International Conference on Educational Data Mining as they were less likely to say they were unfamiliar with the practices. However, the overall adoption of open science practices was significantly lower with only 1% of papers providing open data, 5% providing open materials, and no papers with a preregistration. Based on speculation, the low adoption rates maybe due to 20% of the papers not using a dataset, at-scale datasets and materials that were unable to be released to avoid security issues or sensitive data leaks, or that data were being used in ongoing research and are not considered complete enough for release by the authors. All openly accessible work can be found in an Open Science Framework projectmore » « less
-
null (Ed.)The authors present a new approach to show how interdisciplinary collaborations among a group of institutions can provide a unique opportunity for students to engage across the science-policy nexus using the framework of the Sustainable Development Goals and the United Nations Framework Convention on Climate Change. Through collaboration across seven higher education institutions in the United States and Australia, virtual student research teams worked together across disciplines such as economics, ecology, and other earth and social sciences to address research questions centered on sustainable development goals. The teams presented their findings in person to diplomats and delegates at the 2019 United Nations Conference of the Parties meeting in Madrid, which also had strong qualitative impacts on their perceptions of international science-policy interfaces.more » « less
-
Within the field of education technology, learning analytics has increased in popularity over the past decade. Researchers conduct experiments and develop software, building on each other’s work to create more intricate systems. In parallel, open science — which describes a set of practices to make research more open, transparent, and reproducible — has exploded in recent years, resulting in more open data, code, and materials for researchers to use. However, without prior knowledge of open science, many researchers do not make their datasets, code, and materials openly available, and those that are available are often difficult, if not impossible, to reproduce. The purpose of the current study was to take a close look at our field by examining previous papers within the proceedings of the International Conference on Learning Analytics and Knowledge, and document the rate of open science adoption (e.g., preregistration, open data), as well as how well available data and code could be reproduced. Specifically, we examined 133 research papers, allowing ourselves 15 minutes for each paper to identify open science practices and attempt to reproduce the results according to their provided specifications. Our results showed that less than half of the research adopted standard open science principles, with approximately 5% fully meeting some of the defined principles. Further, we were unable to reproduce any of the papers successfully in the given time period. We conclude by providing recommendations on how to improve the reproducibility of our research as a field moving forward. All openly accessible work can be found in an Open Science Foundation project1.more » « less
-
Within the field of education technology, learning analytics has increased in popularity over the past decade. Researchers conduct experiments and develop software, building on each other’s work to create more intricate systems. In parallel, open science — which describes a set of practices to make research more open, transparent, and reproducible — has exploded in recent years, resulting in more open data, code, and materials for researchers to use. However, without prior knowledge of open science, many researchers do not make their datasets, code, and materials openly available, and those that are available are often difficult, if not impossible, to reproduce. The purpose of the current study was to take a close look at our field by examining previous papers within the proceedings of the International Conference on Learning Analytics and Knowledge, and document the rate of open science adoption (e.g., preregistration, open data), as well as how well available data and code could be reproduced. Specifically, we examined 133 research papers, allowing ourselves 15 minutes for each paper to identify open science practices and attempt to reproduce the results according to their provided specifications. Our results showed that less than half of the research adopted standard open science principles, with approximately 5% fully meeting some of the defined principles. Further, we were unable to reproduce any of the papers successfully in the given time period. We conclude by providing recommendations on how to improve the reproducibility of our research as a field moving forward. All openly accessible work can be found in an Open Science Foundation project1.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.5964/jnc.6143
