Despite increased efforts to assess the adoption rates of open
science and robustness of reproducibility in sub-disciplines of
education technology, there is a lack of understanding of why
some research is not reproducible. Prior work has taken the
first step toward assessing reproducibility of research, but
has assumed certain constraints which hinder its discovery.
Thus, the purpose of this study was to replicate previous
work on papers within the proceedings of the International
Conference on Educational Data Mining and develop metrics
to accurately report on which papers are reproducible
and why. Specifically, we examined 208 papers, attempted
to reproduce them, documented reasons for reproducibility
failures, and asked authors to provide additional information
needed to reproduce their study. Our results showed that
out of 12 papers that were potentially reproducible, only
one successfully reproduced all analyses, and another two
reproduced most of the analyses. The most common failure
for reproducibility was failure to mention libraries needed,
followed by non-seeded randomness.
All openly accessible work can be found in an Open Science
Foundation project1.
more »
« less
How to Open Science: Debugging Reproducibility within the Educational Data Mining Conference
Despite increased efforts to assess the adoption rates of open
science and robustness of reproducibility in sub-disciplines
of education technology, there is a lack of understanding
of why some research is not reproducible. Prior work has
taken the first step toward assessing reproducibility of research,
but has assumed certain constraints which hinder
its discovery. Thus, the purpose of this study was to replicate
previous work on papers within the proceedings of the
International Conference on Educational Data Mining to accurately
report on which papers are reproducible and why.
Specifically, we examined 208 papers, attempted to reproduce
them, documented reasons for reproducibility failures,
and asked authors to provide additional information needed
to reproduce their study. Our results showed that out of 12
papers that were potentially reproducible, only one successfully
reproduced all analyses, and another two reproduced
most of the analyses. The most common failure for reproducibility
was failure to mention libraries needed, followed
by non-seeded randomness.
All openly accessible work can be found in an Open Science
Foundation project1.
more »
« less
- Award ID(s):
- 2118725
- PAR ID:
- 10451112
- Date Published:
- Journal Name:
- EDM 2023
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Despite increased efforts to assess the adoption rates of open science and robustness of reproducibility in sub-disciplines of education technology, there is a lack of understanding of why some research is not reproducible. Prior work has taken the first step toward assessing reproducibility of research, but has assumed certain constraints which hinder its discovery. Thus, the purpose of this study was to replicate previous work on papers within the proceedings of the International Conference on Educational Data Mining to accurately report on which papers are reproducible and why. Specifically, we examined 208 papers, attempted to reproduce them, documented reasons for reproducibility failures, and asked authors to provide additional information needed to reproduce their study. Our results showed that out of 12 papers that were potentially reproducible, only one successfully reproduced all analyses, and another two reproduced most of the analyses. The most common failure for reproducibility was failure to mention libraries needed, followed by non-seeded randomness.more » « less
-
Within the field of education technology, learning analytics has increased in popularity over the past decade. Researchers conduct experiments and develop software, building on each other’s work to create more intricate systems. In parallel, open science — which describes a set of practices to make research more open, transparent, and reproducible — has exploded in recent years, resulting in more open data, code, and materials for researchers to use. However, without prior knowledge of open science, many researchers do not make their datasets, code, and materials openly available, and those that are available are often difficult, if not impossible, to reproduce. The purpose of the current study was to take a close look at our field by examining previous papers within the proceedings of the International Conference on Learning Analytics and Knowledge, and document the rate of open science adoption (e.g., preregistration, open data), as well as how well available data and code could be reproduced. Specifically, we examined 133 research papers, allowing ourselves 15 minutes for each paper to identify open science practices and attempt to reproduce the results according to their provided specifications. Our results showed that less than half of the research adopted standard open science principles, with approximately 5% fully meeting some of the defined principles. Further, we were unable to reproduce any of the papers successfully in the given time period. We conclude by providing recommendations on how to improve the reproducibility of our research as a field moving forward. All openly accessible work can be found in an Open Science Foundation project1.more » « less
-
Within the field of education technology, learning analytics has increased in popularity over the past decade. Researchers conduct experiments and develop software, building on each other’s work to create more intricate systems. In parallel, open science — which describes a set of practices to make research more open, transparent, and reproducible — has exploded in recent years, resulting in more open data, code, and materials for researchers to use. However, without prior knowledge of open science, many researchers do not make their datasets, code, and materials openly available, and those that are available are often difficult, if not impossible, to reproduce. The purpose of the current study was to take a close look at our field by examining previous papers within the proceedings of the International Conference on Learning Analytics and Knowledge, and document the rate of open science adoption (e.g., preregistration, open data), as well as how well available data and code could be reproduced. Specifically, we examined 133 research papers, allowing ourselves 15 minutes for each paper to identify open science practices and attempt to reproduce the results according to their provided specifications. Our results showed that less than half of the research adopted standard open science principles, with approximately 5% fully meeting some of the defined principles. Further, we were unable to reproduce any of the papers successfully in the given time period. We conclude by providing recommendations on how to improve the reproducibility of our research as a field moving forward. All openly accessible work can be found in an Open Science Foundation project1.more » « less
-
There have been numerous efforts documenting the effects of open science in existing papers; however, these efforts typically only consider the author's analyses and supplemental materials from the papers. While understanding the current rate of open science adoption is important, it is also vital that we explore the factors that may encourage such adoption. One such factor may be publishing organizations setting open science requirements for submitted articles: encouraging researchers to adopt more rigorous reporting and research practices. For example, within the education technology discipline, theACM Conference on Learning @ Scale (L@S) has been promoting open science practices since 2018 through a Call For Papers statement. The purpose of this study was to replicate previous papers within the proceedings of L@S and compare the degree of open science adoption and robust reproducibility practices to other conferences in education technology without a statement on open science. Specifically, we examined 93 papers and documented the open science practices used. We then attempted to reproduce the results with invitation from authors to bolster the chance of success. Finally, we compared the overall adoption rates to those from other conferences in education technology. Although the overall responses to the survey were low, our cursory review suggests that researchers at L@S might be more familiar with open science practices compared to the researchers who published in the International Conference on Artificial Intelligence in Education (AIED) and the International Conference on Educational Data Mining (EDM): 13 of 28 AIED and EDM responses were unfamiliar with preregistrations and 7 unfamiliar with preprints, while only 2 of 7 L@S responses were unfamiliar with preregistrations and 0 with preprints. The overall adoption of open science practices at L@S was much lower with only 1% of papers providing open data, 5% providing open materials, and no papers had a preregistration. All openly accessible work can be found in an Open Science Framework project.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.35542/osf.io/4hn8q
