More Like this

1. Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

   https://doi.org/10.7554/eLife.81916  

   Sherratt, Katharine ; Gruson, Hugo ; Grah, Rok ; Johnson, Helen ; Niehus, Rene ; Prasse, Bastian ; Sandmann, Frank ; Deuschel, Jannik ; Wolffram, Daniel ; Abbott, Sam ; et al ( April 2023 , eLife)

   Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022.

   We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1–4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance.

   Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models’ forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models’ forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models’ forecasts of deaths (N=763 predictions from 20 models). Across a 1–4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models.

   Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks.

   AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 (https://www.nfdi4health.de/task-force-covid-19-2) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).

    

   more » « less
2. Machine learning for the educational sciences

   https://doi.org/10.1002/rev3.3310  

   Hilbert, Sven ; Coors, Stefan ; Kraus, Elisabeth ; Bischl, Bernd ; Lindl, Alfred ; Frei, Mario ; Wild, Johannes ; Krauss, Stefan ; Goretzko, David ; Stachl, Clemens ( November 2021 , Review of Education)

   Machine learning (ML) provides a powerful framework for the analysis of high‐dimensional datasets by modelling complex relationships, often encountered in modern data with many variables, cases and potentially non‐linear effects. The impact of ML methods on research and practical applications in the educational sciences is still limited, but continuously grows, as larger and more complex datasets become available through massive open online courses (MOOCs) and large‐scale investigations. The educational sciences are at a crucial pivot point, because of the anticipated impact ML methods hold for the field. To provide educational researchers with an elaborate introduction to the topic, we provide an instructional summary of the opportunities and challenges of ML for the educational sciences, show how a look at related disciplines can help learning from their experiences, and argue for a philosophical shift in model evaluation. We demonstrate how the overall quality of data analysis in educational research can benefit from these methods and show how ML can play a decisive role in the validation of empirical models. Specifically, we (1) provide an overview of the types of data suitable for ML and (2) give practical advice for the application of ML methods. In each section, we provide analytical examples and reproducible R code. Also, we provide an extensive Appendix on ML‐based applications for education. This instructional summary will help educational scientists and practitioners to prepare for the promises and threats that come with the shift towards digitisation and large‐scale assessment in education.

   In 2020, the worldwide SARS‐COV‐2 pandemic forced the educational sciences to perform a rapid paradigm shift with classrooms going online around the world—a hardly novel but now strongly catalysed development. In the context of data‐driven education, this paper demonstrates that the widespread adoption of machine learning techniques is central for the educational sciences and shows how these methods will become crucial tools in the collection and analysis of data and in concrete educational applications. Helping to leverage the opportunities and to avoid the common pitfalls of machine learning, this paper provides educators with the theoretical, conceptual and practical essentials.

   The process of teaching and learning is complex, multifaceted and dynamic. This paper contributes a seminal resource to highlight the digitisation of the educational sciences by demonstrating how new machine learning methods can be effectively and reliably used in research, education and practical application.

   The progressing digitisation of societies around the globe and the impact of the SARS‐COV‐2 pandemic have highlighted the vulnerabilities and shortcomings of educational systems. These developments have shown the necessity to provide effective educational processes that can support sometimes overwhelmed teachers to digitally impart knowledge on the plan of many governments and policy makers. Educational scientists, corporate partners and stakeholders can make use of machine learning techniques to develop advanced, scalable educational processes that account for individual needs of learners and that can complement and support existing learning infrastructure. The proper use of machine learning methods can contribute essential applications to the educational sciences, such as (semi‐)automated assessments, algorithmic‐grading, personalised feedback and adaptive learning approaches. However, these promises are strongly tied to an at least basic understanding of the concepts of machine learning and a degree of data literacy, which has to become the standard in education and the educational sciences.

   Demonstrating both the promises and the challenges that are inherent to the collection and the analysis of large educational data with machine learning, this paper covers the essential topics that their application requires and provides easy‐to‐follow resources and code to facilitate the process of adoption.

    

   more » « less
3. Teaching little kids big sentences: A randomized controlled trial showing that children with DLD respond to complex syntax intervention embedded within the context of preschool/kindergarten science instruction

   https://doi.org/10.1111/1460-6984.12882  

   Owen Van Horne, Amanda J. ; Curran, Maura ; Cook, Susan Wagner ; Cole, Renée ; McGregor, Karla K. ( May 2023 , International Journal of Language & Communication Disorders)

   The language of the science curriculum is complex, even in the early grades. To communicate their scientific observations, children must produce complex syntax, particularly complement clauses (e.g.,I think it will float;We noticed that it vibrates). Complex syntax is often challenging for children with developmental language disorder (DLD), and thus their learning and communication of science may be compromised.

   We asked whether recast therapy delivered in the context of a science curriculum led to gains in complement clause use and scientific content knowledge. To understand the efficacy of recast therapy, we compared changes in science and language knowledge in children who received treatment for complement clauses embedded in a first‐grade science curriculum to two active control conditions (vocabulary + science, phonological awareness + science).

   This 2‐year single‐site three‐arm parallel randomized controlled trial was conducted in Delaware, USA. Children with DLD, not yet in first grade and with low accuracy on complement clauses, were eligible. Thirty‐three 4–7‐year‐old children participated in the summers of 2018 and 2019 (2020 was cancelled due to COVID‐19). We assigned participants to arms using 1:1:1 pseudo‐random allocation (avoiding placing siblings together). The intervention consisted of 39 small‐group sessions of recast therapy, robust vocabulary instruction or phonological awareness intervention during eight science units over 4 weeks, followed by two science units (1 week) taught without language intervention. Pre‐/post‐measures were collected 3 weeks before and after camp by unmasked assessors.

   Primary outcome measures were accuracy on a 20‐item probe of complement clause production and performance on ten 10‐item unit tests (eight science + language, two science only). Complete data were available for 31 children (10 grammar, 21 active control); two others were lost to follow‐up. Both groups made similar gains on science unit tests for science + language content (pre versus post,d= 2.9,p< 0.0001; group,p= 0.24). The grammar group performed significantly better at post‐test than the active control group (d= 2.5,p= 0.049) on complement clause probes and marginally better on science‐only unit tests (d= 2.5,p= 0.051).

   Children with DLD can benefit from language intervention embedded in curricular content and learn both language and science targets taught simultaneously. Tentative findings suggest that treatment for grammar targets may improve academic outcomes.

   We know that recast therapy focused on morphology is effective but very time consuming. Treatment for complex syntax in young children has preliminary efficacy data available. Prior research provides mixed evidence as to children’s ability to learn language targets in conjunction with other information.

   This study provides additional data supporting the efficacy of intensive complex syntax recast therapy for children ages 4–7 with Developmental Language Disorder. It also provides data that children can learn language targets and science curricular content simultaneously.

   As SLPs, we have to talk about something to deliver language therapy; we should consider talking about curricular content. Recast therapy focused on syntactic frames is effective with young children.

    

   more » « less
4. The scalable precision medicine open knowledge engine (SPOKE): a massive knowledge graph of biomedical information

   https://doi.org/10.1093/bioinformatics/btad080  

   Morris, John H. ; Soman, Karthik ; Akbas, Rabia E. ; Zhou, Xiaoyuan ; Smith, Brett ; Meng, Elaine C. ; Huang, Conrad C. ; Cerono, Gabriel ; Schenk, Gundolf ; Rizk-Jackson, Angela ; et al ( February 2023 , Bioinformatics)

   Knowledge graphs (KGs) are being adopted in industry, commerce and academia. Biomedical KG presents a challenge due to the complexity, size and heterogeneity of the underlying information.

   In this work, we present the Scalable Precision Medicine Open Knowledge Engine (SPOKE), a biomedical KG connecting millions of concepts via semantically meaningful relationships. SPOKE contains 27 million nodes of 21 different types and 53 million edges of 55 types downloaded from 41 databases. The graph is built on the framework of 11 ontologies that maintain its structure, enable mappings and facilitate navigation. SPOKE is built weekly by python scripts which download each resource, check for integrity and completeness, and then create a ‘parent table’ of nodes and edges. Graph queries are translated by a REST API and users can submit searches directly via an API or a graphical user interface. Conclusions/Significance: SPOKE enables the integration of seemingly disparate information to support precision medicine efforts.

   The SPOKE neighborhood explorer is available at https://spoke.rbvi.ucsf.edu.

   Supplementary data are available at Bioinformatics online.

    

   more » « less
5. Speech analysis of teaching assistant interventions in small group collaborative problem solving with undergraduate engineering students

   https://doi.org/10.1111/bjet.13449  

   D'Angelo, Cynthia M. ; Rajarathinam, Robin Jephthah ( March 2024 , British Journal of Educational Technology)

   This descriptive study focuses on using voice activity detection (VAD) algorithms to extract student speech data in order to better understand the collaboration of small group work and the impact of teaching assistant (TA) interventions in undergraduate engineering discussion sections. Audio data were recorded from individual students wearing head‐mounted noise‐cancelling microphones. Video data of each student group were manually coded for collaborative behaviours (eg, group task relatedness, group verbal interaction and group talk content) of students and TA–student interactions. The analysis includes information about the turn taking, overall speech duration patterns and amounts of overlapping speech observed both when TAs were intervening with groups and when they were not. We found that TAs very rarely provided explicit support regarding collaboration. Key speech metrics, such as amount of turn overlap and maximum turn duration, revealed important information about the nature of student small group discussions and TA interventions. TA interactions during small group collaboration are complex and require nuanced treatments when considering the design of supportive tools.

   What is already known about this topic

   Student turn taking can provide information about the nature of student discussions and collaboration.

   Real classroom audio data of small groups typically have lots of background noise and present challenges for audio analysis.

   TAs have little training in how to productively intervene with students about collaborative skills.

   What this paper adds

   TA interaction with groups primarily focused on task progress and understanding of concepts with negligible explicit support on building collaborative skills.

   TAs intervened with the groups often which gave the students little time for uptake of their suggestions or deeper discussion.

   Student turn overlap was higher without the presence of TAs.

   Maximum turn duration can be an important real‐time turn metric to identify the least verbally active student participant in a group.

   Implications for practice and/or policy

   TA training should include information about how to monitor groups for collaborative behaviours and when and how they should intervene to provide feedback and support.

   TA feedback systems should keep track of previous interventions by TAs (especially in contexts where there are multiple TAs facilitating) and the duration since previous intervention to ensure that TAs do not intervene with a group too frequently with little time for student uptake.

   Maximum turn duration could be used as a real‐time metric to identify the least verbally active student in a group so that support could be provided to them by the TAs.

    

   more » « less