More Like this

1. C ² Metadata: Automating the Capture of Data Transformations from Statistical Scripts in Data Documentation

   https://doi.org/10.1145/3299869.3320241  

   Song, Jie; Alter, George; Jagadish, H. V. (June 2019, Proceedings of the 2019 International Conference on Management of Data)

   null (Ed.)

   Datasets are often derived by manipulating raw data with statistical software packages. The derivation of a dataset must be recorded in terms of both the raw input and the manipulations applied to it. Statistics packages typically provide limited help in documenting provenance for the resulting derived data. At best, the operations performed by the statistical package are described in a script. Disparate representations make these scripts hard to understand for users. To address these challenges, we created Continuous Capture of Metadata (C2Metadata), a system to capture data transformations in scripts for statistical packages and represent it as metadata in a standard format that is easy to understand. We do so by devising a Structured Data Transformation Algebra (SDTA), which uses a small set of algebraic operators to express a large fraction of data manipulation performed in practice. We then implement SDTA, inspired by relational algebra, in a data transformation specification language we call SDTL. In this demonstration, we showcase C2Metadata’s capture of data transformations from a pool of sample transformation scripts in at least two languages: SPSS®and Stata®(SAS®and R are under development), for social science data in a large academic repository. We will allow the audience to explore C2Metadata using a web-based interface, visualize the intermediate steps and trace the provenance and changes of data at different levels for better understanding of the process. 

   more » « less
2. SDTA: An Algebra for Statistical Data Transformation

   https://doi.org/10.1145/3468791.3468811  

   Song, Jie; Jagadish, H. V.; Alter, George (July 2021, 33rd International Conference on Scientific and Statistical Database Management)

   null (Ed.)

   Statistical data manipulation is a crucial component of many data science analytic pipelines, particularly as part of data ingestion. This task is generally accomplished by writing transformation scripts in languages such as SPSS, Stata, SAS, R, Python (Pandas) and etc. The disparate data models, language representations and transformation operations supported by these tools make it hard for end users to understand and document the transformations performed, and for developers to port transformation code across languages. Tackling these challenges, we present a formal paradigm for statistical data transformation. It consists of a data model, called Structured Data Transformation Data Model (SDTDM), inspired by the data models of multiple statistical transformations frameworks; an algebra, Structural Data Transformation Algebra (SDTA), with the ability to transform not only data within SDTDM but also metadata at multiple structural levels; and an equivalent descriptive counterpart, called Structured Data Transformation Language (SDTL), recently adopted by the DDI Alliance that maintains international standards for metadata as part of its suite of products. Experiments with real statistical transformations on socio-economic data show that SDTL can successfully represent 86.1% and 91.6% respectively of 4,185 commands in SAS and 9,087 commands in SPSS obtained from a repository. We illustrate with examples how SDTA/SDTL could assist with the documentation of statistical data transformation, an important aspect often neglected in metadata of datasets.We propose a system called C2Metadata that automatically captures the transformation and provenance information in SDTL as a part of the metadata. Moreover, given the conversion mechanism from a source statistical language to SDTA/SDTL, we show how functional-equivalent transformation programs could be converted to other functionally equivalent programs, in the same or different language, permitting code reuse and result reproducibility, We also illustrate the possibility of using of SDTA to optimize SDTL transformations using rule-based rewrites similar to SQL optimizations. 

   more » « less
3. Structured data transformation algebra (SDTA) and its applications

   https://doi.org/10.1007/s10619-022-07418-6  

   Song, Jie; Alter, George; Jagadish, H V (September 2022, Distributed and Parallel Databases)

   Statistical analysis is a crucial component of many data science analytic pipelines, and preparing data for such analysis is a large part of the data ingestion step. This task is generally accomplished by writing transformation scripts in languages such as SPSS, Stata, SAS, R, Python (Pandas) etc. The disparate data models, language representations and transformation operations supported by these tools make it hard for end users to understand and document the transformations performed, and for developers to port transformation code across languages. Tackling these challenges, we present a formal paradigm for statistical data transformation called SDTA and embody in a language called SDTL. Experiments with real statistical transformations on socio-economic data show that SDTL can successfully represent 86.1% and 91.6% respectively of 4,185 commands in SAS and 9,087 commands in SPSS obtained from a repository. We illustrate how SDTA/SDTL could assist with the documentation of statistical data transformation, an important aspect often neglected in metadata of datasets. We propose a system called C2Metadata that automatically captures the transformation and provenance information in SDTL as a part of the metadata. Moreover, given the conversion mechanism from a source statistical language to SDTA/SDTL, we show how a data transformation program could be converted to other functionally equivalent programs, permitting code reuse and result reproducibility. We also illustrate the possibility of using SDTA to optimize SDTL transformations using rule-based rewrites similar to SQL optimizations. 

   more » « less
4. NSF FAIR Chemical Data Publishing Guidelines Workshop on Chemical Structures and Spectra: Major Outcomes and Outlooks for the Chemistry Community

   https://doi.org/10.7298/fs2d-hx95  

   Scalfani, Vincent F; McEwen, Leah R. (April 2020, eCommonsCornell)

   null (Ed.)

   The National Science Foundation Office of Advanced Cyberinfrastructure (NSF-OAC) funded a workshop in March 2019 focused on advancing the sharing of machine-readable chemical structures and spectra. Around 40 stakeholders from the chemistry, chemical information, and software communities took part in the two-day workshop entitled “FAIR Chemical Data Publishing Guidelines for Chemical Structures and Spectra.” Major topics discussed included publishing data workflows and guidelines, FAIR criteria/metadata profiles, value propositions, a publisher implementation pilot, and community support and engagement. This report summarizes the workshop conversations, major outcomes, and target areas for further activities. Primary outcomes from the workshop include identification of key metadata elements for sharing machine-readable structures and spectra, a sample of concise author guidelines, and a publisher proposal to accept enhanced supporting information files including these data types and associated metadata alongside articles. Selected target areas for further activities include the creation of author file and metadata packaging tools to facilitate easy compilation of data, and increased training for stakeholders specifically in the generation and handling of machine-readable file formats. We conclude this report with our outlooks and highlight several related community efforts initiated after the workshop. 

   more » « less
5. A FAIR Resource Recommender System for Smart Open Scientific Inquiries

   https://doi.org/10.3390/app15158334  

   Sakib, Syed N; Rubaiat, Sajratul Y; Naha, Kallol; Rahman, Hasan H; Jamil, Hasan M (July 2025, Applied Sciences)

   A vast proportion of scientific data remains locked behind dynamic web interfaces, often called the deep web—inaccessible to conventional search engines and standard crawlers. This gap between data availability and machine usability hampers the goals of open science and automation. While registries like FAIRsharing offer structured metadata describing data standards, repositories, and policies aligned with the FAIR (Findable, Accessible, Interoperable, and Reusable) principles, they do not enable seamless, programmatic access to the underlying datasets. We present FAIRFind, a system designed to bridge this accessibility gap. FAIRFind autonomously discovers, interprets, and operationalizes access paths to biological databases on the deep web, regardless of their FAIR compliance. Central to our approach is the Deep Web Communication Protocol (DWCP), a resource description language that represents web forms, HyperText Markup Language (HTML) tables, and file-based data interfaces in a machine-actionable format. Leveraging large language models (LLMs), FAIRFind combines a specialized deep web crawler and web-form comprehension engine to transform passive web metadata into executable workflows. By indexing and embedding these workflows, FAIRFind enables natural language querying over diverse biological data sources and returns structured, source-resolved results. Evaluation across multiple open-source LLMs and database types demonstrates over 90% success in structured data extraction and high semantic retrieval accuracy. FAIRFind advances existing registries by turning linked resources from static references into actionable endpoints, laying a foundation for intelligent, autonomous data discovery across scientific domains. 

   more » « less