An official website of the United States government
File systems that store metadata on a single machine or via a shared-disk abstraction face scalability challenges, especially in contexts demanding the management of billions of files. Recent work has shown that employing shared-nothing, distributed database system (DDBMS) for metadata storage can alleviate these scalability challenges without compromising on high availability guarantees. However, for low-scale deployments -- where metadata can fit in memory on a single machine -- these DDBMS-based systems typically perform an order of magnitude worse than systems that store metadata in memory on a single machine. This has limited the impact of these distributed database approaches, since they are only currently applicable to file systems of extreme scale. This paper describes FileScale, a three-tier architecture that incorporates a DDBMS as part of a comprehensive approach to file system metadata management. In contrast to previous approaches, FileScale performs comparably to the single-machine architecture at a small scale, while enabling linear scalability as the file system metadata increases.
more »
« less
Handling Detector Characterization Data (Metadata) in XENONnT
Effective metadata management is a consistent challenge faced by many scientific experiments. These challenges are magnified by the evolving needs of the experiment, the intricacies of seamlessly integrating a new system with existing analytical frameworks, and the crucial mandate to maintain database integrity. In this work we present the various challenges faced by experiments that produce a large amount of metadata and describe the solution used by the XENON experiment for metadata management.
more »
« less
- Award ID(s):
- 2112801
- PAR ID:
- 10615378
- Editor(s):
- De_Vita, R; Espinal, X; Laycock, P; Shadura, O
- Publisher / Repository:
- EPJ
- Date Published:
- Journal Name:
- EPJ Web of Conferences
- Volume:
- 295
- ISSN:
- 2100-014X
- Page Range / eLocation ID:
- 01033
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
AbstractManaging, processing, and sharing research data and experimental context produced on modern scientific instrumentation all present challenges to the materials research community. To address these issues, two MaRDA Working Groups on FAIR Data in Materials Microscopy Metadata and Materials Laboratory Information Management Systems (LIMS) convened and generated recommended best practices regarding data handling in the materials research community. Overall, the Microscopy Metadata Group recommends (1) instruments should capture comprehensive metadata about operators, specimens/samples, instrument conditions, and data formation; and (2) microscopy data and metadata should use standardized vocabularies and community standard identifiers. The LIMS Group produced the following guides and recommendations: (1) a cost and benefit comparison when implementing LIMS; (2) summaries of prerequisite requirements, capabilities, and roles of LIMS stakeholders; and (3) a review of metadata schemas and information-storage best practices in LIMS. Together, the groups hope these recommendations will accelerate breakthrough scientific discoveries via FAIR data. Impact statementWith the deluge of data produced in today’s materials research laboratories, it is critical that researchers stay abreast of developments in modern research data management, particularly as it relates to the international effort to make data more FAIR – findable, accessible, interoperable, and reusable. Most crucially, being able to responsibly share research data is a foundational means to increase progress on the materials research problems of high importance to science and society. Operational data management and accessibility are pivotal in accelerating innovation in materials science and engineering and to address mounting challenges facing our world, but the materials research community generally lags behind its cognate disciplines in these areas. To address this issue, the Materials Research Coordination Network (MaRCN) convened two working groups comprised of experts from across the materials data landscape in order to make recommendations to the community related to improvements in materials microscopy metadata standards and the use of Laboratory Information Management Systems (LIMS) in materials research. This manuscript contains a set of recommendations from the working groups and reflects the culmination of their 18-month efforts, with the hope of promoting discussion and reflection within the broader materials research community in these areas. Graphical abstractmore » « less
-
The start of the COVID-19 pandemic with its unique constraints led teachers of all disciplines, across all levels to employ diverse strategies in their quick transition to online learning. Emerging literature has suggested readiness and resources for the transition to remote learning along with intrinsic and extrinsic factors such as motivation and engagement to be frequent challenges faced by teachers. The purpose of this work is twofold: a) conduct a review of the post-pandemic literature to summarize the challenges of teaching engineering during the pandemic, and b) summarize the experiences of secondary teachers when teaching a pre-college engineering curriculum at the beginning of the pandemic. Collectively the literature review and experimental findings enable greater understanding and sense-making of adaptations made to prepare for future abrupt changes that may arise. Focus group data (n= 39) of high school engineering teachers were analyzed using sensemaking theory to reveal several challenges faced and strategies adopted during this unprecedented time. Our review findings suggest Lack of Engagement and Motivation, Scheduling, the Right Resources, and Training with Technology were the most frequent challenges noted in the literature. Our empirical findings showed similar challenges faced by pre-college teachers which were summarized into four themes: Teaching logistics, Time Management, Available Support, and Regulations. The themes provide the foundation for mitigation strategies for teachers during a crisis or abrupt change in the future.more » « less
-
ABSTRACT This paper reports on a demonstration of YAMZ (Yet Another Metadata Zoo) as a mechanism for building community consensus around metadata terms. The demonstration is motivated by the complexity of the metadata standards environment and the need for more user-friendly approaches for researchers to achieve vocabulary consensus. The paper reviews a series of metadata standardization challenges, explores crowdsourcing factors that offer possible solutions, and introduces the YAMZ system. A YAMZ demonstration is presented with members of the Toberer materials science laboratory at the Colorado School of Mines, where there is a need to confirm and maintain a shared understanding for the vocabulary supporting research documentation, data management, and their larger metadata infrastructure. The demonstration involves three key steps: 1) Sampling terms for the demonstration, 2) Engaging graduate student researchers in the demonstration, and 3) Reflecting on the demonstration. The results of these steps, including examples of the dialog provenance among lab members and voting, show the ease with YAMZ can facilitate building metadata vocabulary consensus. The conclusion discusses implications and highlights next steps.more » « less
-
null; null (Ed.)Biodiversity image repositories are crucial sources of training data for machine learning approaches to biological research. Metadata, specifically metadata about object quality, is putatively an important prerequisite to selecting sample subsets for these experiments. This study demonstrates the importance of image quality metadata to a species classification experiment involving a corpus of 1935 fish specimen images which were annotated with 22 metadata quality properties. A small subset of high quality images produced an F1 accuracy of 0.41 compared to 0.35 for a taxonomically matched subset of low quality images when used by a convolutional neural network approach to species identification. Using the full corpus of images revealed that image quality differed between correctly classified and misclassified images. We found the visibility of all anatomical features was the most important quality feature for classification accuracy. We suggest biodiversity image repositories consider adopting a minimal set of image quality metadata to support future machine learning projects.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1051/epjconf/202429501033
