An official website of the United States government
DOI 10.1109/COMPSAC.2019.10284 Abstract—Hadoop is a popular data-analytics platform based on the MapReduce model. When analyzing extremely big data, hard disk drives are commonly used and Hadoop performance can be optimized by improving I/O performance. Hard disk drives have different performance depending on whether data are placed in the outer or inner disk zones. In this paper, we propose a method that uses knowledge of job characteristics to place data in hard disk drives so that Hadoop performance is improved. Files of a job that intensively and sequentially accesses the storage device are placed in outer disk tracks which have higher sequential access speed than inner tracks. Temporary and permanent files are placed in the outer and inner zones, respectively. This enables repeated usage of the faster zones by avoiding the use of the faster zones by permanent files. Our evaluation demonstrates that the proposed method improves the performance of Hadoop jobs by 15.0% over the normal case when file placement is not used. The proposed method also outperforms a previously proposed placement approach by 9.9%.
more »
« less
Moving towards Real-time Data-driven Quality Monitoring: A Case Study of Hard Disk Drives
Since its emergence, the cloud manufacturing concept has been transforming the manufacturing and remanufacturing industry into a big data and service-oriented environment. The aggressive push toward data collection in cloud-based and cyber-physical systems provides both challenges and opportunities for predictive analytics. One of the key applications of predictive analytics in such domains is predictive quality management that aims to fully exploit the potentials provided by the enormous data collected via cloud-based systems. As a case study, a data set of hard disk drives’ Self-Monitoring, Analysis and Reporting Technology (SMART) attributes from a cloud-storage service provider has been analyzed to derive some insights about the challenges and opportunities of using product lifecycle data. An analysis of time-to-failure monitoring of hard disk drives in real-time has been carried out and the corresponding challenges have been discussed.
more »
« less
- Award ID(s):
- 1705621
- PAR ID:
- 10065240
- Date Published:
- Journal Name:
- Procedia Manufacturing, SME North American Manufacturing Research Conference
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This study aims to discuss the state-of-the-art digital factory (DF) development combining digital twins (DTs), sensing devices, laser additive manufacturing (LAM) and subtractive manufacturing (SM) processes. The current shortcomings and outlook of the DF also have been highlighted. A DF is a state-of-the-art manufacturing facility that uses innovative technologies, including automation, artificial intelligence (AI), the Internet of Things, additive manufacturing (AM), SM, hybrid manufacturing (HM), sensors for real-time feedback and control, and a DT, to streamline and improve manufacturing operations. Design/methodology/approachThis study presents a novel perspective on DF development using laser-based AM, SM, sensors and DTs. Recent developments in laser-based AM, SM, sensors and DTs have been compiled. This study has been developed using systematic reviews and meta-analyses (PRISMA) guidelines, discussing literature on the DTs for laser-based AM, particularly laser powder bed fusion and direct energy deposition, in-situ monitoring and control equipment, SM and HM. The principal goal of this study is to highlight the aspects of DF and its development using existing techniques. FindingsA comprehensive literature review finds a substantial lack of complete techniques that incorporate cyber-physical systems, advanced data analytics, AI, standardized interoperability, human–machine cooperation and scalable adaptability. The suggested DF effectively fills this void by integrating cyber-physical system components, including DT, AM, SM and sensors into the manufacturing process. Using sophisticated data analytics and AI algorithms, the DF facilitates real-time data analysis, predictive maintenance, quality control and optimal resource allocation. In addition, the suggested DF ensures interoperability between diverse devices and systems by emphasizing standardized communication protocols and interfaces. The modular and adaptable architecture of the DF enables scalability and adaptation, allowing for rapid reaction to market conditions. Originality/valueBased on the need of DF, this review presents a comprehensive approach to DF development using DTs, sensing devices, LAM and SM processes and provides current progress in this domain.more » « less
-
Abstract. Flash based solid state drives (SSDs) have established them- selves as a higher-performance alternative to hard disk drives in cloud and mobile environments. Nevertheless, SSDs remain a performance bot- tleneck of computer systems due to their high I/O access latency. A com- mon approach for improving the access latency is prefetching. Prefetch- ing predicts future block accesses and preloads them into main memory ahead of time. In this paper, we discuss the challenges of prefetching in SSDs, explain why prior approaches fail to achieve high accuracy, and present a neural network based prefetching approach that signi cantly outperforms the state-of the-art. To achieve high performance, we ad- dress the challenges of prefetching in very large sparse address spaces, as well as prefetching in a timely manner by predicting ahead of time. We collect I/O trace les from several real-world applications running on cloud servers and show that our proposed approach consistently outper- forms the existing stride prefetchers by up to 800 and prior prefetching approaches based on Markov chains by up to 8. Furthermore, we pro- pose an address mapping learning technique to demonstrate the applica- bility of our approach to previously unseen SSD workloads and perform a hyperparameter sensitivity study.more » « less
-
In this review, we discuss the role of sensor analytics point solutions (SNAPS), a reduced complexity machine-assisted decision support tool. We summarize the approaches used for mobile phone-based chemical/biological sensors, including general hardware and software requirements for signal transduction and acquisition. We introduce SNAPS, part of a platform approach to converge sensor data and analytics. The platform is designed to consist of a portfolio of modular tools which may lend itself to dynamic composability by enabling context-specific selection of relevant units, resulting in case-based working modules. SNAPS is an element of this platform where data analytics, statistical characterization and algorithms may be delivered to the data either via embedded systems in devices, or sourced, in near real-time, from mist, fog or cloud computing resources. Convergence of the physical systems with the cyber components paves the path for SNAPS to progress to higher levels of artificial reasoning tools (ART) and emerge as data-informed decision support, as a service for general societal needs. Proof of concept examples of SNAPS are demonstrated both for quantitative data and qualitative data, each operated using a mobile device (smartphone or tablet) for data acquisition and analytics. We discuss the challenges and opportunities for SNAPS, centered around the value to users/stakeholders and the key performance indicators users may find helpful, for these types of machine-assisted tools.more » « less
-
Today’s problems require a plethora of analytics tasks to be conducted to tackle state-of-the-art computational challenges posed in society impacting many areas including health care, automotive, banking, natural language processing, image detection, and many more data analytics-related tasks. Sharing existing analytics functions allows reuse and reduces overall effort. However, integrating deployment frameworks in the age of cloud computing are often out of reach for domain experts. Simple frameworks are needed that allow even non-experts to deploy and host services in the cloud. To avoid vendor lock-in, we require a generalized composable analytics service framework that allows users to integrate their services and those offered in clouds, not only by one, but by many cloud compute and service providers.We report on work that we conducted to provide a service integration framework for composing generalized analytics frame-works on multi-cloud providers that we call our Generalized AI Service (GAS) Generator. We demonstrate the framework’s usability by showcasing useful analytics workflows on various cloud providers, including AWS, Azure, and Google, and edge computing IoT devices. The examples are based on Scikit learn so they can be used in educational settings, replicated, and expanded upon. Benchmarks are used to compare the different services and showcase general replicability.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
