skip to main content

Title: RAFI: Risk-Aware Failure Identification to Improve the RAS in Erasure-coded Data Centers
Data reliability and availability, and serviceability (RAS) of erasure-coded data centers are highly affected by data repair induced by node failures. Compared to the recovery phase of the data repair, which is widely studied and well optimized, the failure identification phase of the data repair is less investigated. Moreover, in a traditional failure identification scheme, all chunks share the same identification time threshold, thus losing opportunities to further improve the RAS. To solve this problem, we propose RAFI, a novel risk-aware failure identification scheme. In RAFI, chunk failures in stripes experiencing different numbers of failed chunks are identified using different time thresholds. For those chunks in a high risk stripe (a stripe with many failed chunks), a shorter identification time is adopted, thus improving the overall data reliability and availability. For those chunks in a low risk stripe (one with only a few failed chunks), a longer identification time is adopted, thus reducing the repair network traffic. Therefore, the RAS can be improved simultaneously. We use both simulations and prototyping implementation to evaluate RAFI. Results collected from extensive simulations demonstrate the effectiveness and efficiency of RAFI on improving the RAS. We implement a prototype on HDFS to verify the correctness and evaluate more » the computational cost of RAFI. « less
Authors:
; ;
Award ID(s):
1717660 1702474 1547804
Publication Date:
NSF-PAR ID:
10065106
Journal Name:
The USENIX Annual Technical Conference (ATC)
Sponsoring Org:
National Science Foundation
More Like this
  1. Data reliability and availability, and serviceability (RAS) of erasure-coded data centers are highly affected by data repair induced by node failures. In a traditional failure identification scheme, all chunks share the same identification time threshold, thus losing opportunities to further improve the RAS. To solve this problem, we propose RAFI, a novel risk-aware failure identification scheme. In RAFI, chunk failures in stripes experiencing different numbers of failed chunks are identified using different time thresholds. For those chunks in a high-risk stripe, a shorter identification time is adopted, thus improving the overall data reliability and availability. For those chunks in amore »low-risk stripe, a longer identification time is adopted, thus reducing the repair network traffic. Therefore, RAS can be improved simultaneously. We also propose three optimization techniques to reduce the additional overhead that RAFI imposes on management nodes' and to ensure that RAFI can work properly under large-scale clusters. We use simulation, emulation, and prototyping implementation to evaluate RAFI from multiple aspects. Simulation and prototype results prove the effectiveness and correctness of RAFI, and the performance improvement of the optimization techniques on RAFI is demonstrated by running the emulator.« less
  2. Millions of Californians access drinking water via domestic wells, which are vulnerable to drought andunsustainable groundwater management. Groundwater overdraft and the possibility of longer droughtduration under climate change threatens domestic well reliability, yet we lack tools to assess the impact ofsuch events. Here, we leverage 943 469 well completion reports and 20 years of groundwater elevationdata to develop a spatially-explicit domestic well failure model covering California’s Central Valley. Ourmodel successfully reproduces the spatial distribution of observed domestic well failures during the severe2012–2016 drought(n=2027). Next, the impact of longer drought duration(5–8years)on domestic wellfailure is evaluated, indicating that if the 2012–2016 droughtmore »would have continued into a 6 to 8 year longdrought, a total of 4037–5460 to 6538–8056 wells would fail. The same drought duration scenarios withan intervening wet winter in 2017 lead to an average of498 and 738 fewer well failures. Additionally, wemap vulnerable wells at high failure risk andfind that they align with clusters of predicted well failures.Lastly, we evaluate how the timing and implementation of different projected groundwater managementregimes impact groundwater levels and thus domestic well failure. When historic overdraft persists until2040, domestic well failures range from 5966 to 10 466(depending on the historic period considered).When sustainability is achieved progressively between 2020 and 2040, well failures range from 3677 to6943, and from 1516 to 2513 when groundwater is not allowed to decline after 2020.« less
  3. Smart grids can be vulnerable to attacks and accidents, and any initial failures in smart grids can grow to a large blackout because of cascading failure. Because of the importance of smart grids in modern society, it is crucial to protect them against cascading failures. Simulation of cascading failures can help identify the most vulnerable transmission lines and guide prioritization in protection planning, hence, it is an effective approach to protect smart grids from cascading failures. However, due to the enormous number of ways that the smart grids may fail initially, it is infeasible to simulate cascading failures at amore »large scale nor identify the most vulnerable lines efficiently. In this paper, we aim at 1) developing a method to run cascading failure simulations at scale and 2) building simplified, diffusion based cascading failure models to support efficient and theoretically bounded identification of most vulnerable lines. The goals are achieved by first constructing a novel connection between cascading failures and natural languages, and then adapting the powerful transformer model in NLP to learn from cascading failure data. Our trained transformer models have good accuracy in predicting the total number of failed lines in a cascade and identifying the most vulnerable lines. We also constructed independent cascade (IC) diffusion models based on the attention matrices of the transformer models, to support efficient vulnerability analysis with performance bounds.« less
  4. The objective of International Ocean Discovery Program (IODP) Expedition 384 was to carry out engineering tests with the goal of improving the chances of success in deep (>1 km) drilling and coring in igneous ocean crust. A wide range of tools and technologies for potential testing were proposed by the Deep Crustal Drilling Engineering Working Group in 2017 based on reports from recent crustal drilling expeditions. The JOIDES Resolution Facility Board further prioritized the testing opportunities in 2018. The top priority of all recommendations was an evaluation of drilling and coring bits because rate of penetration and bit wear andmore »tear are the prevalent issue in deep crustal drilling attempts, and bit failures often require an excessive amount of fishing and hole cleaning time. The plan included drilling in basalt with three different types of drill bits: a tungsten carbide insert (TCI) tricone bit, a polycrystalline diamond compact (PDC) bit, and a more novel TCI/PDC hybrid bit. In addition, a TCI bit was to be paired with an underreamer with expanding cutter blocks instead of extending arms. Finally, a type of rotary core barrel (RCB) PDC coring bit that was acquired for the R/V JOIDES Resolution several years ago but never deployed would also be given a test run. A second objective was added when additional operating time became available for Expedition 384 as a result of the latest schedule changes. This objective included the assessment and potential improvement of current procedures for advanced piston corer (APC) core orientation. Expedition 384 began in Kristiansand, Norway, on 20 July 2020. The location for tests was based on various factors, including the JOIDES Resolution's location at the time, our inability to obtain territorial clearance in a short period of time, and a suitable combination of sediment and igneous rock for the drilling and coring operations. IODP Expedition 395, which was postponed due to the COVID-19 pandemic, had proposed sites that were suitable for our testing and offered the opportunity to carry out some serendipitous sampling, logging, and casing work for science. We first spent 3 days triple coring the top 70 m of sediment at Site U1554 (Proposed Site REYK-6A) to obtain cores for evaluating potential problems with the magnetic core orientation tools and for assessing other potential sources of errors that might explain prior anomalous core orientation results. Comparison of the observed core orientation from magnetic orientation tools to the expected orientation based on the paleomagnetic directions recorded in the cores revealed an 180° misalignment in the assembly of one of the tools. This misalignment appears to have persisted over several years and could explain most of the problems previously noted. The assembly part was fixed, and this problem was eliminated for future expeditions. We subsequently spent 20 days at Site U1555 (Proposed Site REYK-13A) to test the three types of drill bits, an underreamer, and a coring bit in six holes. The TCI bits were the best performers, the TCI/PDC hybrid bit did not stand up to the harsh formation, and the PDC bit did not get sufficient run time because of a mud motor failure. The cutter block underreamer is not considered able to perform major hole opening in basalt but could be useful for knocking out ledges. The PDC coring bit cut good quality basalt cores at an unacceptably low rate. In the seventh and final hole (U1555G), we used a regular RCB coring bit to recover the entire 130 m basalt section specified in the Expedition 395 Scientific Prospectus and provided the project team with shipboard data and samples. The basalt section was successfully wireline logged before the logging winch motor failed, which precluded further operations for safety reasons. Additional operations plans in support of Expedition 395, including coring, logging, and casing at Site U1554, had to be canceled, and Expedition 384 ended prematurely on 24 August in Kristiansand.« less
  5. A major challenge in mobile crowdsensing applications is the generation of false (or spam) contributions resulting from selfish and malicious behaviors of users, or wrong perception of an event. Such false contributions induce loss of revenue owing to undue incentivization, and also affect the operational reliability of the applications. To counter these problems, we propose an event-trust and user-reputation model, called QnQ, to segregate different user classes such as honest, selfish, or malicious. The resultant user reputation scores, are based on both `quality' (accuracy of contribution) and `quantity' (degree of participation) of their contributions. Specifically, QnQ exploits a rating feedbackmore »mechanism for evaluating an event-specific expected truthfulness, which is then transformed into a robust quality of information (QoI) metric to weaken various effects of selfish and malicious user behaviors. Eventually, the QoIs of various events in which a user has participated are aggregated to compute his reputation score, which in turn is used to judiciously disburse user incentives with a goal to reduce the incentive losses of the CS application provider. Subsequently, inspired by cumulative prospect theory (CPT), we propose a risk tolerance and reputation aware trustworthy decision making scheme to determine whether an event should be published or not, thus improving the operational reliability of the application. To evaluate QnQ experimentally, we consider a vehicular crowdsensing application as a proof-of-concept. We compare QoI performance achieved by our model with Jøsang's belief model, reputation scoring with Dempster-Shafer based reputation model, and operational (decision) accuracy with expected utility theory. Experimental results demonstrate that QnQ is able to better capture subtle differences in user behaviors based on both quality and quantity, reduces incentive losses, and significantly improves operational accuracy in presence of rogue contributions« less