skip to main content


This content will become publicly available on May 8, 2024

Title: Chipmunk: Investigating Crash-Consistency in Persistent-Memory File Systems
We present Chipmunk, a new framework to test persistent-memory (PM) file systems for crash-consistency bugs. Using Chipmunk, we discovered 23 new bugs across five PM file systems; most bugs have been confirmed and fixed by developers. The discovered bugs have serious consequences, including making the file system un-mountable or breaking rename atomicity. We present a detailed study of the bugs found using Chipmunk and discuss important lessons learned for designing and testing PM file systems.  more » « less
Award ID(s):
1751277
NSF-PAR ID:
10469308
Author(s) / Creator(s):
; ; ; ; ;
Publisher / Repository:
ACM
Date Published:
Page Range / eLocation ID:
718 to 733
Format(s):
Medium: X
Location:
Rome Italy
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    Persistent Memory (PM) can be used by applications to directly and quickly persist any data structure, without the overhead of a file system. However, writing PM applications that are simultaneously correct and efficient is challenging. As a result, PM applications contain correctness and performance bugs. Prior work on testing PM systems has low bug coverage as it relies primarily on extensive test cases and developer annotations. In this paper we aim to build a system for more thoroughly testing PM applications. We inform our design using a detailed study of 63 bugs from popular PM projects. We identify two application-independent patterns of PM misuse which account for the majority of bugs in our study and can be detected automatically. The remaining application-specific bugs can be detected using compact custom oracles provided by developers. We then present AGAMOTTO, a generic and extensible system for discovering misuse of persistent memory in PM applications. Unlike existing tools that rely on extensive test cases or annotations, AGAMOTTO symbolically executes PM systems to discover bugs. AGAMOTTO introduces a new symbolic memory model that is able to represent whether or not PM state has been made persistent. AGAMOTTO uses a state space exploration algorithm, which drives symbolic execution towards program locations that are susceptible to persistency bugs. AGAMOTTO has so far identified 84 new bugs in 5 different PM applications and frameworks while incurring no false positives. 
    more » « less
  2. We present SplitFS, a file system for persistent memory (PM) that reduces software overhead significantly compared to state-of-the-art PM file systems. SplitFS presents a novel split of responsibilities between a user-space library file system and an existing kernel PM file system. The user-space library file system handles data operations by intercepting POSIX calls, memory-mapping the underlying file, and serving the read and overwrites using processor loads and stores. Metadata operations are handled by the kernel PM file system (ext4 DAX). SplitFS introduces a new primitive termed relink to efficiently support file appends and atomic data operations. SplitFS provides three consistency modes, which different applications can choose from, without interfering with each other. SplitFS reduces software overhead by up-to 4× compared to the NOVA PM file system, and 17× compared to ext4 DAX. On a number of micro-benchmarks and applications such as the LevelDB key-value store running the YCSB benchmark, SplitFS increases application performance by up to 2× compared to ext4 DAX and NOVA while providing similar consistency guarantees. 
    more » « less
  3. null (Ed.)
    High development velocity is critical for modern systems. This is especially true for Linux file systems which are seeing increased pressure from new storage devices and new demands on storage systems. However, high velocity Linux kernel development is challenging due to the ease of introducing bugs, the difficulty of testing and debugging, and the lack of support for redeployment without service disruption. Existing approaches to high-velocity development of file systems for Linux have major downsides, such as the high performance penalty for FUSE file systems, slowing the deployment cycle for new file system functionality. We propose Bento, a framework for high velocity development of Linux kernel file systems. It enables file systems written in safe Rust to be installed in the Linux kernel, with errors largely sandboxed to the file system. Bento file systems can be replaced with no disruption to running applications, allowing daily or weekly upgrades in a cloud server setting. Bento also supports userspace debugging. We implement a simple file system using Bento and show that it performs similarly to VFS-native ext4 on a variety of benchmarks and outperforms a FUSE version by 7x on 'git clone'. We also show that we can dynamically add file provenance tracking to a running kernel file system with only 15ms of service interruption. 
    more » « less
  4. Fast, byte-addressable persistent memory (PM) is becoming a reality in products. However, porting legacy kernel file systems to fully support PM requires substantial effort and encounters the challenge of bridging the gap between block-based access granularity and byte-addressability. Moreover, new PM-specific file systems remain far from production-ready, preventing them from being widely used. In this paper, we propose P2CACHE, a novel in-kernel caching mechanism to explore how legacy kernel file systems can effectively evolve in the face of fast, byte-addressable PM. P2CACHE exploits a read/write-distinguishable memory hierarchy upon a tiered memory system involving both PM and DRAM. P2CACHE leverages PM to serve all write requests for instant data durability and strong crash consistency while using DRAM to serve most read I/Os for high I/O performance. Further, P2CACHE employs a simple yet effective synchronization model between PM and DRAM by leveraging device-level parallelism. Our evaluation shows that P2CACHE can significantly increase the performance of legacy kernel file systems -- e.g., by 200x for RocksDB on Ext4 -- meanwhile equipping them with instant data durability and strong crash consistency, similar to PM-specialized file systems. 
    more » « less
  5. Abstract Background

    Widespread vector control has been essential in reducing the global incidence and prevalence of malaria, despite now stalled progress. Long-lasting insecticide-treated nets (LLINs) have historically been, and remain, one of the most commonly used vector control tools in the campaign against malaria. LLINs are effective only with proper use, adherence, retention and community adoption, which historically have relied on the successful control of secondary pests, including bed bugs. The emergence of pyrethroid-resistant bed bugs in malaria-endemic communities and failure to control infestations have been suggested to interfere with the effective use of LLINs. Therefore, the behavioral interactions of bed bugs with commonly used bed nets should be better understood.

    Methods

    To investigate the interactions between bed bugs (Cimex lectulariusL.) and LLINs, insecticide-susceptible and pyrethroid-resistant bed bugs were challenged to pass through two commonly used LLINs in two behavioral assays, namely host (blood meal)-seeking and aggregation-seeking assays. The proportions blood-fed and aggregated bed bugs, aggregation time and mortality were quantified and analyzed in different bed bug life stages.

    Results

    Overall, both the insecticide-susceptible bed bugs and highly resistant bed bugs showed a varying ability to pass through LLINs based on treatment status and net design. Deltamethrin-treated nets significantly impeded both feeding and aggregation by the susceptible bed bugs. While none of the tested LLINs significantly impeded feeding (passage of unfed bed bugs through the nets) of the pyrethroid-resistant bed bugs, the untreated bed net, which has small mesh holes, impeded passage of fed bed bugs. Mortality was only seen in the susceptible bed bugs, with significantly higher mortality on deltamethrin-treated nets (63.5 ± 10.7%) than on permethrin-treated nets (2.0 ± 0.9%).

    Conclusions

    Commonly used new LLINs failed to prevent the passage of susceptible and pyrethroid-resistant bed bugs in host- and aggregation-seeking bioassays. The overall low and variable mortality observed in susceptible bed bugs during both assays highlighted the potential of LLINs to impose strong selection pressure for the evolution of pyrethroid resistance. Already, the failure to control bed bug infestations has been implicated as a contributing factor to the abandonment or misuse of LLINs. For the first time to our knowledge, we have shown the potential of LLINs in selecting for resistant secondary pest populations and so their potential role in stalling malaria control programs should be further investigated.

    Graphical Abstract

    The emergence of pyrethroid-resistant bed bugs in malaria-endemic communities may interfere with the effective use of pyrethroid-impregnated bed nets. We assessed the interactions of two bed bug strains with commonly used bed nets using two behavioral assays, namely host (blood meal)-seeking by unfed bed bugs and aggregation-seeking by freshly fed bed bugs. These assays assessed the passage of bed bugs through various bed nets in response to host cues and aggregation stimuli, respectively. Conditioned paper is a section of file folder paper that has been exposed to bed bugs and has been impregnated with feces and aggregation pheromone; it is attractive to aggregation-seeking fed bed bugs. An unconditioned ramp is a similar section of file folder paper that allows bed bugs to traverse the bed net and gain access to a blood-meal source.

     
    more » « less