The landscape of modern storage is undertaking a disruptive change. The emergence of new storage technologies, such as persistent memory [13], CXL SSD [4,5,29], and others [8,10,35], are producing faster, larger, and cheaper storage devices. These devices o!er new bandwidth, latency, and capacity features, such as near-memory speed, byte addressability, and cache coherence. Accordingly, many new, specialized "le systems are being actively developed to fully utilize these new storage tiers [11-14, 20, 27, 30, 33, 34].
New devices are commonly integrated into heterogeneous storage hierarchies [17,18,21,29] for cost e$ciency and practical constraints (e.g., hardware acquisition and deployment). Tiered "le systems [16, 19,26,31,32] are a common design for heterogeneous storage. They expose the storage hierarchy as a single device to the host system and leverage the characteristics of each tier (device) internally to dynamically migrate "les and blocks across tiers, minimizing latency and maximizing throughput while exposing large capacity.
We argue that the current practice of developing tiered "le systems cannot keep up with the pace of storage innovation. First, their development inevitably lags behind devicespeci"c "le systems, as it involves integrating new device support and new tiering policies on every storage hierarchy extension. Second, it has become increasingly hard to cover all possible device con"gurations found in real-world storage infrastructures into one monolithic tiered "le system.
In this paper, we propose to solve this problem using Mux, a new tiered "le system that accesses di!erent device types indirectly through device-speci"c "le systems, rather than directly through device drivers. As shown in Figure 1, Mux slots between the Virtual File System (VFS) and device-speci"c, native "le systems; it realizes tiering policies (e.g., data placement, migration, and replication) and delegates I/O requests to native "le systems. The design leverages the opportunity that native "le systems are highly optimized and can be directly used as building blocks of a tiered "le system. Di!erent from existing meta "le systems that combine native "le systems such as OverlayFS [6], MergerFS [7], and UnionFS [24], Mux exposes the entire hierarchy as a single device to the host and transparently dispatches reads and writes to native "le systems as per tiering policies. Mux distributes an individual "le across tiers; it bookkeeps which "le system stores the "le blocks. The block distribution can happen both synchronously at allocation time (e.g., by append) and asynchronously by block-level data migration.
The design treats extensibility-seamless integration of new device types-as a "rst-class principle by decoupling tiering from device management. To integrate new devices, dedicated "le systems can be plugged directly into the stack through a well-de"ned interface (e.g., Linux VFS), without modi"cation. To extend tiering policies, Mux o!ers a modular interface (via kernel module or eBPF) for users to register tiering rules or device pro"les. We contend that the separation of concerns-tiering and device specialization-would enable progressive evolution and #exible integration of heterogeneous "le and storage systems.
Mux must address correctness challenges. Given the distribution of a "le across "le systems, which participating "le systems are in charge of maintaining "le metadata upto-date? This metadata ownership challenge is unique, as a monolithic tiered "le system stores a single instance of all attributes. We opt for distributing the metadata maintenance. We introduce metadata a"nity to assign each metadata attribute to a "le system owner. For example, the "le system that stores the last byte of the "le is the owner of the logical "le size and the "le system that performed the last update is the owner of the last modi"ed timestamp. Mux bookkeeps which "le system is the owner of an attribute and lazily synchronizes participating "le systems. For metadata that cannot have a single owner such as disk consumption, Mux manages them across all related "le systems.
Mux must also guarantee that data movement (e.g., for migration) across "le systems does not interfere with user access-users' updates on blocks during migration are not lost or overwritten. The problem is also faced by monolithic tiered "le systems and is commonly solved by locking the "le or blocks in #ight [16,31,32]. Di!erently, in Mux, migrations involve multiple native "le systems, but no universal lock among them exists. Instead of global locks, we opt for optimistic concurrency control [15]. Mux maintains a versioning counter per "le to detect potential con#icts based on version di!erences, therefore minimizing the impact of con#ict checking on the critical path.
Despite introducing an additional indirection layer, Mux can directly bene"t from device-speci"c optimizations of native "le systems. This bene"t can outweigh the indirection overhead in practice. We compare Mux which uses NOVA for persistent memory (PM), XFS for SSD, and Ext4 for HDD, with Strata [16], a research tiered "le system. Mux achieves 1.46x higher throughput for device I/O and 2.59x faster data migration over Strata, because it utilizes specialized production-ready "le systems (e.g., device-friendly journaling and caching scheme). Compared with direct access to per-device "le systems (with no tiering), Mux adds a worst-case read latency overhead of 6.6% to 87.3%, as well as a write throughput overhead of 1.6% to 3.5%, across NOVA, XFS, and Ext4 on PM, SSD, and HDD, respectively.
We describe the design of Mux and its preliminary results, and discuss a few open problems. A full-#edged implementation of Mux is in progress.
We describe the Mux design that composes individual "le systems into a tiered "le system. The overarching goal is to aggregate the advantages of underlying device-speci"c "le systems, while o!ering extensible tiering policies.
Mux must:
• manage the asynchronous distribution of data blocks (of "les) across heterogeneous storage devices, • guarantee the consistency of "le metadata and data, • dispatch user requests as individual per-device I/O operations and merge their results into uni"ed responses,
/dir file_A file_B file_C file_A:
Block Lookup
Table File System 1 File System 2 blk1 2 3 4 5 6 ... file_A (sparse file) 3 ... 1 4 /dir file_A file_B /dir file_A file_C ... 2 5 6 file_A (sparse file) User's view: User-defined Policy -permission -mtime -... Metadata Mux. Meta Cached FS Meta -file size -atime -... Metadata Figure 2: An overview of Mux.
• provide a user interface to specify or extend tiering rules, • orchestrate shared "le caching,
• eliminate races between user requests and asynchronous data movement (e.g., for migration).
Our performance goal is to match or exceed traditional "monolithic" tiered "le systems, given the same tiering policy.
Mux slots between the VFS layer and device-speci"c "le systems (Figure 1). Figure 2 shows the components of Mux and how "le metadata and data are mapped to device-speci"c "le systems.
The main challenge is the crossing of "le system boundaries. In Mux, a "le is typically distributed across multiple "le systems, creating challenges for its indexing and metadata management. At the same time, data movement across "le systems must not race or interfere with user requests, but the global locking is missing. Mux introduces a set of global metadata to bookkeep "le distribution and devise e$cient synchronization mechanisms.
Mux provides a uniform namespace to user applications. The underlying "le systems selected by a user are mounted at di!erent mount points. Mux merges these mount points in a single directory tree and presents it to user applications (Figure 2). A "le can be distributed across multiple "le systems; the same "le name exists in di!erent "le systems. The design is inspired by OverlayFS [6] which overlays one "le system on top of another, presenting a merged directory view to applications. In OverlayFS, all data updates are applied to the topmost "le system. We extend this design to enable "ner-grained and dynamic selection of the underlying "le systems, including write operations.
To add a new device and the corresponding "le system, the user only needs to mount the new "le system and register it with Mux, along with a policy to manage it. To remove a device, data must be migrated "rst. Adding or removing a device can be done at runtime.
Mux is presented to the VFS layer as a standalone "le system, making the OS send "le operations to Mux through the existing VFS interface. Upon receiving a "le operation, Mux splits it based on the block-to-underlying "le system mappings, if required. Mux sends the split requests to devicespeci"c "le systems by calling the same VFS function that invokes it, but with di!erent "le handles, lengths, and o!sets.
User-De!ned Policy. Mux decouples tiering policies from "le system implementation. It exposes an interface for users to specify policies on data placement and user request dispatching. All the placement and migration policies in existing tiered "le systems [16, 19,26,31,32] can be expressed using simple functions. For example, the data placement policy of TPFS [32] can be simply implemented by a function that returns di!erent device IDs based on the I/O size, synchronicity, and access history. Currently, the policy is encoded as a kernel module or an eBPF extension so the policy functions can be directly called.
Mux can support block-level data distribution and place data #exibly as per user-de"ned policies. For example, LRU-based or Hot-Cold block-level data migration schemes employed by existing tiered "le systems [16, 31,32] can be easily supported by Mux.
Block-level data distribution requires Mux to maintain the mapping from a block to the underlying "le systems (a "le system's internal index is invisible to Mux). We use a mapping table, named Block Lookup Table, to locate a block of a "le, as shown in Figure 2. Mux maintains a per-"le block lookup table and tracks in which device the recent version of a block is stored. Since the table maps "le o!sets to devices, that are small in size, we use an extent tree as a high-performance data structure.
Note that a "le can be stored on multiple devices as a result of load balancing and data migration. Mux leverages sparse "les to preserve the "le o!set of a block across di!erent underlying "le systems (Figure 2). It enables Mux to avoid an additional translation while preventing space waste from allocating the entire "le.
Managing "le blocks across "le systems creates a unique challenge. Which "le system should keep the metadata attributes up to date, given that multiple ones may participate in metadata-update operations (e.g. appends)? We opt for metadata multiplexing and introduce the notion of metadata a"nity. For each metadata attribute, there is an a"nitive #le system at any given point in time, that holds the most up-to-date value for the attribute. This design reduces small writes due to metadata updates and avoids waiting on slow devices for synchronization.
For example, when the "le is created, the host "le system is the a$nitive for all metadata. Later, if another "le system performs an append guided by Mux (e.g., tiering the new blocks to multiple devices) the "le system that allocated the last block of the operation becomes the a$nitive for the "le size attribute. During a write operation, the "le system that overwrites the last block of the operation becomes the a$nitive for the last update timestamp. During a read operation, the "le system that fetches the last block of the operation becomes the a$nitive for the last access timestamp, etc. Mux bookkeeps the a$nitive "le system per attribute, to orchestrate incoming requests.
Mux caches metadata attributes in a collective inode, avoiding fetching metadata from di!erent "le systems (which is expensive in the multiplexed design).
Mux maintains its own metadata like block lookup table, "le a$nity table, etc. The space overhead of such metadata is marginal. For example, in a block lookup table, one byte per 4 KB of user data is su$cient with a simple byte array, leading to less than 0.025% of space overhead.
In heterogeneous storage, data movement is critical to performance and cost e$ciency. Traditional tiered "le systems realize transparent migration by implementing intra-"le system locking. It enables the migration and application's "le operations to be performed atomically with respect to each other. However, such locking is insu$cient in Mux. In Mux, blocks are moved from one "le system to another, which requires a synchronization mechanism across the "le systems. Moreover, applications' "le operations are an external operation to underlying device-speci"c "le systems. Hence, Mux requires a new synchronization mechanism to support transparent migration.
We design the synchronization mechanism using optimistic concurrency control [15], which is called OCC Synchronizer. Our insight is that data movement does not change the content of the data; so, a data movement process is considered successful if the content of the data remains unchanged throughout the process. With this principle, we introduce a version number to each "le to identify the potential con#icts. The version number is incremented at the start and end of a data movement process, allowing Mux to detect the possibility of a con#ict while handling an application's "le write by comparing the version number before and after the write operation and checking a migration #ag on the "le. If a potential con#ict is detected, Mux searches for blocks that have changed during migration. If any blocks are found to have been modi"ed, Mux retries the migration of those blocks. If all tasks that ran during the migration process are con#ictfree, the migrated block will be atomically committed and made visible in the block lookup table. If OCC Synchronizer keeps failing to migrate the block, Mux will resort to a lockbased migration. This scheme minimizes the critical path of user requests and enables the parallel execution of migration without pessimistic blocking or lock contention.
Since a data block migrated using OCC Synchronizer is only atomically made visible when proven to be con#ictfree, it avoids data corruption due to data race. Also, since any tasks that could generate a con#ict are checked before committing the block, and those tasks are proven not to have modi"ed the migrated data block, the migrated data is also guaranteed to be up-to-date. If any con#ict occurs, the migrated data block will be dropped and overwritten in place in the next migration attempt. This ensures that there are no side e!ects from failed attempts. OCC Synchronizer also guarantees that the migration process will be completed in a "nite amount of time, due to the limited number of retries, thus ensuring the replication lag is bounded.
Mux includes a cache manager to use Storage Class Memory (SCM) devices (e.g., persistent memory) as caching devices. Recent studies [9,16,19] show that using SCM for caching improves storage performance, as SCM has a large capacity and is accessible via memory operations. Existing tiered "le systems [16,31] assume SCM-based caches.
In the Mux design, while each "le system may use DRAM as its page cache, the cache cannot be shared across devices. Moreover, as storage continues to grow, DRAM is di$cult to scale. Using SCM devices to o%oad DRAM page caches helps alleviate the scalability problem. SCM "le systems [12-14, 20, 27, 33, 34] commonly use the DAX (Direct Access) interface, where memory mapping a "le provides direct access to the physical storage, minimizing software overhead. Mux uses DAX to implement data cache to minimize software overhead.
Mux can create one "le for all caches, which helps reduce the overhead of managing multiple "les as well as disk fragmentation. Alternatively, Mux can preallocate the cache "le to ensure cache availability and reduce block allocation overhead. Mux uses DAX memory mapping for the cache "le. If the caching memory pool is depleted, the data can be written back or evicted. We use Multi-generational LRU [3] for cache replacement, which is also the algorithm Linux uses for its page caches.
We report the performance of an early prototype of Mux, in comparison to Strata, a traditional tiered "le system. We also report the overhead of Mux over native device-speci"c "le systems when accessing "les directly (without tiering).
We use a storage hierarchy that consists of Intel Optane Persistent Memory 200 (as PM), Intel Optane SSD DC P4800X (as SSD), and Seagate Exos X18 (as HDD), paired with Intel Xeon 6346 CPU and 256GB DRAM. We use a prototype of Mux that utilizes specialized production-ready "le systems, including NOVA [27] for PM, XFS [25] for SSD, and Ext4 [22] for HDD. We compare Mux with Strata [16], a research tiered "le system. Mux and Strata use the same tiering policy to manage the storage hierarchy.
Strata follows the traditional tiered "le system design. It is designed as a monolithic "le system that manages all three devices internally. Our Mux prototype implements the tiering mechanism internally, but delegates the I/O to NOVA, XFS, and Ext4. We implement a simple LRU policy that evicts cold data to the slower device if no space left on faster devices, and promotes data back upon access.
We focus on data migration, which is arguably the most basic tiering feature. Mux directly supports data migration between any pair of devices (Figure 3a). As VFS abstracts out device-speci"c details, supporting a migration path takes a single line of code to invoke the migration function. However, this is not the case for Strata, where adding a path requires manually matching the threading model, block size, and call context of the paired devices. If two devices are not wired, data migration between them either is unsupported (N/S) or must be relayed via other devices. Among all six migration pairs, Strata supports only two (Figure 3a). Also, it supports neither SSD to HDD demotion nor any kind of promotion.
We measure the maximum throughput of data migration between each device pair by continuously writing data to saturate upper-tier devices, forcing data migrations. As shown in Figure 3a, the throughputs of inter-device data migration in Mux are much higher than those in Strata. Speci"cally, the throughput of PM to SSD migration is increased by 2.59x. We also measure the I/O throughput of individual devices. We run Strata's microbenchmark [2] with 90GB random writes and measure the throughput when the I/O request is always directed to the target devices. As shown in Figure 3b, the I/O throughput of Mux is 1.08x, 1.46x, and 1.07x higher for PM, SSD, and HDD respectively over Strata.
We attribute the performance advantage of Mux over Strata for two reasons. First, Mux leverages the device-speci"c optimizations provided by the underlying "le systems. For example, we observe that Strata "rst writes data to a log
Strata Mux (NOVA, xfs, ext4) N/S: Not Supported Numbers are reported in MB/s Source Device Target Device Source Device Target Device (a) Extensibility P M S S D H D D 0 100 200 300 400 500 600 Throughput (MB/s) Strata Mux (b) Performance on persistent memory and then digests the log to actual "le blocks on "nal storage devices. However, such logging is not necessary on persistent memory devices. The optimized "le system we use for PM, namely NOVA, is di!erent in that it writes data directly to the "le blocks using DAX and uses the CLFLUSH instruction to guarantee data consistency. Hence, Strata's design results in write ampli"cation and performance degradation. Second, unlike Strata, Mux uses independent "le systems to drive devices. In Strata, the "le extent tree that contains both block o!set and device index has to be partially locked during block-level data migration, which may prevent access to blocks that do not require migration. In contrast, since Mux works on top of device-speci"c "le systems and has its own separate meta"le storage, locking "le blocks for migration will not a!ect access to other blocks. We acknowledge that Strata is a research system and its performance may not represent an ideal tiered "le system. On the other hand, Mux uses production-ready, highly optimized device-speci"c "le systems, which shows the advantage of the Mux design-the extensibility allows Mux to directly bene"t from device-speci"c optimizations of well-developed "le systems.
Mux explores a tradeo!: introducing an additional "le system layer above the device-speci"c "le systems to get the performance bene"t of per-device specialization for free. One may suspect an 100% overhead of Mux, as it doubles the work of "le systems. To understand the worst case of the indirection overhead, we run a microbenchmark that repeatedly reads one single byte from a 10GB "le randomly. Compared to the native "le systems-NOVA, XFS, and Ext4 (with no tiering), Mux increases the latency by 52.4%, 87.3%, and 6.6% on PM, SSD, and HDD, respectively.
We also run a microbenchmark that repeatedly writes four megabytes to a "le sequentially to evaluate write throughput overhead. Compared to the native "le systems, Mux decreases the throughput by 1.6%, 2.2%, and 3.5% on PM, SSD, and HDD, respectively.
Note that some of the above overhead may also appear in an ideal tiered "le system; how to quantify the Mux overhead over an ideal implementation remains our future work.
With the initial prototype that shows the promises, we are developing a full-#edged Mux. We discuss several challenging problems we encounter and ideas to address them. Crash Consistency. Currently, the crash consistency properties of Mux are composed of those of the participating "le systems. Mux sends fsync requests to all the "le systems that are responsible for a given "le and synchronizes the completion of the fsync operations. Upon a crash, Mux relies on each participating "le system to recover the data blocks it stores. However, di!erent "le systems have di!erent crash consistency guarantees [23] and the overall crash consistency of a "le is a!ected by defects in any participating "le system, assuming no data replication. We believe that a much stronger crash consistency guarantee can be designed for Mux by utilizing the diversity of device-speci"c "le systems and by the opportunity for data replication across devices. Feature Imparity of File Systems. A common thread in the Mux design is the di!erence between di!erent "le systems, despite that they all implement the VFS interface. For example, "le systems often have special features that the others do not. Some "le systems extend metadata attributes in uncommon ways. Oftentimes, even for the same metadata attribute, its semantics can vary (e.g., FAT records timestamps with a two-second granularity [1]). We currently do not consider any extended metadata attributes or special features. How to enable users to e!ectively manage these system-speci"c features and metadata is an open problem. Improving The I/O Scheduler. High-performance I/O often requires I/O schedulers to maximize resource utilization and minimize contention. The I/O scheduler should identify request types, estimate their costs, and reorder them to optimize performance [28]. We currently use a simple scheduling algorithm based on device pro"les (performance characteristics and feature sets). We expect more intelligent schedulers to be developed, that consider dynamic states such as load spikes on devices as well as instantaneous I/O activities for garbage collection and in-device DRAM #ushing to SSDs. Con!guring Mux. As the Mux design can easily integrate many existing "le systems, an emerging problem is how to "nd the best con"guration of "le systems for a given workload or a given set of storage devices. Sharing Mux among multiple applications may also require scheduling schemes that support priority, deadline, and/or quota, which may dispatch I/Os and accessed data blocks to "le systems with di!erent performances, or ensure that high-priority tasks are not impeded by reordering and splitting requests.
Distributed Mux. One ambitious idea is to extend Mux in a distributed manner. By designing a Mux-to-Mux interconnection (e.g., through Remote Procedure Call) at the Mux layer and a distributed tiering policy, it is possible that a set of machines mounting traditional "le systems can be integrated into a distributed storage system. Certainly, to realize that, we will need to address many open problems, some of them are: (1) how to achieve transparency across the OS boundary? (2) how to support dynamic joining and leaving of nodes? and (3) how to handle latency variations and network faults? We plan to start with attaching networked "le systems as one of the underlying "le systems.