A custom storage backend can give a distributed storage system control over parts of the I/O path that a general-purpose local file system cannot easily expose. Ceph’s experience offers a practical case: after a decade of using file-system-based storage, the project developed BlueStore to manage raw devices directly. The change addressed persistent challenges involving transactions, metadata performance and support for emerging storage interfaces.
Why distributed storage backends matter
A distributed storage system combines devices across multiple machines into a shared service. On each storage node, backend software manages the locally attached devices and processes I/O requests. Its choices affect how data is persisted, how metadata is organized and how operations such as recovery, replication and object updates are carried out.
Using a local file system such as XFS or ext4 has clear advantages. Mature file systems provide established code for persistence and block allocation, familiar file and directory abstractions, and standard tools for inspecting on-disk contents. But a distributed system may need behaviors that do not map cleanly to those abstractions. Ceph’s experience highlights three especially important friction points.
Three limitations of file-system-based backends
Transactions require extra machinery
Distributed storage software often needs to group several changes into one atomic transaction. POSIX file operations do not generally provide that transaction model. A backend built on a local file system must therefore find another way to make a sequence of updates consistent.
Ceph’s earlier FileStore backend used a user-space write-ahead log (WAL). This brought its own costs. A transaction had to be logged and synchronized before its operations could be applied, limiting the ability to pipeline read-modify-write work. Recovery was also complicated by operations such as renames and clones, which are not always safe to replay if some changes were already applied before a crash. In addition, journaling data in the WAL and then writing it to the file system could result in double writes.
Using a key-value database for metadata and logging can simplify some of these issues, but combining it with a journaling file system may add further synchronization work. In Ceph’s NewStore design, file data and database metadata required separate synchronous steps. The broader lesson is that transaction costs depend on the entire storage stack, not just the application-level log.
Local metadata behavior can become a cluster bottleneck
Operations that look like ordinary file-system housekeeping can have a system-wide impact when repeated across many storage nodes. Ceph needed to enumerate large collections of objects in a predictable order. FileStore represented collections as directories, but large directory scans and the need to split directories as they grew created expensive metadata work. The resulting load could reduce throughput while storage nodes processed the directory hierarchy.
This matters beyond Ceph. A backend’s metadata layout should be evaluated against the distributed system’s actual operations, including enumeration, recovery, scrubbing and rebalancing. Fast sequential data access alone does not guarantee that metadata-heavy operations will scale.
New device interfaces challenge general-purpose file systems
Some storage devices expose interfaces that differ from the conventional block-device model. The whitepaper discusses host-managed shingled magnetic recording (SMR) drives and Zoned Namespace (ZNS) SSDs. These devices use zones with write constraints that favor sequential, copy-on-write approaches. That can conflict with the in-place overwrite assumptions of established file systems.
When a distributed system depends on a local file system, adopting such hardware may depend on changes to that file system first. A purpose-built backend can instead be designed around the device interface, though this shifts more responsibility for storage management into the backend itself.
How BlueStore changes the design
BlueStore was designed to run in user space directly on raw storage devices. It separates object data from metadata: data is written to the device, while metadata is maintained in RocksDB. RocksDB runs on BlueFS, a minimal user-space file system tailored to its needs. This is not a general replacement for POSIX file systems; it is a specialized layer built to support the database BlueStore relies on.
Direct control of the I/O path lets BlueStore coordinate data and metadata handling without relying on a general-purpose file system to provide the required behavior. The design uses direct I/O for data and changes to RocksDB’s WAL handling to reduce synchronization overhead. It also stores metadata in key-value namespaces, making it possible to organize object and allocation information separately from the layout of ordinary files and directories.
BlueStore uses copy-on-write for larger writes: new data is placed in a newly allocated extent, and metadata is updated after the data is persisted. Cloning can then share extents through reference counting rather than copying all of the data immediately. Smaller writes follow a deferred-write path, which can batch operations and adapt write behavior to HDDs and SSDs.
Space management is another explicit backend responsibility. BlueStore tracks allocated space persistently and maintains in-memory structures to find free regions. The whitepaper describes a bitmap-based allocator whose memory use is fixed at 35 MiB per terabyte of capacity. Taking on these responsibilities makes a custom backend more specialized, but also gives it greater influence over allocation and I/O behavior.
Practical considerations for infrastructure teams
Ceph’s experience does not mean every distributed storage platform should bypass a local file system. Mature file systems remain valuable when their abstractions and performance characteristics fit the workload. The case for a custom backend becomes stronger when the system’s consistency model, metadata access patterns or target hardware repeatedly conflict with those abstractions.
When assessing storage architecture, technical teams should consider:
- Whether the backend can provide the atomic update behavior required by the distributed system.
- How local metadata operations behave at the scale and concurrency of production workloads.
- Whether logging and synchronization cause redundant writes or limit read-modify-write throughput.
- How the design will handle copy-on-write operations, snapshots and evolving device interfaces.
- Whether the team can support the additional complexity of managing storage directly.
Conclusion
BlueStore demonstrates the potential of a custom storage backend: when software controls the I/O path, it can align data placement, metadata management and transaction handling with the needs of a distributed system. That control also brings implementation and operational responsibility. For engineers and infrastructure decision-makers, the key is to judge the backend against real workload requirements and hardware plans—not simply to assume that a general-purpose file system is either always sufficient or always a bottleneck.