Ceph’s move to BlueStore illustrates the trade-offs of building distributed storage on local file systems. Direct device access gives a purpose-built backend more control over transactions, metadata, I/O behavior and support for changing storage hardware.