Ceph NVMe storage can deliver high performance for software-defined storage workloads, but the SSD is only one part of the system. CPU capacity, network design, OSD placement, replication and workload settings all influence cluster behavior. A reference architecture based on Red Hat Ceph Storage 3.2 and Samsung PM1725a NVMe SSDs illustrates how these elements were assembled and tested in a five-node all-flash cluster.
The results offer a useful example of what a tuned configuration achieved under defined synthetic workloads—not a universal performance guarantee. For engineers and infrastructure teams, the central lesson is to evaluate the full Ceph data path and match configuration decisions to the intended workload.
How Ceph distributes and protects data
Ceph presents clients with logical storage pools, while the cluster handles object placement and the work of storing, replicating and recovering data. Clients use the CRUSH algorithm to determine where objects belong, communicating directly with the relevant object storage daemons (OSDs) rather than routing every operation through a centralized broker.
Objects are mapped to placement groups (PGs), which provide a practical unit for distributing data across OSDs. Pools define characteristics such as replication or erasure coding, while the CRUSH map governs placement within the cluster. Monitor daemons maintain the cluster map; OSDs store data and manage replication, recovery, backfilling and rebalancing.
This distributed design makes the cluster’s behavior dependent on more than drive speed. The number of OSDs, PG configuration, CPU resources, network capacity and recovery activity all contribute to how a workload performs.
Why BlueStore matters in an all-flash cluster
The reference architecture uses BlueStore, Ceph’s OSD backend in the tested release. Unlike the older FileStore approach, BlueStore writes objects directly to block devices rather than placing a local file system between the OSD and the storage device. Its metadata is managed with RocksDB, and BlueStore verifies data and metadata using checksums by default.
BlueStore also supports copy-on-write operations used for snapshots and can use multiple devices for different roles. These design features shape the I/O path and the management of metadata. They do not remove the need to tune and validate a deployment: the paper’s results were obtained with specific software versions, hardware and Ceph parameters.
Inside the Ceph NVMe storage reference architecture
The test cluster consisted of five x86 storage nodes, each equipped with ten 1.6 TB PM1725a NVMe SSDs. The configuration ran Red Hat Ceph Storage 3.2 (Luminous 12.2.8) with BlueStore, two OSDs per SSD, 80 OSD daemons in total, 8,192 placement groups and a 2x replication factor. Each node also had dual Xeon Gold 6152 processors, 128 GB of DRAM and a 100 GbE dual-port network adapter.
Two separate 100 GbE networks connected the nodes: a public network for client and monitor traffic, and a cluster network for OSD heartbeats, replication, peering and recovery. This separation makes those traffic roles explicit and reflects an important planning consideration for throughput-oriented Ceph deployments.
The benchmark used ten client servers and a RADOS Block Device (RBD) pool. The clients created 200 images of 75 GB each, representing 15 TB of logical data; 2x replication brought the stored dataset to 30 TB. Fio generated the workloads, and the reported results were averages across three runs.
Benchmark results: read, write and latency
At a queue depth of 32, the cluster reported average 4 KB random-read throughput of 2,255 KIOPS, with average latency of 2.85 ms and 99.99th-percentile latency of 153.21 ms. For 4 KB random writes at the same queue depth, average throughput was 493 KIOPS, average latency was 12.97 ms and 99.99th-percentile latency was 74.20 ms.
As queue depth increased, both random-read and random-write throughput rose, as did latency. The reported tail latency increased notably at higher queue depths: from QD8 for random reads and QD16 for random writes. That distinction matters when assessing storage for latency-sensitive applications. Peak IOPS alone does not describe the experience of requests near the tail of the latency distribution.
For 128 KB sequential workloads across 200 clients, average throughput reached 18.8 GB/s for writes and 51.6 GB/s for reads. Throughput leveled off as client count increased, while average latency continued to rise. This illustrates why capacity planning should consider concurrency and response time alongside aggregate bandwidth.
Planning considerations for deployment
The reference design points to several variables to validate before deploying an all-flash Ceph cluster:
- Node and monitor layout: The guidance recommends at least three storage nodes for redundancy and an odd number of monitors; three or more monitors are typical for production clusters.
- Replication policy: The benchmark used 2x replication. Replication choices affect data protection and usable capacity and should be made for the application’s requirements.
- CPU and OSD density: OSDs consume CPU for placement and data movement. Too many OSDs per SSD can also introduce context-switching overhead, so density should be tested rather than assumed.
- Network capacity: The guidance calls for at least 10 GbE for an NVMe-based cluster and recommends 50–100 GbE for throughput-oriented workloads, with separate public and cluster networks.
- Operating-system and Ceph tuning: Kernel, network and Ceph settings can affect the I/O path. The tested configuration used tuned parameters, so its results should not be attributed to hardware alone.
What the results mean for storage design
This five-node reference architecture demonstrates that an all-flash Ceph cluster can combine high random-read IOPS with substantial sequential bandwidth when its storage, compute, network and software configuration are designed together. Its measured performance belongs to the specific Red Hat Ceph Storage 3.2 environment and benchmark method described here; results can vary with server configuration, settings and workload.
For practical planning, use the figures as a reference point, not a sizing promise. Define the target workload, test realistic queue depths and concurrency, monitor average and tail latency, and validate network and CPU headroom alongside drive performance. That system-level approach is essential to translating Ceph NVMe storage capability into dependable application performance.