NVMe over Fabrics storage addresses a central challenge in data-intensive environments: how to make fast flash capacity available to multiple servers without giving up the low latency associated with local NVMe drives. Mission Peak, a 2018 reference design developed by Samsung, AIC, Mellanox, E8 Storage and Memorysolution, brought together NVMe SSDs, Ethernet networking and shared-storage software in a compact system. Its design offers a useful example of how storage density, network architecture and software must work together to serve demanding workloads.
Why local NVMe creates a shared-storage challenge
NVMe SSDs connect through PCI Express (PCIe), an interface designed to provide high bandwidth and direct access to the host CPU. By contrast, SATA and SAS were developed for earlier storage technologies and introduce interface and controller overheads that can limit SSD performance. NVMe also supports many parallel I/O queues, helping multiple applications and users issue commands concurrently.
But installing high-performance SSDs inside individual servers can leave capacity fragmented. Drives are available only to the server that contains them, creating separate “islands” of storage. As a result, some systems may have unused capacity while other workloads need more. Shared storage can improve flexibility, but traditional SAN, NAS and DAS approaches rely on protocols and architectures that were not designed around NVMe’s low-latency characteristics.
How NVMe over Fabrics storage extends access
NVMe over Fabrics (NVMe-oF) applies the NVMe command set across a network fabric, including Ethernet, Fibre Channel or InfiniBand. The goal is to let remote servers access NVMe storage while avoiding unnecessary protocol translation and overhead associated with legacy, HDD-oriented storage protocols. The fabric becomes part of the storage path, so its bandwidth, latency and configuration matter alongside the SSDs and host systems.
Mission Peak’s design used Ethernet and Remote Direct Memory Access (RDMA). RDMA enables data to move between systems without requiring the receiving server’s CPU to process every step through its kernel networking layers. This can reduce CPU involvement and network-path overhead, leaving more processing capacity available to applications. The actual results depend on the complete system configuration, including network links, storage software, hosts and workload.
Software architecture separates control from data movement
The Mission Peak solution paired its hardware with E8 Storage software. In this architecture, controller software handles centralized control and management, while host-side agents perform much of the data-path work. Those agents connect hosts directly to shared storage volumes using NVMe and RDMA, rather than routing every data operation through a central controller.
This division is intended to prevent the management controller from becoming a bottleneck as hosts access storage. The design also supports shared read/write volumes, which can be relevant to clustered applications such as parallel databases and file systems. The source describes support for connecting up to 126 host servers per controller and adding appliances to expand capacity. These are capabilities of the described E8 architecture, not a guarantee that every workload or deployment will scale identically.
Availability considerations are also part of the software design. Host agents operate independently for their respective servers, and the described architecture allows other hosts to continue data-path operations if one host loses access to the controller. For system designers, this highlights the importance of evaluating both the fast path for routine I/O and the behavior of management components during failures.
NF1 SSDs and the pursuit of storage density
Mission Peak was built around Samsung NF1 SSDs, a compact enterprise form factor designed to use space more efficiently than conventional 2.5-inch drives. The NF1 design added features such as hot-plug support and dual ports, according to the whitepaper, while enabling a greater number of SSDs to fit into a short-depth chassis design.
The reference system used a custom AIC backplane to accommodate up to 36 front-accessible, hot-swappable NF1 drives in a 1U chassis. A dual-socket platform and PCIe switching provided connectivity for the drive population, while expansion slots supported high-speed networking. The whitepaper described a maximum system capacity of approximately 550TB using 36 high-capacity drives. Its proof-of-concept configuration was different: it used 20 3.84TB drives, for 76.8TB of installed SSD capacity. Distinguishing the reference design’s stated maximum from the tested configuration is important when interpreting the figures.
What the proof of concept demonstrated
The described proof of concept connected four dual-socket compute nodes to a Mission Peak storage target over 100Gbps Ethernet. The target had two 100Gbps network interfaces, and the setup used Mellanox ConnectX-5 adapters and a 16-port switch. Reported measurements included 2 million 4KB random-read IOPS, 260,000 4KB random-write IOPS, 18GB/s sequential-read throughput and 13GB/s sequential-write throughput. Reported queue-depth-one latency was 70 microseconds for reads and 36 microseconds for writes.
The whitepaper cautioned that the performance test used a similar setup and that final proof-of-concept figures were not yet available. It also noted that performance was limited by the number of compute nodes in that configuration. The higher figures cited for the broader design—up to 10 million IOPS and 40GB/s read throughput—were associated with a larger host and network configuration. They should therefore be understood as stated system potential, not as measurements from the four-node proof of concept.
Practical implications for infrastructure teams
For engineers and decision-makers assessing shared flash storage, Mission Peak illustrates several design considerations:
- Plan the full data path. SSD capability alone does not determine application performance; PCIe connectivity, network bandwidth, RDMA and host-side software all contribute.
- Match capacity to workload needs. Shared storage can reduce the isolation of locally installed drives, but deployment planning should account for actual usable capacity and workload behavior.
- Validate the intended configuration. Distinguish benchmark results from projected or maximum system capabilities, and test with representative hosts, network settings and I/O patterns.
- Assess operational requirements. Consider shared-volume access, expansion, management and failure behavior alongside throughput and latency.
The whitepaper also described a remote proof-of-concept model in which customers could test pre-installed equipment at a colocation facility. In principle, this approach reduces the integration and setup work required before evaluation; the stated one-to-two-week timeframe applied to that specific offering and should not be treated as a universal PoC duration.
Conclusion
Mission Peak demonstrated how NVMe-oF, RDMA networking, purpose-built storage software and compact NF1 SSDs could be combined in a high-density shared-storage design. Its lasting engineering lesson is that low-latency shared flash depends on the architecture as a whole: storage media, host connectivity, network fabric, software data paths and failure handling must be evaluated together. For teams considering similar systems, configuration-specific testing is essential to determine whether the design meets real workload, capacity and operational requirements.