Access Official Technical PDF

Share Summary:LinkedInEmail

Data center hardware reliability is not simply a matter of choosing components with low expected failure rates. In large, distributed environments, a fault in one device can affect interdependent software running across many servers. Field measurements of DRAM, solid-state drives (SSDs) and network equipment show why operators need to understand how failures arise, how they surface in workloads, and which mitigations are effective.

A large-scale study of Facebook’s production infrastructure examined memory errors over 14 months, SSD behavior over nearly four years, seven years of in-data-center network incidents and 18 months of wide-area network repair records. Although the measurements reflect one organization and its systems, the operational lessons are relevant to anyone planning for reliability across complex infrastructure: analyze each device type on its own terms, account for interactions with software and system design, and use observed data to guide mitigation.

Why data center hardware reliability is a system-level concern

Modern services distribute workloads across many servers, and the programs within those workloads may share data or communicate with one another. Distribution supports scale and availability, but it also means that a fault isolated to one component can affect more than the device itself. Commodity hardware adds another consideration: systems are not designed around the assumption that every component will remain fault-free.

Reliability analysis must also distinguish between a fault and an error. A fault is the underlying cause of a malfunction; an error is how that malfunction appears to software using the device. Error correction can sometimes prevent a fault from producing a visible error. As a result, logs and incident records offer important operational evidence, but they do not necessarily capture every underlying hardware fault.

DRAM failure analysis: look beyond memory chips

The memory study found that errors were unevenly distributed: a relatively small share of servers accounted for a large share of reported errors. It also found that failures involving memory controllers and memory channels—not just DRAM chips—contributed substantially to the observed errors. This matters operationally because replacing a DIMM may not address a problem rooted elsewhere in the memory subsystem.

Several system characteristics were associated with different failure rates. More recent DRAM chip fabrication technologies showed higher failure rates in the systems examined, while DIMM capacity alone did not produce a consistent reliability trend. DIMM architecture also mattered: configurations with fewer chips and lower transfer widths had lower observed error rates. Workload type was associated with differences of up to 6.5 times in server failure rate, while CPU and memory utilization alone did not show clear trends.

The study also evaluated mitigations. Page offlining—removing pages of memory associated with errors from use—reduced the observed memory error rate by around 67% in the deployment examined. The work identified practical implementation challenges, underscoring that a technique’s potential effectiveness and its operational deployment are separate considerations. Modeling also indicated that design choices, including lower-density DIMMs and fewer processors, could reduce baseline server failure rates by up to 57.7% in the configurations analyzed.

SSD reliability is not a simple wear-out curve

SSD failure rates did not increase steadily with the amount of data written to flash. Instead, the observed behavior included distinct periods: early detection of less reliable cells, early failures followed by removal of affected cells, a useful-life period, and eventual wearout. This pattern means that a rising write count should not be treated as a direct, universal indicator of rising failure risk.

How data is laid out and handled inside the drive also matters. Sparse logical data layouts were associated with higher failure rates, while dense layouts could also be detrimental under some conditions. The study examined factors including reads, writes, block erases, page copies, discarded blocks, temperature and bus power. It found that higher temperatures were associated with increased failure rates most clearly in SSDs that did not use throttling; temperature and PCIe bus power were also correlated in the systems studied.

Software-level write totals require careful interpretation. Operating-system write counts did not always correspond to the amount of data physically written to flash because system software and SSD controllers buffer data. For reliability planning, this means that write metrics should be considered alongside buffering, controller behavior and access patterns rather than used as a standalone measure of flash wear.

Network incidents involve hardware, software and operations

Network reliability extends beyond the failure rate of individual switches or links. The study connected network events with software-level incidents and compared older cluster network designs with newer fabric designs. Incidents associated with the older cluster design increased over time until adoption of the newer design; the cluster networks had 2.8 times the software incidents of fabric networks in the comparison reported.

Incident causes included maintenance, faulty hardware and misconfiguration. The study found that human causes produced twice as many errors as hardware causes among the failures that software could not repair. This points to the importance of considering operational processes and configuration practices alongside component selection and automated recovery.

The WAN analysis also showed considerable variation in failure and repair behavior across edge nodes and link vendors. Path diversity helped the network tolerate failures even when repairs took a long time. In other words, resilience depends not only on preventing faults but also on network architecture and the ability to continue operating when a component is unavailable.

Turn operational evidence into reliability decisions

The findings support a practical approach to infrastructure reliability:

  • Measure by device and subsystem. Separate memory-controller and channel errors from DRAM-related errors, and interpret SSD and network metrics in their own contexts.
  • Use models to compare designs. Field-derived models can help evaluate relative reliability across configurations, but their conclusions should be applied to the systems and conditions they represent.
  • Plan for mitigation as well as prevention. Techniques such as page offlining, SSD throttling and network path diversity address different failure mechanisms and operate at different layers.
  • Include software and operating practices. Workload type, data layout, buffering, maintenance and configuration can influence observed reliability.

For industrial automation leaders and system integrators, the broader lesson is to treat reliability as a property of the complete system rather than a specification attached to an individual device. The evidence shows that failure patterns can be uneven, non-monotonic and shaped by design and operation. Measuring those patterns, modeling them carefully and matching mitigations to the underlying failure mechanism gives teams a stronger basis for reliability decisions.

Access Official Technical PDF

About The Author