Data center hardware reliability is not simply a matter of selecting components with low expected failure rates. In large, distributed systems, memory errors, storage faults or network incidents can affect workloads that span many interconnected servers. A large-scale field study of Facebook’s infrastructure shows why operators need to understand how failures appear in production, which conditions are associated with them, and how software and system design can limit their effects.
The study examined DRAM, flash-based solid-state drives (SSDs), and network devices using operational data collected over different periods. Its findings are specific to the systems and fleet studied, rather than universal guarantees for other environments. Even so, its central lesson is broadly useful to technical decision-makers: measure failures in context, then use those measurements to guide design and response.
Why component failures can become system incidents
Modern services combine programs that share data and communicate across servers. A fault in one device may therefore affect more than the machine that contains it. At the same time, large fleets use many devices, increasing the opportunities for faults to occur. Commodity hardware can make large-scale infrastructure practical, but reliability also depends on how the surrounding software detects, contains and recovers from problems.
It is useful to distinguish a fault, the underlying cause of a malfunction, from an error, the observable effect when software uses a malfunctioning device. Error correction and other recovery mechanisms may prevent a fault from becoming a visible software error. Consequently, a useful reliability program should track both device-level signals and the operational incidents they lead to.
DRAM failure analysis: look beyond memory capacity
The DRAM study analyzed memory errors across a server fleet over fourteen months. One important finding was that errors were highly concentrated: a relatively small group of servers accounted for a large share of observed errors. Average error rates could therefore conceal a substantially different experience among the systems with the most problems.
The analysis also found that memory-related problems were not limited to DRAM chips. The memory controller and memory channel contributed a majority of errors in the systems examined. This matters when diagnosing recurring incidents: replacing a DIMM may not address a fault originating elsewhere in the memory subsystem.
Several design and operating characteristics were associated with different failure rates:
- More recent DRAM chip fabrication technologies, represented by higher chip density, showed higher failure rates in the observed fleet.
- DIMM architecture mattered; configurations with fewer chips and lower transfer widths had the lowest error rates in the study.
- Workload type was associated with failure-rate differences, while CPU and memory utilization alone showed no clear trend.
A statistical model based on the field data indicated that design choices such as using lower-density DIMMs and fewer processors could reduce the baseline server failure rate by up to 57.7% in the modeled comparisons. The study also evaluated page offlining—the removal of affected memory pages from use—and reported a 67% reduction in memory error rate. These results describe the studied systems and should not be treated as a prediction for every server configuration.
SSD reliability follows a lifecycle, not a simple wear curve
SSD failure rates did not increase steadily with the amount of data written to flash. Instead, the observed drives passed through distinct periods associated with early detection, early failure, useful life and wearout. Differences among flash blocks, along with how faults are detected and affected blocks are managed, help explain why a single wear measure may not describe the failure behavior of an entire drive population.
The findings also point to operational factors beyond write volume. Sparse logical data layouts—where data is spread non-contiguously across the address space—were associated with higher failure rates in the examined platforms. Dense layouts could also have negative effects under some conditions. The study found that higher temperatures generally increased failure rates most noticeably in SSDs without throttling; temperature trends differed among drives at the higher temperatures examined.
Software-level write totals were not always a reliable measure of flash wear. Operating-system and controller buffering meant that reported writes did not necessarily translate directly into writes to flash cells. For reliability analysis, operators should therefore distinguish host-level workload metrics from device-level activity where those measurements are available.
Network reliability includes people, processes and recovery
The network study covered seven years of intra-data-center incidents and eighteen months of repairs involving the wide-area backbone. Many incidents that automated software could not repair involved maintenance, hardware faults or misconfiguration. The study attributed more errors to human actions than to hardware, underscoring the importance of configuration control and operational procedures alongside component quality.
Network design and remediation also influenced incident impact. Newer fabric networks had lower-severity software incidents than older cluster networks in the studied environment. The analysis further found that network devices built from commodity chips had lower software incident rates than third-party devices, a result associated with their integration with automated failover and remediation software. These observations highlight that incident rates alone are not enough: severity, repair time, redundancy and the ability to restore service also matter.
Turning field data into reliability decisions
For infrastructure and operations teams, the practical value of this work is its measurement approach. A useful reliability program can:
- Record errors and incidents alongside device attributes, workload, age and operating conditions.
- Separate component faults from symptoms observed by applications or operators.
- Track how failures are distributed, rather than relying only on fleet-wide averages.
- Evaluate mitigation techniques in production, including their limitations and implementation challenges.
- Use models to compare design choices, while checking whether the underlying data applies to the systems being evaluated.
For industrial automation teams, these are transferable principles for managing complex infrastructure, not evidence that the same failure patterns or numerical results apply to plant systems. The key is to connect equipment-level diagnostics with the operational effects experienced by dependent applications and services.
Conclusion
Large-scale hardware reliability depends on more than component specifications or age. DRAM errors can originate outside the memory chips, SSD behavior can change across distinct lifecycle periods, and network incidents can reflect configuration and maintenance as well as hardware faults. Systematic measurement and modeling help operators identify these patterns, target mitigations and prepare for failures. The most useful reliability decisions are grounded in operational data that reflects the devices, workloads and recovery processes of the environment in question.