Deep learning benchmarking is most useful when it explains not just which platform is faster, but why performance changes across workloads. A comparison of Google Cloud TPU v2 and v3, an NVIDIA V100 GPU and an Intel Skylake CPU found no universal winner. Results depended on factors such as model architecture, batch size, memory capacity, bandwidth and software support. For automation teams evaluating compute platforms for machine learning, that is a practical reminder: benchmarks need to reflect the models and operating conditions that matter to the deployment.
Why deep learning benchmarking needs more than a few models
A small collection of popular neural networks can miss important behavior. A model may run efficiently on an accelerator while another model of the same broad category encounters memory or communication limits. Microbenchmarks have a different limitation: they measure individual operations, but do not capture dependencies and overheads across an end-to-end training workload.
The ParaDnn benchmark suite addresses these gaps by generating parameterized fully connected (FC), convolutional neural network (CNN) and recurrent neural network (RNN) models. Varying model attributes—including layer count, width, input size and batch size—allows performance to be examined across a broad design space. The study also includes six real-world models, providing reference points alongside generated workloads.
This approach makes it possible to distinguish a result tied to one particular model from a pattern that holds across a wider range. For system designers, that distinction matters: optimizing for a single workload can obscure bottlenecks that appear when model structure or training parameters change.
TPU, GPU and CPU strengths depend on workload
In the evaluated configurations, TPU performance benefited from large batches and workloads that made good use of matrix operations. The TPU’s systolic arrays were well suited to high-throughput computation, and TPU performance relative to the GPU generally improved for larger CNNs. The benchmark also found higher floating-point utilization on TPU than GPU for the CNN and RNN workloads examined.
GPU performance showed advantages in other conditions. Small batches were a better fit for the GPU’s flexible execution, while its memory system benefited larger FC models. Across the FC models, 62% ran faster on TPU, but that overall share does not mean TPU was best for every FC workload: larger models and smaller batches could favor GPU.
The CPU remained relevant, particularly where memory capacity was decisive. In the tested setup, it supported the largest FC models because its available memory was greater than that of the individual GPU or TPU cores. Its peak compute performance was lower, but a platform’s ability to hold a model can be more important than peak FLOPS when the alternative is an out-of-memory failure.
These comparisons were made using a single V100 package, one TPU board and a specific CPU instance. They do not establish how multi-GPU systems or larger TPU configurations will perform. Platform comparisons should therefore be read in the context of their tested configurations, not treated as universal rankings.
Memory bandwidth and communication can limit throughput
Peak compute is only one part of accelerator performance. The study’s roofline analysis showed that many FC and CNN operations on TPU were constrained by memory bandwidth. Increasing model width can raise the amount of data that must move through memory, while increasing batch size can increase computation performed per weight or activation transfer. As a result, a workload’s balance between computation and data movement changes with its parameters.
Even models classified as compute-bound contained meaningful memory-bound work. For example, the analysis identified memory-bound operations accounting for 19.9% of Transformer execution time and 13% of ResNet-50 execution time. This illustrates why optimizing only the largest matrix multiplications may leave other parts of a training pipeline limiting end-to-end performance.
Communication between accelerator cores also matters. On the evaluated multi-core TPU, synchronization and data exchange reduced utilization, with the effect varying by model type and batch size. Larger batches helped amortize some communication overhead by increasing computation without proportionally increasing weight transfers. The broader lesson is to account for interconnects and synchronization when moving from single-chip measurements to multi-chip training.
Include the host and software stack in the benchmark
Accelerator throughput can be limited by the system that prepares and supplies data. In the study, data infeed—including host-side preparation and transfer to the TPU—constrained some real-world CNN workloads. Removing data preparation improved measured performance for RetinaNet and SqueezeNet using bfloat16. ResNet-50 also became infeed-limited with bfloat16, after device computation accelerated. These results show how improving one stage can expose a bottleneck elsewhere.
Software maturity is another variable. Within seven months, updates to specialized TPU software improved performance by as much as 2.5 times for CNNs, seven times for FC models and 9.7 times for RNNs in the evaluated cases. The study also reported substantial performance improvements from TensorFlow and CUDA updates. Hardware comparisons should therefore record framework, compiler and software versions, rather than attributing every difference to the processor.
Build benchmarks around operational questions
For a useful evaluation, industrial teams can:
- Test representative model families. Include the relevant FC, CNN or RNN structures rather than relying on one familiar model.
- Sweep meaningful parameters. Vary batch size and model dimensions to expose changes in utilization, memory pressure and throughput.
- Measure the complete training path. Track data preparation, host-to-device transfer and multi-device communication alongside accelerator execution.
- Record platform and software configuration. Memory capacity, device count, precision mode and software version affect what a result means.
- Separate throughput from feasibility. Report both training speed and whether the model fits in available memory.
Deep learning benchmarking is not a search for one processor that wins every test. The practical goal is to match workload characteristics to platform strengths, while identifying limits in memory, data movement, communication and software. For technical decision-makers, broad, repeatable testing offers a more dependable basis for selecting and configuring training infrastructure than a single headline performance number.