Access Official Technical PDF

Share Summary:LinkedInEmail

DNN hardware co-design can help embedded AI teams address a persistent challenge: a neural network that performs well in isolation may not be the best fit for its target hardware. Architecture choices affect accuracy and computational demand, while implementation choices such as data precision and parallelism affect latency, throughput and resource use. Efficient Differentiable DNN (EDD) co-search brings these decisions into a shared optimization process rather than treating hardware implementation as a fixed step after model design.

Why search the model and implementation together?

Neural architecture search (NAS) automates the selection of a DNN’s structure. Hardware-aware NAS improves on architecture-only search by accounting for target-device characteristics, often using estimated performance from a predetermined implementation. But if the implementation itself is fixed, the search cannot fully explore how implementation decisions might influence the most suitable model.

EDD addresses this gap by searching over both DNN architecture variables and hardware implementation variables. Depending on the target, implementation variables can include quantization precision, FPGA parallelism and loop-tiling factors. GPU-related choices may include batch size, although the reported GPU experiments use batch size one for latency evaluation.

The aim is to balance model accuracy against a selected implementation objective. That objective can represent latency, throughput, energy use or model complexity, subject to hardware resource limits. The method’s formulation is designed to accommodate different devices and objectives; the specific performance and resource models still need to reflect the hardware being targeted.

How differentiable co-search works

EDD represents candidate DNN operations and implementation options using trainable search variables. In the described approach, each DNN block contains candidate operations based on MBConv building blocks. Gumbel-Softmax sampling lets the search select among those candidates while maintaining a differentiable formulation.

Quantization choices are included in the same search space. Rather than choosing a network first and applying quantization afterward, the method can evaluate how precision affects both accuracy and implementation performance. This supports mixed precision in the formulation, though platform and framework constraints may limit which precision combinations are practical.

The search estimates performance and resource use at several levels: candidate operations, DNN blocks and the full network. These estimates feed into a combined objective that considers accuracy loss, implementation performance loss and resource constraints. Because the objective is differentiable with respect to the search variables, gradient-based optimization can update architecture and implementation decisions together.

Accounting for resource sharing

Resource modeling depends on accelerator architecture. In a pipelined FPGA design, different layers may have their own accelerator resources, so resource estimates can be summed across blocks. In a recursive design, operations may reuse the same hardware. Counting every layer’s resources independently would overstate the total, so EDD includes a resource-sharing formulation that counts a shared operation resource only when the search selects it.

This distinction matters when comparing designs under a resource ceiling. A search that ignores reuse could reject a feasible recursive implementation, while a pipelined design must account for the resources needed to support its separate stages.

Different targets call for different objectives

For a GPU target, the reported EDD formulation focuses on inference latency. The GPU experiments use a uniform precision across the network because the framework support described in the study did not support mixed-precision inference in the same way as the search formulation. The reported tests also distinguish between devices: latency comparisons in the main model table use a Titan RTX, while a separate precision comparison uses a 1080 Ti with TensorRT.

For FPGA targets, the implementation choices include parallel factors that determine how many multiplications can be performed concurrently. The study considers two architectures:

  • Recursive FPGA accelerator: layers of the same type share hardware, and the optimization focuses on end-to-end latency.
  • Pipelined FPGA accelerator: layers use separate accelerator resources, and the optimization focuses on throughput while respecting the overall resource budget.

These examples show why hardware-aware model design is not simply a matter of minimizing network size. The preferred architecture can depend on whether the system values low latency, sustained throughput, resource sharing or another defined objective.

What the reported results show

The researchers evaluated EDD with three targets: a low-latency GPU, a recursive FPGA accelerator and a pipelined FPGA accelerator. The search used a subset of ImageNet sampled from 100 classes; the resulting models were then trained from scratch on the full 1,000-class ImageNet. Each model was produced within a reported 12 GPU-hour search.

For the GPU, EDD-Net-1 achieved 11.17 ms inference latency on a Titan RTX, compared with 15.72 ms for Proxyless-GPU in the reported comparison. The paper describes this as 1.40 times faster, with similar accuracy to the compared NAS models. For a recursive FPGA implementation on a ZCU102, EDD-Net-2 recorded 7.96 ms latency in the comparison table.

For the pipelined FPGA case, EDD-Net-3 achieved 40.2 frames per second on a ZC706, compared with 27.7 frames per second for the DNNBuilder VGG16 comparison. The reported throughput improvement was 1.45 times, and EDD-Net-3 also had lower top-1 and top-5 error in that comparison. These results apply to the configurations and evaluation conditions described in the study; they are not a guarantee of equivalent gains on other devices or workloads.

Practical implications for embedded AI teams

For engineers and system integrators, the central lesson is to make hardware constraints part of model selection rather than relying only on post-search implementation tuning. A useful co-search process needs:

  • A clearly defined target metric, such as end-to-end latency or throughput.
  • Implementation and resource models suited to the selected GPU, FPGA or accelerator architecture.
  • Explicit treatment of precision choices and any platform restrictions.
  • Resource accounting that reflects whether hardware is shared or replicated.

EDD’s formulation provides a way to connect these variables through gradient-based search. The final design still requires training and implementation refinement for the target platform, as the study notes. For industrial automation teams evaluating embedded vision or other edge inference workloads, the practical value is a more integrated design process: model structure, precision and accelerator configuration can be considered together against the system’s actual performance priorities.

Conclusion

DNN hardware co-design offers a structured alternative to designing a network first and adapting it to hardware later. By combining architecture, quantization and implementation variables in a differentiable search, EDD targets accuracy and hardware performance within resource constraints. Its GPU and FPGA results illustrate the potential of that approach, while also underscoring the need to tailor objectives and implementation models to each deployment platform.

Access Official Technical PDF

About The Author