Tutorial on Hardware Architectures for Deep Neural Networks
- Joel Emer MIT, NVIDIA
- Vivienne Sze MIT
- Yu-Hsin Chen MIT
- Tien-Ju Yang MIT
Email: eyeriss at mit dot edu
Overview
Deep neural networks (DNNs) are currently widely used for many AI applications including computer vision, speech recognition, robotics, etc. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, designing efficient hardware architectures for deep neural networks is an important step towards enabling the wide deployment of DNNs in AI systems.
In this tutorial, we will provide an overview of DNNs, discuss the tradeoffs of the various architectures that support DNNs including CPU, GPU, FPGA and ASIC, and highlight important benchmarking/comparison metrics and design considerations. We will then describe recent techniques that reduce the computation cost of DNNs from both the hardware architecture and network algorithm perspective. Finally, we will discuss the different hardware requirements for inference and training.
An overview paper based on the tutorial "Efficient Processing of Deep Neural Networks: A Tutorial and Survey" is available here.
Slides from ISCA Tutorial (June 24, 2017)
- Background of Deep Neural Networks [ slides ]
- Survey of DNN Development Resources [ slides ]
- Survey of DNN Hardware [ slides ]
- DNN Accelerator Architectures [ slides ]
- Advanced Technology Opportunities [ slides ]
- Network and Hardware Co-Design [ slides ]
- Benchmarking Metrics [ slides ]
- Tutorial Summary [ slides ]
- References [ slides ]
Entire Tutorial [ slides ]
Slides from MICRO-49 Tutorial (Oct 16, 2016)
- Background of Deep Neural Networks [ slides ]
- Survey of DNN Development Resources [ slides ]
- Survey of DNN Hardware [ slides ]
- DNN Accelerator Architectures [ slides ]
- Advanced Technology Opportunities [ slides ]
- Network and Hardware Co-Design [ slides ]
- Benchmarking Metrics [ slides ]
- Hardware Requirements for Training [ slides ]
- References [ slides ]
Entire Tutorial [ slides ]
Video
BibTeX
@article{2017_dnn_piee,
title={Efficient processing of deep neural networks: A tutorial and survey},
author={Sze, Vivienne and Chen, Yu-Hsin and Yang, Tien-Ju and Emer, Joel},
journal={Proceedings of the IEEE},
year={2017}
} Participant Takeaways
- Understand the key design considerations for DNN
- Be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics
- Understand the tradeoffs between various architectures and platforms
- Assess the utility of various optimization approaches
- Understand recent implementation trends and opportunities