Tutorial on Hardware Accelerators for Deep Neural Networks

Welcome to the DNN tutorial website!

• A summary of all DNN related papers from our group can be found here.
• DNN related websites and resources can be found here.
• To find out more about the Eyeriss project, please go here.
• To find out more about other on-going research in the Energy-Efficient Multimedia Systems (EEMS) group at MIT, please go here.
• Follow @eems_mit or subscribe to our mailing list for updates on the Tutorial (e.g., notification of when slides will be posted or updated)

We will be giving a two day short course on Designing Efficient Deep Learning Systems on July 17-18, 2023 on MIT Campus (with a virtual option). To find out more, please visit MIT Professional Education.

Recent News

• 11/17/2022
  

  Updated link to our book on Efficient Processing of Deep Neural Networks at here.

• 9/1/2020
  

  New article on "How to Evaluate Deep Neural Network Processors: TOPS/W (Alone) Considered Harmful" in SSCS Magazine is now available here.

• 6/25/2020
  

  Our book on Efficient Processing of Deep Neural Networks is now available here.

• 6/15/2020
  

  Excerpt of forthcoming book on Efficient Processing of Deep Neural Networks, Chapter on "Key Metrics and Design Objectives" available here.

• 5/29/2020
  

  Videos of ISCA tutorial on Timeloop/Accelergy Tutorial: Tools for Evaluating Deep Neural Network Accelerator Designs available here.

• 4/17/2020
  

  Our book on Efficient Processing of Deep Neural Networks now available for pre-order here.

• 2/16/2020
  

  Excerpt of forthcoming book on Efficient Processing of Deep Neural Networks, Chapter on "Advanced Technologies" available here.

• 12/09/2019
  

  Video and slides of NeurIPS tutorial on Efficient Processing of Deep Neural Networks: from Algorithms to Hardware Architectures available here.

• 11/11/2019
  

  We will be giving a two day short course on Designing Efficient Deep Learning Systems at MIT in Cambridge, MA on July 20-21, 2020. To find out more, please visit MIT Professional Education.

• 9/22/2019
  

  Slides for ICIP tutorial on Efficient Image Processing with Deep Neural Networks available here.

• 9/20/2019
  

  Code released for NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications here.

• All News

Overview

Deep neural networks (DNNs) are currently widely used for many AI applications including computer vision, speech recognition, robotics, etc. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, designing efficient hardware architectures for deep neural networks is an important step towards enabling the wide deployment of DNNs in AI systems.

This tutorial provides a brief recap on the basics of deep neural networks and is for those who are interested in understanding how those models are mapping to hardware architectures. We will provide frameworks for understanding the design space for deep neural network accelerators including managing data movement, handling sparsity, and importance of flexibility. This is an intermediate-level tutorial that will go beyond the material in the previous incarnations of this tutorial.

An overview paper based on the tutorial "Efficient Processing of Deep Neural Networks: A Tutorial and Survey" is available here.

Our book based on the tutorial "Efficient Processing of Deep Neural Networks" is available here.

This book provides a structured treatment of the key principles and techniques for enabling efficient processing of deep neural networks (DNNs). The book includes background on DNN processing; a description and taxonomy of hardware architectural approaches for designing DNN accelerators; key metrics for evaluating and comparing different designs; features of the DNN processing that are amenable to hardware/algorithm co-design to improve energy efficiency and throughput; and opportunities for applying new technologies. Readers will find a structured introduction to the field as well as a formalization and organization of key concepts from contemporary works that provides insights that may spark new ideas.

An excerpt of the book on "Key Metrics and Design Objectives" and "Advanced Technologies" available at here.

Participant Takeaways

• Understand the key design considerations for DNN
• Be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics
• Understand the tradeoffs between various architectures and platforms
• Assess the utility of various optimization approaches
• Understand recent implementation trends and opportunities

Slides from ISCA Tutorial (June 22, 2019)

• Overview of Deep Neural Networks [ slides ]
• Popular DNNs and Datasets [ slides ]
• Benchmarking Metrics [ slides ]
• DNN Kernel Computation [ slides ]
• DNN Accelerators (part 1) [ slides ]
• DNN Accelerators (part 2) [ slides ]
• DNN Model and Hardware Co-Design (precision) [ slides ]
• DNN Processing Near/In Memory [ slides ]
• DNN Model and Hardware Co-Design (sparsity) [ slides ]
• Sparse DNN Accelerators [ slides ]
• Tutorial Summary [ slides ]

Slides from Previous Versions of Tutorial

ISCA 2017, CICS/MTL 2017, MICRO 2016

@article{2017_dnn_piee,
  title={Efficient processing of deep neural networks: A tutorial and survey},
  author={Sze, Vivienne and Chen, Yu-Hsin and Yang, Tien-Ju and Emer, Joel},
  journal={Proceedings of the IEEE},
  year={2017}
}