Tutorial on Hardware Architectures for Deep Neural Networks
MICRO-50 (Full Day: October 15, 2017)


Email: eyeriss at mit dot edu


Welcome to the DNN tutorial website!

  • A summary of all DNN related papers from our group can be found here. Other related websites and resources can be found here.
  • To find out more about the Eyeriss project, please go here.
  • To find out more about other on-going research in the Energy-Efficient Multimedia Systems (EEMS) group at MIT, please go here.

Updates

or subscribe to our mailing list for updates on the Tutorial (e.g., notification of when slides will be posted or updated)

  • 04/17/2018

    New paper on "NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications" [ PDF ].

  • 03/31/2018

    We will be giving a two day short course on "Designing Efficient Deep Learning Systems" on MIT campus on July 23-24, 2018. To find out more, please visit MIT Professional Education.

  • 02/12/2018

    New paper on "Understanding the Limitations of Existing Energy-Efficient Design Approaches for Deep Neural Networks" [ PDF ].

  • 11/22/2017

    Our paper on "Efficient Processing of Deep Neural Networks: A Tutorial and Survey" is the cover story for the December issue of Proceedings of the IEEE. The final version is available here.

  • 09/17/2017

    We will be giving a two day short course on “Designing Efficient Deep Learning Systems” in Mountain View, California on March 28-29, 2018. To find out more, please visit MIT Professional Education.

  • 06/25/2017

    Updated slides posted here from ISCA 2017.

  • 05/22/2017

    We will be giving an updated version of our tutorial at MICRO-50.

  • 03/27/2017

    Updated slides posted here from the CICS/MTL tutorial.

  • 03/27/2017

    New paper on "Efficient Processing of Deep Neural Networks: A Tutorial and Survey" available on arXiv. [ LINK ]

  • 03/25/2017

    DNN Energy Estimation Website available online. [ LINK ]

  • 01/21/2017

    We will be giving an updated version of our tutorial at ISCA 2017.

  • 01/17/2017

    New paper on “Hardware for Machine Learning: Challenges and Opportunities” will be presented at IEEE CICC 2017. Available on arXiv. [ LINK ]

  • 11/15/2016

    New paper on “Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning” will be presented at CVPR 2017. Available on arXiv. [ LINK ]

  • 11/12/2016

    DNN Tutorial Slides now available online.

  • 10/16/2016

    Full day tutorial held at MICRO-49


Overview


Deep neural networks (DNNs) are currently widely used for many AI applications including computer vision, speech recognition, robotics, etc. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, designing efficient hardware architectures for deep neural networks is an important step towards enabling the wide deployment of DNNs in AI systems.

In this tutorial, we will provide an overview of DNNs, discuss the tradeoffs of the various architectures that support DNNs including CPU, GPU, FPGA and ASIC, and highlight important benchmarking/comparison metrics and design considerations. We will then describe recent techniques that reduce the computation cost of DNNs from both the hardware architecture and network algorithm perspective. Finally, we will discuss the different hardware requirements for inference and training.

Register for the two day short course in Mountain View, California (March 28-29, 2018) here.

An overview paper based on the tutorial "Efficient Processing of Deep Neural Networks: A Tutorial and Survey" is available here.


Slides from ISCA Tutorial (June 24, 2017)

  • Background of Deep Neural Networks [ slides ]
  • Survey of DNN Development Resources [ slides ]
  • Survey of DNN Hardware [ slides ]
  • DNN Accelerator Architectures [ slides ]
  • Advanced Technology Opportunities [ slides ]
  • Network and Hardware Co-Design [ slides ]
  • Benchmarking Metrics [ slides ]
  • Tutorial Summary [ slides ]
  • References [ slides ]

Entire Tutorial [ slides ]


Slides from CICS/MTL Tutorial (March 27, 2017)

  • Background of Deep Neural Networks [ slides ]
  • Survey of DNN Development Resources [ slides ]
  • Survey of DNN Hardware [ slides ]
  • DNN Accelerator Architectures [ slides ]
  • Network and Hardware Co-Design [ slides ]

Entire Tutorial [ slides ]


Slides from MICRO-49 Tutorial (Oct 16, 2016)

  • Background of Deep Neural Networks [ slides ]
  • Survey of DNN Development Resources [ slides ]
  • Survey of DNN Hardware [ slides ]
  • DNN Accelerator Architectures [ slides ]
  • Advanced Technology Opportunities [ slides ]
  • Network and Hardware Co-Design [ slides ]
  • Benchmarking Metrics [ slides ]
  • Hardware Requirements for Training [ slides ]
  • References [ slides ]

Entire Tutorial [ slides ]


BibTeX


@article{2017_dnn_piee,
  title={Efficient processing of deep neural networks: A tutorial and survey},
  author={Sze, Vivienne and Chen, Yu-Hsin and Yang, Tien-Ju and Emer, Joel},
  journal={Proceedings of the IEEE},
  year={2017}
}                

Participant Takeaways

  • Understand the key design considerations for DNN
  • Be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics
  • Understand the tradeoffs between various architectures and platforms
  • Assess the utility of various optimization approaches
  • Understand recent implementation trends and opportunities


Related Papers

  • T.-J. Yang, A. Howard, B. Chen, X. Zhang, A. Go, V. Sze, H. Adam, "NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications," arXiv, April 2018. [ paper arXiv ]
  • Y.-H. Chen*, T.-J. Yang*, J. Emer, V. Sze, "Understanding the Limitations of Existing Energy-Efficient Design Approaches for Deep Neural Networks," SysML Conference, February 2018. [ paper PDF | talk video ] Selected for Oral Presentation
  • V. Sze, T.-J. Yang, Y.-H. Chen, J. Emer, "Efficient Processing of Deep Neural Networks: A Tutorial and Survey," Proceedings of the IEEE, vol. 105, no. 12, pp. 2295-2329, December 2017. [ paper PDF ]
  • T.-J. Yang, Y.-H. Chen, J. Emer, V. Sze, "A Method to Estimate the Energy Consumption of Deep Neural Networks," Asilomar Conference on Signals, Systems and Computers, Invited Paper, October 2017. [ paper PDF | slides PDF ]
  • T.-J. Yang, Y.-H. Chen, V. Sze, "Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. [ paper arXiv | poster PDF | DNN energy estimation tool LINK | DNN models LINK ] Highlighted in MIT News
  • Y.-H. Chen, J. Emer, V. Sze, "Using Dataflow to Optimize Energy Efficiency of Deep Neural Network Accelerators," IEEE Micro's Top Picks from the Computer Architecture Conferences, May/June 2017. [ PDF ]
  • A. Suleiman*, Y.-H. Chen*, J. Emer, V. Sze, "Towards Closing the Energy Gap Between HOG and CNN Features for Embedded Vision," IEEE International Symposium of Circuits and Systems (ISCAS), Invited Paper, May 2017. [ paper PDF | slides PDF | talk video ]
  • V. Sze, Y.-H. Chen, J. Emer, A. Suleiman, Z. Zhang, "Hardware for Machine Learning: Challenges and Opportunities," IEEE Custom Integrated Circuits Conference (CICC), Invited Paper, May 2017. [ paper arXiv | slides PDF ] Received Best Invited Paper Award
  • Y.-H. Chen, T. Krishna, J. Emer, V. Sze, "Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks," IEEE Journal of Solid State Circuits (JSSC), ISSCC Special Issue, Vol. 52, No. 1, pp. 127-138, January 2017. [ PDF ]
  • Y.-H. Chen, J. Emer, V. Sze, "Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks," International Symposium on Computer Architecture (ISCA), pp. 367-379, June 2016. [ paper PDF | slides PDF ] Selected for IEEE Micro’s Top Picks special issue on "most significant papers in computer architecture based on novelty and long-term impact" from 2016
  • Y.-H. Chen, T. Krishna, J. Emer, V. Sze, "Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks," IEEE International Conference on Solid-State Circuits (ISSCC), pp. 262-264, February 2016. [ paper PDF | slides PDF | poster PDF | demo video | project website ] Highlighted in EETimes and MIT News.
* Indicates authors contributed equally to the work


Related Websites and Resources

  • Eyeriss Project Website [ LINK ]
  • DNN Energy Estimation Website [ LINK ]
  • DNN Processor Benchmarking Website [ LINK ]