DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers. As convolutions during forward/backward passes can now operate on low bitwidth weights and activations/gradients respectively, DoReFa-Net can use bit convolution kernels to accelerate both training and inference. Moreover, as bit convolutions can be efficiently implemented on CPU, FPGA, ASIC and GPU, DoReFa-Net opens the way to accelerate training of low bitwidth neural network on these hardware. Our experiments on SVHN and ImageNet datasets prove that DoReFa-Net can achieve comparable prediction accuracy as 32-bit counterparts. For example, a DoReFa-Net derived from AlexNet that has 1-bit weights, 2-bit activations, can be trained from scratch using 6-bit gradients to get 46.1% top-1 accuracy on ImageNet validation set. The DoReFa-Net AlexNet model is released publicly.
Keywords for this software
References in zbMATH (referenced in 11 articles )
Showing results 1 to 11 of 11.
- Ashbrock, Jonathan; Powell, Alexander M.: Stochastic Markov gradient descent and training low-bit neural networks (2021)
- Huang, Di; Zhang, Rui; Zhang, Xishan; Wu, Fan; Wang, Xianzhuo; Jin, Pengwei; Liu, Shaoli; Li, Ling; Chen, Yunji: A decomposable Winograd method for N-D convolution acceleration in video analysis (2021)
- Liu, Chunlei; Ding, Wenrui; Hu, Yuan; Zhang, Baochang; Liu, Jianzhuang; Guo, Guodong; Doermann, David: Rectified binary convolutional networks with generative adversarial learning (2021)
- Long, Ziang; Yin, Penghang; Xin, Jack: Learning quantized neural nets by coarse gradient method for nonlinear classification (2021)
- Ramezani-Kebrya, Ali; Faghri, Fartash; Markov, Ilya; Aksenov, Vitalii; Alistarh, Dan; Roy, Daniel M.: NUQSGD: provably communication-efficient data-parallel SGD via nonuniform quantization (2021)
- Dong, Yinpeng; Ni, Renkun; Li, Jianguo; Chen, Yurong; Su, Hang; Zhu, Jun: Stochastic quantization for learning accurate low-bit deep neural networks (2019)
- Neta Zmora, Guy Jacob, Lev Zlotnik, Bar Elharar, Gal Novik: Neural Network Distiller: A Python Package For DNN Compression Research (2019) arXiv
- Yin, Penghang; Zhang, Shuai; Lyu, Jiancheng; Osher, Stanley; Qi, Yingyong; Xin, Jack: Blended coarse gradient descent for full quantization of deep neural networks (2019)
- Hubara, Itay; Courbariaux, Matthieu; Soudry, Daniel; El-Yaniv, Ran; Bengio, Yoshua: Quantized neural networks: training neural networks with low precision weights and activations (2018)
- Yin, Penghang; Zhang, Shuai; Lyu, Jiancheng; Osher, Stanley; Qi, Yingyong; Xin, Jack: BinaryRelax: a relaxation approach for training deep neural networks with quantized weights (2018)
- Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun: ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017) arXiv