Understanding BatchNorm in Ternary Training

  • Eyyüb Sari
  • Vahid Partovi Nia

Abstract

Neural networks are comprised of two components, weights and
activation function. Ternary weight neural networks (TNNs) achieve
a good performance and offer up to 16x compression ratio. TNNs
are difficult to train without BatchNorm and there has been no study
to clarify the role of BatchNorm in a ternary network. Benefiting
from a study in binary networks, we show how BatchNorm helps in
resolving the exploding gradients issue.

Published
2020-01-02
How to Cite
Sari, E., & Nia, V. (2020). Understanding BatchNorm in Ternary Training. Journal of Computational Vision and Imaging Systems, 5(1), 2. Retrieved from https://openjournals.uwaterloo.ca/index.php/vsl/article/view/1646