Deep rewiring: training very sparse deep networks
Neuromorphic hardware tends to pose limits on the connectivity of deep networks
that one can run on them. But also generic hardware and software
implementations of deep learning run more efficiently on sparse networks.
Several methods exist for pruning connections of a neural network after it
was trained without connectivity constraints. We present an algorithm, DEEP
R, that enables us to train directly a sparsely connected neural network.
DEEP R automatically rewires the network during supervised training so that
connections are there where they are most needed for the task, while its
total number is all the time strictly bounded. We demonstrate that DEEP R
can be used to train very sparse feedforward and recurrent neural networks on
standard benchmark tasks with just a minor loss in performance. DEEP R is
based on a rigorous theoretical foundation that views rewiring as stochastic
sampling of network configurations from a posterior.
Reference: G. Bellec, D. Kappel, W. Maass, and R. Legenstein.
Deep rewiring: training very sparse deep networks.
International Conference on Learning Representations (ICLR), 2018.