Seminar Computational Intelligence A (708.111)

SS 2012

Institut für Grundlagen der Informationsverarbeitung (708)

Assoc. Prof. Dr. Robert Legenstein

Office hours: by appointment (via e-mail)

E-mail: robert.legenstein@igi.tugraz.at
Homepage: www.igi.tugraz.at/legi/



Location: IGI-seminar room, Inffeldgasse 16b/I, 8010 Graz
Date: starting from March 14th  2012, every Wednesday, 16.15 - 18.00 p.m.

Content of the seminar:

This year, we will discuss in Seminar A scientific articles about Deep Learning. Deep learning has emerged in the last years as one very powerful method for training deep neural architectures. It can overcome many problems of traditional neural network approaches and shows excellent practical performance.
Each student of this seminar will be able to choose a paper of his or her interest and, after preparation and discussion with Robert Legenstein, present it in a seminar talk.
Alternatively, students can also choose to present one of the recently published papers from our institute.

Slides from first session:

Slides first session

Doodle Poll for Talks

Doodle Poll

Update

Ressources for deep learning at http://deeplearning.net/

A paper by Hinton that discusses some practical aspects when working with deep architectures: Practical Guide. Can also be chosen for a seminar talk.

Talks:

Date Speaker
Paper

Mar 28, 2012

Robert Legenstein

A quick introduction to Boltzmann Machines

Apr 25, 2012

Daniel Markl

Reducing the dimensionality of data with neural networks, Slides


May 23, 2012


Stefan Grabuschnig


Learning Deep Architectures for AI (1), Slides

May 23, 2012

Teresa Klatzer

Learning Deep Architectures for AI (2), Slides




Jun 6, 2012

Florian Hubner

Unsupervised learning of image transformations, Slides

Jun 13, 2012

Markus Eger

The Recurrent Temporal Restricted Boltzmann Machine, Slides

Jun 20, 2012

Gernot Griesbacher

Neural sampling: A model for stochastic computation in recurrent networks of spiking neurons, Slides

Jun 20, 2012

Michael Rath

Probabilistic inference in general graphical models through sampling in stochastic networks of spiking neurons, Slides
Jul 04, 2012
 Philipp Singer
Discovering Binary Codes for Documents by Learning Deep Generative Models, Slides

Papers on Deep Learning :

Basic papers:

Hinton, G. E. and Salakhutdinov, R. R.
Reducing the dimensionality of data with neural networks.
Science, Vol. 313. no. 5786, pp. 504 - 507, 28 July 2006.
[ full paper ] [ supporting online material (pdf) ] [ Matlab code ]
The science paper that made deep networks popular
Hinton, G. E., Osindero, S. and Teh, Y.
A fast learning algorithm for deep belief nets
Neural Computation 18, pp 1527-1554. 2006. [pdf]
The basis for deep learning: the contrastive divergence learning algorithm

Applications:

Taylor, G. W., Hinton, G. E. and Roweis, S.
Modeling human motion using binary latent variables
Advances in Neural Information Processing Systems, 19
MIT Press, Cambridge, MA, 2007 [pdf]

Memisevic, R. and Hinton, G. E..
Unsupervised learning of image transformations
Computer Vision and Pattern Recognition (CVPR-07), 2007 [pdf]
A longer version: Technical Report UTML TR 2006-005  [pdf]

Salakhutdinov R. R, Mnih, A. and Hinton, G. E.
Restricted Boltzmann Machines for Collaborative Filtering
International Conference on Machine Learning, Corvallis, Oregon, 2007 [pdf]

Extensions of deep networks:

Sutskever, I., Hinton, G. E. and Taylor, G. W.
The Recurrent Temporal Restricted Boltzmann Machine
Advances in Neural Information Processing Systems 21, MIT Press, Cambridge, MA
[pdf]

Memisevic, R. and Hinton, G. E.
Learning to represent spatial transformations with factored higher-order Boltzmann machines
 Neural Computation, Vol 22, pp 1473-1492 [pdf]

Cognitive-Science Applications:

Hinton, G. E. and Salakhutdinov, R.
Discovering Binary Codes for Fast Document Retrieval by Learning Deep Generative Models
Topics in Cognitive Science, Vol 3, pp 74-91 [pdf]


Ruslan Salakhutdinov, Josh Tenenbaum , Antonio Torralba.
Learning to Learn with Compound Hierarchical-Deep Models
Neural Information Processing Systems (NIPS 25), 2012 [ pdf]

Misc:

Ruslan Salakhutdinov and Geoffrey Hinton.
An Efficient Learning Procedure for Deep Boltzmann Machines
MIT Technical Report MIT-CSAIL-TR-2010-037, 2010 [ pdf]
Yoshua Bengio. 
Learning Deep Architectures for AI
Foundations and Trends in Machine Learning: Vol. 2: No. 1, pp 1-127, 2009
[pdf]

IGI-Papers:

L. Büsing, J. Bill, B. Nessler, and W. Maass 
Neural dynamics as sampling: A model for stochastic computation in recurrent networks of spiking neurons. PLoS Computational Biology, published 03 Nov 2011. doi:10.1371/journal.pcbi.1002211 (pdf)
This paper shows how Boltzman machines can be implemented by networks of spiking neurons.

D. Pecevski, L. Büsing, and W. Maass 
Probabilistic inference in general graphical models through sampling in stochastic networks of spiking neurons.
 PLoS Computational Biology, 7(12):e1002294, 2011 (pdf)
Shows how networks of spiking neurons can implement probabilistic inference in graphical models.

 




           02/12 rh