Seminar Computational Intelligence B (708.112)

Content of the seminar: Neural Architecture Search

How to choose the architecture and hyperparameters of deep neural networks? There is no clear answer to this question and often many architectures have to be cross-validated to find a good solution to the problem at hand. Due to the recent availability of large-scale computing power, researchers have proposed algorithms that search for good architectures. We will discuss in this seminar recent work in this direction.

Topics:

Talks should be no longer than 35 minutes, and they should be clear, interesting and informative, rather than a reprint of the material. Select what parts of the material you want to present, and what not, and then present the selected material well (including definitions not given in the material: look them up on the web or if that is not successful, ask the seminar organizers). Often diagrams or figures are useful for a talk. On the other hand, giving in the talk numbers of references that are listed at the end is a no-no (a talk is an online process, not meant to be read). For the same reasons you can also quickly repeat earlier definitions or so if you suspect that the audience may not remember it.

Papers/Literature:

Motivation: why does NAS matter?

• Saxe et al., 2010, On random weights and unsupervised feature learning

Random search

• Bergstra et al., 2012, Random search for hyper-parameter optimization

• Negrinho et al., 2017, DeepArchitect: automatically designing and training deep architectures

Reinforcement learning

• Zoph et al., 2017, Neural architecture search with reinforcement learning

• Zhong et al., 2018, Practical block-wise neural network architecture generation

• Cai et al., 2018, Path-level network transformation for efficient architecture search (see also this paper)

• Pham et al., 2018, Efficient neural architecture search via parameter sharing

Evolutionary methods

• Real et al., 2017, Large-scale evolution of image classifiers

• Chen et al., 2018, Reinforced evolutionary neural architecture search

• Liu et al., 2018, Hierarchical representations for efficient architecture search

Gradient-based methods

• Liu et al., 2018, DARTS: differentiable architecture search

• Luo et al., 2018, Neural architecture optimization

Surrogate model methods

• Liu et al., 2018, Progressive neural architecture search

Hypernetworks

• Brock et al., 2017, SMASH: one-shot model architecture search through hypernetworks

• Zhang et al., 2019, Graph hypernetworks for neural architecture search

General rules:

TALKS: