The success of deep neural networks still defies detailed theoretical understanding. In particular, a lack of crisp mathematical models to describe real-world data sets currently prevents an understanding of how data sets impact training and generalization of neural networks. This talk will present two contributions towards this goal: 1) a comparison of the dynamics and the performance of two-layer neural networks trained on data sets containing random i.i.d. inputs with networks trained on more realistic inputs, e.g. MNIST images, 2) the introduction of a generative model for data sets, where high-dimensional inputs lie on a lower-dimensional manifold and have labels that depend only on their position within this manifold. Training networks on data sets drawn from this “hidden manifold model” reproduces the main phenomena seen during training on MNIST, and differs strongly form the training on i.i.d. inputs.
Copyright. All Rights Reserved.