Landscape Analysis for Overcomplete Tensor and Neural Collapse

Qing Qu
University of Michigan
Center for Data Science

In this talk, we provide the first global landscape analysis for overcomplete tensor decomposition and neural collapse. For both problems, we show that landscapes have similar benign global geometric structures. First, overcomplete tensor decomposition relates to many applications in representation learning, such as overcomplete dictionary learning and convolutional dictionary learning. Under tight frame assumptions of the overcomplete components, we show that the nonconvex loss over the sphere has no spurious local minimizers. Second, recent seminal work by Donoho et al. showed a prevalence phenomenon during the terminal phase of network training - neural collapse. By studying the optimization landscape of the training loss under a unconstrained feature model, we provide the theoretical justification for this phenomenon, which could have broad implications for network training, design, and beyond.

Presentation (PDF File)

Back to Workshop IV: Efficient Tensor Representations for Learning and Computational Complexity