Dense Associative Memories (also known as Modern Hopfield Networks) are recurrent neural networks with fixed point attractor states that are described by an energy function. In contrast to conventional Hopfield Networks, which were popular in the 1980s, their modern versions have a very large memory storage capacity, which makes them appealing tools for many problems in AI and neuroscience. In this talk, I will provide an intuitive understanding and a mathematical framework for this class of models, and will give examples of problems in AI that can be tackled using these new ideas. Specifically, I will explore the relationship between Dense Associative Memories and two prominent generative AI models: transformers and diffusion models. I will present a neural network, called the Energy Transformer, which unifies energy-based modeling, associative memories, and transformers in a single architecture. Furthermore, I will discuss an emerging perspective that views diffusion models as Dense Associative Memories operating above the critical memory storage capacity. This insight opens up interesting avenues for leveraging associative memory theory to analyze the memorization-generalization transition in diffusion models, revealing intriguing possibilities for future research.
Back to Workshop II: Theory and Practice of Deep Learning
Copyright. All Rights Reserved.