Deep learning (the training of deep neural nets) is a simple idea, which has had many extraordinary applications throughout industry and science over the last decade. In mathematics the impact has so-far been modest at best. I will discuss a few instances where it has proved useful, and led to interesting (pure) mathematics. I will also discuss what can be learned from these examples, and try to guess an answer to the question in the title. I will also reflect on my experience as a pure mathematician interacting with deep learning