Despite the outstanding performance of neural networks in “standard” classification settings (Empirical Risk Minimization-ERM), these same networks famously underperform in worst-case situations. Even when we explicitly optimize them for robustness (robust ERM), they often exhibit large (robust) generalization gaps and (robust) overfitting during training. What changes from ERM to robust ERM?
In this talk, we approach this question from a learning theoretic perspective and put the implicit bias of optimization at the forefront. We show that model specification (in particular, the optimization algorithm and architecture) becomes much more important in robust settings, than in standard ones.
Based on our findings, we speculate on current and future challenges in robust machine learning.
Copyright. All Rights Reserved.