Self-driving cars are not expected to reach full adoption for at least another 35 years. So, in the meantime, how will self-driving cars change urban mobility? This talk describes techniques in machine learning and optimization critical for enabling mixed-autonomy mobility, the gradual and complex integration of automated vehicles into the existing transportation system. Leveraging and advancing techniques in model-free deep reinforcement learning and control theory, the talk explores and quantifies the potential impact of a small fraction of automated vehicles on low-level traffic flow dynamics such as congestion on a variety of important traffic contexts. To address computational limitations of existing methods to large-scale control systems, generic reinforcement learning techniques for improved variance reduction are developed. Together, these optimization methods and empirical findings demonstrate how small changes in vehicles, sensors, and infrastructure can be harnessed for significant impact on urban mobility, and shed light into the future study of mixed autonomy systems.