Can machine help reduce police violence and misconduct? Can it help prevent children from getting lead poisoning? Can it help cities better target limited resources to improve lives of citizens? We're all aware of the machine learning hype right now but turning this hype into any social impact takes effort. In this talk, I'll discuss lessons learned from our work at University of Chicago while working on dozens of projects over the past few years with non-profits and governments on high-impact social challenges. These lessons span from challenges these organizations face when trying to use machine learning, to understanding how to effectively train and build cross-disciplinary teams to do practical machine learning, as well as what machine learning and social science research challenges need to be tackled, and what tools and techniques need to be developed in order to have a social and policy impact with machine learning.