Machine learning algorithms have come to dominate several industries. After decades of resistance from examiners and auditors, machine learning is now moving from the research bureau to the application stack for credit scoring and a range of other credit risk applications. This migration is not without risks and new challenges. Much of the research is now shifting from how best to create the models to how best to use the models in a regulatory compliant business context. This article examines the impressive array of machine learning methods and areas of application for credit risk. In this investigation, we create a taxonomy to think about how different machine learning components are matched to create specific algorithms. The reasons why machine learning is successful over simple linear methods are explored through a specific loan example. Throughout, we highlight open-ended questions, ideas for improvements, and a framework for thinking about how to choose the best machine learning method for a specific problem.