A few years ago, I wrote and recorded the edX course Implementing Predictive Analytics with Spark in Azure HDInsight, which teaches you how to use the Spark MLLib library to build machine learning solutions in a Spark Azure HDInsight cluster.
Microsoft now also offers Spark capabilities in the Azure Databricks service. This repo contains versions of the lab files that have been modified to use Azure Databricks.
If you’re completely new to Spark and Databricks, you might want to check out the Introduction to Databricks Labs first!
The following videos show some of the key concepts and techniques used in the labs. You can watch these to get an overview of using Spark to do machine learning in Azure Databricks; or if you prefer, you can skip straight to the labs. To watch the videos, click the image (or right-click / CTRL+click to open the video in a separate browser tab).