Winning a Kaggle competition is an art by itself, but we just want to show you how the Apache SparkML tooling can be used efficiently to do so.
We'll use an archived competition for this offered by BOSCH, a German multinational engineering and electronics company, on production line performance data. Details for the competition data can be found at https://www.kaggle.com/c/bosch-production-line-performance/data.
The challenge data comes in three ZIP packages but we only use two of them. One contains categorical data, one contains continuous data, and the last one contains timestamps of measurements, which we will ignore for now.
If you extract the data, you'll get three large CSV files. So the first thing that we want to do is re-encode them into parquet in order to be more space-efficient:
def convert(filePrefix : String) = {
val basePath = "yourBasePath"
var df = spark
.read
.option("header"...