In the Feature extraction section of Chapter 2, Data Preparation for Spark ML, we have reviewed a few methods for feature extraction and discussed their implementation on Apache Spark. All the techniques discussed there will be applied to our datasets here.

Besides feature development, for this project, we will also need to spend a lot of effort to merge various datasets together to obtain more features.

Therefore, for this project, we actually need to conduct feature development, then data merging and reorganizing, and then feature selection, which is to utilize all the techniques discussed in Chapter 2, Data Preparation for Spark ML, and also in Chapter 3, A Holistic View on Spark. A lot of work has been completed to produce several good datasets for this big project, with the techniques described earlier.

As an exercise, we will focus on some of the key tasks, which are to reorganize data per day, then merge datasets, and finally conduct feature selection to...