In this section, we will cover the following topics:
- Changing the component to work with Spark pre-2.x
- Mock testing pre-2.x
- RDD mock testing
Let's start with the mocking data sources from the third section of this chapter—Mocking data sources using partial functions.
Since we were testing UserDataLogic.loadAndGetAmount, notice that everything operates on the DataFrame and thus we had a SparkSession and DataFrame.
Now, let's compare it to the Spark pre-2.x. We can see that this time, we are unable to use DataFrames. Let's assume that the following example shows our logic from the previous Sparks:
test("mock loading data from hive"){
//given
import spark.sqlContext.implicits._
val df = spark.sparkContext
.makeRDD(List(UserTransaction("a", 100), UserTransaction("b", 200)))
.toDF()
.rdd...