In this section, we will try to implement our own RDD that inherits the parent properties of RDD.
We will go through the following topics:
- Extending an RDD
- Chaining a new RDD with the parent
- Testing our custom RDD
In this section, we will try to implement our own RDD that inherits the parent properties of RDD.
We will go through the following topics:
This is a simple test that has a lot of hidden complexity. Let's start by creating a list of the record, as shown in the following code block:
class InheritanceRdd extends FunSuite {
val spark: SparkContext = SparkSession
.builder().master("local[2]").getOrCreate().sparkContext
test("use extended RDD") {
//given
val rdd = spark.makeRDD(List(Record(1, "d1")))
The Record is just a case class that...