In this section, we will learn how to save data in plain text format. The following topics will be covered:
- Saving data in plain text format
- Loading plain text data
- Testing
We will save our data in plain text format and investigate how to save it into the Spark directory. We will then load the plain text data, and then test and save it to check whether we can yield the same results code. This is our SavePlainText.scala file:
package com.tomekl007.chapter_4
import java.io.File
import com.tomekl007.UserTransaction
import org.apache.spark.sql.SparkSession
import org.apache.spark.{Partitioner, SparkContext}
import org.scalatest.{BeforeAndAfterEach, FunSuite}
import org.scalatest.Matchers._
import scala.reflect.io.Path
class SavePlainText extends FunSuite with BeforeAndAfterEach{
val spark: SparkContext = SparkSession.builder().master("local[2]"...