In this recipe, we'll see how to create a new DataFrame from Scala case classes.
Note
The code for this recipe can be found at https://github.com/arunma/ScalaDataAnalysisCookbook/blob/master/chapter1-spark-csv/src/main/scala/com/packt/scaladata/spark/csv/DataFrameFromCaseClasses.scala.
We create a new entity called
Employeewith theidandnamefields, like this:case class Employee(id:Int, name:String)Similar to the previous recipe, we create
SparkContextandSQLContext.val conf = new SparkConf().setAppName("colRowDataFrame").setMaster("local[2]") //Initialize Spark context with Spark configuration. This is the core entry point to do anything with Spark val sc = new SparkContext(conf) //The easiest way to query data in Spark is to use SQL queries. val sqlContext=new SQLContext(sc)
We can source these employee objects from a variety of sources, such as an RDBMS data source, but for the sake of this example, we construct a list...