Throughout this book we have demonstrated many data science techniques that, by using the power of Spark, will scale across petabytes of data. Hopefully, you have found these techniques sufficiently useful that you want to start using them in your own analytics and, indeed, have been inspired to create data science pipelines of your own.
Writing your own analytics is definitely a challenge! It can be huge fun at times and it's great when they work well. But there are times when getting them to run at scale and efficiently (or even at all) can seem like a daunting task.
Sometimes, with scarce feedback, you can get stuck in a seemingly endless loop waiting for task after task to complete not even knowing whether your job will fail at the very last hurdle. And let's face it, seeing a dreaded OutOfMemoryError at the end of a 20-hour job is no fun for anyone! Surely there must be a better way to develop analytics that run well on Spark and don't lead to wasted time and poorly...