Sign In Start Free Trial
Account

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Big Data on Kubernetes
  • Table Of Contents Toc
Big Data on Kubernetes

Big Data on Kubernetes

By : Neylson Crepalde
close
close
Big Data on Kubernetes

Big Data on Kubernetes

By: Neylson Crepalde

Overview of this book

In today's data-driven world, organizations across different sectors need scalable and efficient solutions for processing large volumes of data. Kubernetes offers an open-source and cost-effective platform for deploying and managing big data tools and workloads, ensuring optimal resource utilization and minimizing operational overhead. If you want to master the art of building and deploying big data solutions using Kubernetes, then this book is for you. Written by an experienced data specialist, Big Data on Kubernetes takes you through the entire process of developing scalable and resilient data pipelines, with a focus on practical implementation. Starting with the basics, you’ll progress toward learning how to install Docker and run your first containerized applications. You’ll then explore Kubernetes architecture and understand its core components. This knowledge will pave the way for exploring a variety of essential tools for big data processing such as Apache Spark and Apache Airflow. You’ll also learn how to install and configure these tools on Kubernetes clusters. Throughout the book, you’ll gain hands-on experience building a complete big data stack on Kubernetes. By the end of this Kubernetes book, you’ll be equipped with the skills and knowledge you need to tackle real-world big data challenges with confidence.
Table of Contents (18 chapters)
close
close
Lock Free Chapter
1
Part 1:Docker and Kubernetes
5
Part 2: Big Data Stack
10
Part 3: Connecting It All Together

Summary

In this chapter, you learned how to deploy and manage key big data technologies such as Apache Spark, Apache Airflow, and Apache Kafka on Kubernetes. Deploying these tools on Kubernetes provides several benefits, including simplified operations, better resource utilization, scaling, high availability, and unified cluster management.

You started by deploying the Spark operator on Kubernetes and running a Spark application to process data from Amazon S3. This allows you to leverage Kubernetes for running Spark jobs in a cloud-native way, taking advantage of dynamic resource allocation and scaling.

Next, you deployed Apache Airflow on Kubernetes using the official Helm chart. You configured Airflow to run with the Kubernetes executor, enabling it to dynamically launch tasks on Kubernetes. You also set up remote logging to Amazon S3 for easier monitoring and debugging. Running Airflow on Kubernetes improves reliability, scalability, and resource utilization for orchestrating...

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Big Data on Kubernetes
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist download Download options font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected

Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon