Book Image

Apache Kafka 1.0 Cookbook

By : Alexey Zinoviev, Raúl Estrada
Book Image

Apache Kafka 1.0 Cookbook

By: Alexey Zinoviev, Raúl Estrada

Overview of this book

Apache Kafka provides a unified, high-throughput, low-latency platform to handle real-time data feeds. This book will show you how to use Kafka efficiently, and contains practical solutions to the common problems that developers and administrators usually face while working with it. This practical guide contains easy-to-follow recipes to help you set up, configure, and use Apache Kafka in the best possible manner. You will use Apache Kafka Consumers and Producers to build effective real-time streaming applications. The book covers the recently released Kafka version 1.0, the Confluent Platform and Kafka Streams. The programming aspect covered in the book will teach you how to perform important tasks such as message validation, enrichment and composition.Recipes focusing on optimizing the performance of your Kafka cluster, and integrate Kafka with a variety of third-party tools such as Apache Hadoop, Apache Spark, and Elasticsearch will help ease your day to day collaboration with Kafka greatly. Finally, we cover tasks related to monitoring and securing your Apache Kafka cluster using tools such as Ganglia and Graphite. If you're looking to become the go-to person in your organization when it comes to working with Apache Kafka, this book is the only resource you need to have.
Table of Contents (18 chapters)
Title Page
Credits
About the Author
About the Reviewers
www.PacktPub.com
Customer Feedback
Dedication
Preface

Installing Kafka


This is the first step. This recipe shows how to install Apache Kafka.

Getting ready

Ensure that you have at least 4 GB of RAM in your machine; the installation directory will be /usr/local/kafka/ for Mac users and /opt/kafka/ for Linux users. Create these directories.

How to do it...

Go to the Apache Kafka home page at http://kafka.apache.org/downloads, as in Figure 1-1, Apache Kafka download page:

Figure 1-1. Apache Kafka download page

The current available version of Apache Kafka is 0.10.2.1, as a stable release. A major limitation with Kafka since 0.8.x is that it is not backward-compatible. So, we cannot replace this version for one prior to 0.8. Once you've downloaded the latest available release, let's proceed with the installation.

Remember, for Mac users, replace the directory /opt/ for /usr/local/ in the examples.

Installing Java in Linux

We need Java 1.7 or later. Download and install the latest JDK from Oracle's website: http://www.oracle.com/technetwork/java/javase/downloads/index.html

  1. Change the file mode:
> chmod +x jdk-8u131-linux-x64.rpm
  1. Go to the directory in which you want to perform the installation:
> cd <directory path name>
  1. Run the rpm installer with the command:
> rpm -ivh jdk-8u131-linux-x64.rpm
  1. Finally, add the environment variable JAVA_HOME. This command will write the JAVA_HOME environment variable to the file /etc/profile:
> echo "export JAVA_HOME=/usr/java/jdk1.8.0_131" >> /etc/profile

 Installing Scala in Linux

The following are the steps to install Scala in Linux:

  1. Download the latest Scala binary from: http://www.scala-lang.org/download
  2. Extract the downloaded file scala-2.12.2.tgz:
> tar xzf scala-2.12.2.tgz
  1. Most tutorials agree that the best place to set environment variables is in the /etc/profile file.

  2. Create the SCALA_HOME environment variable:
> export SCALA_HOME=/opt/scala
  1. Add the Scala bin directory to the PATH variable:
> export PATH=$PATH:$SCALA_HOME/bin

Installing Kafka in Linux

The following are the steps to install Kafka in Linux:

  1. Extract the downloaded file kafka_2.10-0.10.2.1.tgz:
> tar xzf kafka_2.10-0.10.2.1.tgz
  1. Create the KAFKA_HOME environment variable:
> export KAFKA_HOME=/opt/kafka_2.10-0.10.2.1
  1. Add the Kafka bin directory to the PATH variable:
> export PATH=$PATH:$KAFKA_HOME/bin

Now Java, Scala, and Kafka are installed.

There's more...

To do all these steps in command-line mode, there is a powerful tool for Mac users called brew (the equivalent in Linux would be yum).

To install from the command line, we use the following steps:

  1. With brew, install sbt (Scala build tool):
> brew install sbt

If you already have it (downloaded in the past), upgrade it:

> brew upgrade sbt

The output is similar to:

> brew upgrade sbt
==> Upgrading 1 outdated package, with result:
sbt 0.13.15
==> Upgrading sbt
==> Using the sandbox
==> Downloading https://github.com/sbt/sbt/releases/download/v0.13.15/sbt-0.13.15.tgz
==> Downloading from https://github-cloud.s3.amazonaws.com/releases/279553/09838df4-23c6-11e7-9276-14
######################################################################## 100.0%
==> Caveats
You can use $SBT_OPTS to pass additional JVM options to SBT:
   SBT_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
This formula is now using the standard lightbend sbt launcher script.
Project specific options should be placed in .sbtopts in the root of your project.
Global settings should be placed in /usr/local/etc/sbtopts
==> Summary
/usr/local/Cellar/sbt/0.13.15: 378 files, 63.3MB, built in 1 minute 5 seconds
  1. With brew, install Scala:
> brew install scala

If you already have it (downloaded in the past), upgrade it:

> brew upgrade scala

The output is similar to:

> brew install scala
==> Using the sandbox
==> Downloading https://downloads.lightbend.com/scala/2.12.2/scala-2.12.2.tgz
######################################################################## 100.0%
==> Downloading https://raw.githubusercontent.com/scala/scala-tool-support/0a217bc/bash-completion/sr
######################################################################## 100.0%
==> Caveats
To use with IntelliJ, set the Scala home to:
/usr/local/opt/scala/idea
Bash completion has been installed to:
/usr/local/etc/bash_completion.d
==> Summary
/usr/local/Cellar/scala/2.12.2: 44 files, 19.9MB, built in 19 seconds
Mist:Downloads admin1$ scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL
  1. With brew, install Kafka (it also installs ZooKeeper):
> brew install kafka

If you already have it (downloaded in the past), upgrade it:

> brew upgrade kafka

The output is similar to:

> brew install kafka
==> Installing dependencies for kafka: zookeeper
==> Installing kafka dependency: zookeeper
==> Downloading https://homebrew.bintray.com/bottles/zookeeper-3.4.9.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring zookeeper-3.4.9.sierra.bottle.tar.gz
==> Using the sandbox
==> Caveats
To have launched start zookeeper now and restart at login:
brew services start zookeeper
Or, if you don't want/need a background service you can just run:
zkServer start
==> Summary
/usr/local/Cellar/zookeeper/3.4.9:
242 files, 18.2MB
==> Installing kafka
==> Downloading https://homebrew.bintray.com/bottles/kafka-0.10.2.0.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring kafka-0.10.2.0.sierra.bottle.tar.gz
==> Caveats
To have launchd start kafka now and restart at login:
brew services start kafka
Or, if you don't want/need a background service you can just run:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
==> Summary
/usr/local/Cellar/kafka/0.10.2.0: 145 files, 37.3MB

See also