Filtering data with Snapkite Engine

The amount of tweets that you'll receive via the Twitter Streaming API is more than you can ever consume, so we need to find a way to filter that stream of data into a meaningful set of tweets that we can display and interact with. I recommend that you take a quick look at the Twitter Streaming API documentation at https://dev.twitter.com/streaming, and in particular, take a look at this page that describes the way you can filter an incoming stream at https://dev.twitter.com/streaming/reference/post/statuses/filter. You'll notice that Twitter provides very few filters that we can apply, so we need to find a way to filter that stream of data even further.

Luckily, there is a Node.js application just for this. It's called Snapkite Engine. It connects to the Twitter Streaming API, filters it using the available filters and according to the rules that you define, and outputs the filtered tweets to a web socket connection. Our proposed React application can listen to the events on that socket connection and process tweets as they arrive.

Let's install Snapkite Engine.

First, you need to clone the Snapkite Engine repository. Cloning means that you're copying the source code from a GitHub server to your local directory. In this book, I'll assume that your local directory is your home directory. Open Terminal/Command Prompt and type the following commands:

cd ~
git clone https://github.com/snapkite/snapkite-engine.git

This should create the ~/snapkite-engine/ folder. We're now going to install all the other node modules that snapkite-engine depends on. One of them is the node-gyp module. Depending on what platform you're using, Unix or Windows, you will need to install other tools that are listed on this web page: https://github.com/TooTallNate/node-gyp#installation.

Once you install them, you're ready to install the node-gyp module:

npm install --global node-gyp

Now navigate to the ~/snapkite-engine directory:

cd snapkite-engine/

Then run the following command:

npm install

This command will install the Node.js modules that Snapkite Engine depends on. Now let's configure Snapkite Engine. Assuming that you're in the ~/snapkite-engine/ directory, copy the ./example.config.json file to ./config.json by running the following command:

cp example.config.json config.json

Or if you're using Windows, run this command:

copy example.config.json config.json

Open config.json in your favorite text editor. We will now edit the configuration properties. Let's start with trackKeywords. This is where we will tell what keywords we want to track. If we want to track the keyword "my", then set it as follows:

"trackKeywords": "my"

Next, we need to set our Twitter Streaming API keys. Set consumerKey, consumerSecret, accessTokenKey, and accessTokenSecret to the keys you saved when you created your Twitter App. Other properties can be set to their defaults. If you're curious to learn about what they are, check out the Snapkite Engine documentation at https://github.com/snapkite/snapkite-engine.

Our next step is to install Snapkite Filters. Snapkite Filter is a Node.js module that validates tweets according to a set of rules. There are a number of Snapkite Filters out there, and we can use any combination of them to filter our stream of tweets as we like. You can find a list of all the available Snapkite Filters at https://github.com/snapkite/snapkite-filters.

In our application, we'll use the following Snapkite Filters:

Let's install them. Navigate to the ~/snapkite-engine/filters/ directory:

cd ~/snapkite-engine/filters/

Then clone all Snapkite Filters by running these commands:

git clone https://github.com/snapkite/snapkite-filter-is-possibly-sensitive.git
git clone https://github.com/snapkite/snapkite-filter-has-mobile-photo.git
git clone https://github.com/snapkite/snapkite-filter-is-retweet.git
git clone https://github.com/snapkite/snapkite-filter-has-text.git

The next step is to configure them. In order to do so, you need to create a configuration file for each Snapkite Filter in JSON format and define some properties in it. Luckily, each Snapkite Filter comes with an example configuration file that we can duplicate and edit as needed. Assuming that you're in the ~/snapkite-engine/filters/ directory, run the following commands (use copy and replace the forward slashes with the backward slashes on Windows):

cp snapkite-filter-is-possibly-sensitive/example.config.json snapkite-filter-is-possibly-sensitive/config.json
cp snapkite-filter-has-mobile-photo/example.config.json snapkite-filter-has-mobile-photo/config.json
cp snapkite-filter-is-retweet/example.config.json snapkite-filter-is-retweet/config.json
cp snapkite-filter-has-text/example.config.json snapkite-filter-has-text/config.json

We don't need to change any of the default settings in these config.json files, as they're already configured to fit our purposes.

Finally, we need to tell Snapkite Engine which Snapkite Filters it should use. Open the ~/snapkite-engine/config.json file in a text editor and look for this:

"filters": []

Now replace that with the following:

"filters": [
  "snapkite-filter-is-possibly-sensitive",
  "snapkite-filter-has-mobile-photo",
  "snapkite-filter-is-retweet",
  "snapkite-filter-has-text"
]

Well done! You've successfully installed Snapkite Engine with a number of Snapkite Filters. Now let's check if we can run it. Navigate to ~/snapkite-engine/ and run the following command:

npm start

You should see no error messages, but if you do and you're not sure how to fix them, then please go to https://github.com/fedosejev/react-essentials/issues, create a new issue and copy/paste the error message that you get.

Next, let's set up our project's structure.