Note
Note: This section is optional. It demonstrates how to re-implement parts of the data pipeline with cloud-based streaming services to achieve greater scalability
Implementing the entire data pipeline in a single Notebook gave us high productivity during development and testing. We can experiment with the code and test the changes very rapidly with a very small footprint. Also, performances have been reasonable because we have been working with a relatively small amount of data. However, it is quite obvious that we wouldn't use this architecture in production and the next question we need to ask ourselves is where are the bottlenecks that would prevent the application from scaling as the quantity of streaming data coming from Twitter increases dramatically.
In this section, we identify two areas for improvement:
In the Tweepy stream, the incoming data is sent to the
RawTweetsListener
instance for processing using the...