Having a basic design sketched out is often useful, especially in distributed systems where many components will be communicating with each other in different ways. We don't want to spend too long on this stage because our design is likely to evolve as we get stuck into the details, but we will look at a high-level outline so we can discuss the constituents and how they fit together.
The preceding image shows the basic overview of the system we are going to build:
Twitter's streaming API allows long-running connections where tweet data is streamed as quickly as possible.
twittervotes
is a program we will write that reads tweets and pushes the votes into the messaging queue.twittervotes
pulls the relevant tweet data, figures out what is being voted for (or rather, which options are mentioned), and pushes the vote into NSQ.NSQ is an open source, real-time distributed messaging platform designed to operate at scale, built...