There is no better way to gain a deeper understanding of the BSP model principles than to build, from scratch, our very own scalable Pregel-like graph processing system in Go.
Here are a few of the design requirements for the system we will be building:
- Graphs will be represented as a collection of vertices and directed edges. Each vertex will be assigned a unique ID. In addition, both vertices and edges can optionally store a user-defined value.
- At every super-step, the system executes a user-defined compute function for every vertex in the graph.
- Compute functions are allowed to inspect and modify the internal state of the vertex they are invoked on. They can also iterate the list of outgoing edges and exchange messages with other vertices.
- Any outgoing messages that are produced during a super-step will be buffered and delivered to...