Before approaching the problem of parallelizing our crawler, let's discuss our strategy of how we are going to approach it.
A parallel solution with Akka
Strategy
We are going to model our problem of creating the actors' system with a tree of actors. This is because the task itself naturally forms a tree.
We have already discussed how all the links and pages on the internet comprise a graph. However, we have also discussed, in our example, two things that are not desirable when processing and traversing this graph—cycles and repetitions of work. So, if we have visited a certain node, we are not going to visit it anymore.
This means that our graph becomes a tree on processing time. This means you cannot...