When we work with very large datasets, often we talk about structuring our program concurrently. But one big problem when dealing with very large datasets concurrently is coordinating and managing the flow of data between different parts of our program. If one part produces data too quickly, or another part processes it too slowly, depending on how you look at it, the message queue between the two can get backed up. If that happens, memory will get filled up with the messages and data waiting to be processed.
The solution for this in Clojure is quite simple: use seque. This uses an instance of java.util.concurrent.LinkedBlockingQueue to pull values from a lazy sequence. It works ahead of where we're pulling values out of the queue, but not too far ahead. And once we've wrapped a sequence with seque, we can treat it just like any other sequence.
user=> (take 20 (seque 5 (range Integer/MAX_VALUE))) (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17...