As already seen in Chapter 3, CLR Internals, CLR has some limitations in memory management.

Working with engineering data means having to deal with a huge dataset of more than a million records.

Although we can load a simple integer array with millions of items in memory, the same thing will be impossible when the number rises by a lot, or the data type becomes heavier than a simple integer value.

The .NET has a complete enumerator-like execution model that can help us handle a billion items without ever having to deal with all such items in memory, all together. Here is an example on sliding processing:

static void Main(string[] args)
{
    //dataset
    //this dataset will not be streamed until needed
    var enumerableDataset = RetrieveHugeDataset();

    //start using the enumerable
    //this will actually start executing code within RetrieveHugeDataset method
    foreach (var item in enumerableDataset)
        if (item % 12 == 0)
            Console.WriteLine("->...