As mentioned earlier, complete dataset retrieval is something that is poorly optimized in any application scenario. When dealing with a huge dataset, trying to load more than a million rows together in the same materialized Entity Framework query or something similar will surely result in an OutOfMemoryException output.

As seen in the Stream-like querying section in Chapter 7, Database Querying, ADO.NET gives us the ability to execute queries without having to put all the data together in our memory, as we would have to with the old DbCommand class or new Entity Framework ones. Sometimes, we are in the need of executing a lot of logic that needs frequent lookups at a data source. Let's talk about the previous example again.

What if a GPS device always sends the same position? In our system, we will take this position, produce a Stage 1 message, send it to the Stage 1 queue, and then down to our grid engine, which will make a request to our database to check whether such...