Long-running processes

Normally, these processes are controlled by ugly behemoths called batch jobs or scheduled tasks. They run in the dead of night and update our data depending upon the business rules of the day. But what happens when this is no longer good enough?

The data needs of most companies are growing year on year. What happens when the database contains so many records that the nightly batch job takes longer than off-peak hours will permit, and causes too much of a performance penalty to be run during peak hours? What happens if you work in an industry where there is no such thing as off-peak hours? And, of course, let's not forget that most batch jobs are notoriously brittle and prone to failure.

Perhaps, the more troublesome problem with the old way of doing things is our own user's expectations. Consider the frequent flyer that has just flown the last mile to achieve Gold status. They can check their miles on their smartphone app, and they know they should be able...