In the job observer pattern, you can create an Auto Scaling group, based upon the number of messages in the queue to process. The job observer pattern helps you to maintain performance through increasing or decreasing the number of server instances used in job processing.
The following diagram depicts the job observer pattern:
In the preceding architecture, the first fleet of Amazon EC2 servers is on the left-hand side, running batch jobs and putting messages in the queue, for example, image metadata. The second fleet of EC2 servers on the right-hand side is consuming and processing those messages, for example, image encoding. As the message reaches a certain threshold, Amazon CloudWatch triggers Auto Scaling to add the additional server in the consumer fleet to speed up the job processing. Auto Scaling also removes additional servers when the queue depth goes below the threshold.
The job observer pattern computes scale with job...
