Book Image

BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation

By : Qiang Ding
Book Image

BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation

By: Qiang Ding

Overview of this book

Control-M is one of the most widely used enterprise class batch workload automation platform. With a strong knowledge of Control-M, you will be able to use the tool to meet ever growing batch needs. There has been no book that can guide you to implement and manage this powerful tool successfully... until now. With this book you will quickly master Control-M and be able to call yourself "a Control-M" specialist! "BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation" will lead you into the world of Control-M and guide you to implement and maintain a Control-M environment successfully. By mastering this workload automation tool, you will see new opportunities opening up before you. With this book you will be able to take away and put into practice knowledge from every aspect of Control-M ñ implementation, administration, design and management of Control-M job flows, and more importantly how to move into workload automation and let batch processing utilize the cloud. You will start off with batch processing and workload automation, and then get an understanding of how Control-M meets these needs. Then we will look more in depth at the technical details of Control-M, and finally look at how to work with it to meet critical business needs. Throughout the book, you will learn important concepts and features, as well as learn from the Author's experience, accumulated over many years. By the end of the book you will be set up to work efficiently with this tool and also understand how to utilize the latest features of Control-M.
Table of Contents (16 chapters)
BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation
Credits
About the Author
Acknowledgement
About the Reviewers
www.PacktPub.com
Preface
Index

Introduce batch processing


We hear about hot IT topics everyday, everywhere. Pick up a tech magazine, visit an IT website, or subscribe to a weekly newsletter and you will see topics about cloud computing, SOA, BPM/BPEL, data warehouse, ERP — you name it! Even on TV and at the cinemas, you may see something such as an "Iron Man 2 in theatres soon + Sun/Oracle in data centre now" commercial. In the recent years, IT has become a fashion more than simply a technology, but how often do you hear the words "batch processing" mentioned in any of the articles or IT road shows?

The history of batch processing

Batch processing is not a new IT buzz word. In fact, it has been a major IT concept since the very early stage of electronic computing. Unlike today, where we can run programs on a personal computer whenever required and expect an instant result, the early mainframe computers could handle non-interactive processing only.

In the beginning, punched cards were used as the media for storing data (refer to the following). Mainframe system programmers were required to store their program and input data onto these cards by punching a series of dots and pass them to the system operators for processing. Each time the system operators had to stack up the program cards followed by the input data cards onto a special card reader so the mainframe computer could load the program into memory and process the data. The execution could run for hours or days and it would stop only when the entire process was complete or in case an error occurred. Computer processing power was expensive at that time. In order to minimize a computer's idle time and improve efficiency, the input data for each program was normally accumulated in large quantities and then queued up for processing. In this manner, lots data could be processed at once, rather than frequently re-stacking the program card multiple times for ea ch small amount of input data. Therefore, the process was called batch processing.

This is a file from the Wikimedia Commons. Commons is a freely licensed media file repository (http://en.wikipedia.org/wiki/File:Blue-punch-card-front-horiz.png).

With time, the punched card technology lost its glory and became obsolete and it was replaced by much more advanced storage technologies. However, the batch mode processing method amazingly survived and continued to play a major role in the computing world to handle critical business tasks. Although the surrounding technology has changed significantly, the underlying batch concept is still the same. In order to increase efficiency, programs written for batch-mode process (a.k.a batch jobs) are normally set to run when large amounts of input data are accumulated and ready for processing. Besides, a lot of routine procedure-type business processes are naturally required to be processed in batch mode. For example, monthly billing and fortnightly payrolls are typical batch-oriented business processes.

Note

The dictionary definition of batch processing: (Computer Science) A set of data to be processed in a single program run.

Traditionary Batch jobs are known to process large amounts of data at once. Therefore, it is not practical to expect the output to be given immediately. But there is still a predefined deadline for every batch processing, either it is set by the business requirement (also known as SLA — Service Level Agreement) or simply because it needs to finish so that the dependent batch processing tasks can start. For example, a group of batch jobs of an organization need to generate the payroll data and send it to the bank by Monday 5am, so the bank can have enough time to process it and ensure the funds get transferred to each of the organizaiton's employee's account by 8am Tuesday morning.

The rules and requirements of those business processes that require batch processing are becoming more and more sophisticated. This makes batch processing not only very time-consuming, but also task intensive, that is, the business process is required to be achieved in more than one step or even by many related jobs one after another (also known as job flow). In order for the computer system to execute the job or a job flow without human interaction, relevant steps within a job or jobs within a job flow need to be prearranged in the required logical order and the input data needs to be ready prior to the runtime of the job's step.

Batch processing versus interactive processing

As the time goes by, computer systems got another major improvement by having the ability to handle users' action-driven interactive processing (also called transaction processing or online processing). This was a milestone in computing history because it changed the way human minds work with computers forever, that is, for certain types of requests, users no longer need to wait for the processing to happen (only in batch mode during a certain period of time). Users can send a request to the computer for immediate processing. Such requests can be a data retrieving or modifying request. For example, someone checking his or her bank account balance on an ATM machine or someone placing a buy or sell order for a stock through an online broking website. In contrast to batch processing, computer systems handle each of the user requests individually at the time when it is submitted. CICS (Customer Information Control System) is a typical mainframe application designed for handling high-volume online transactions on the other hand, there is personal computer started to gain popularity which designed and optimised to work primarily in interactive mode.

In reality, we often see that batch processing and transaction processing share the same computing facility and data source in an enterprise class computing environmnet. As interactive processing aims at providing a fast response for user requests generated on a random basis, in order to ensure that there are sufficient resources available on the system for processing such requests, the resource intensive batch jobs that used to occupy the entire computing facility 24/7 had to be set to run only during a time frame when user activities are low, which back at the time is more likely to be during night, that is, as we often hear a more seasoned IT person with mainframe background call it nightly batch.

Here's an example of a typical scenario in a batch processing and transaction processing shared environment for an online shopping site:

  • 7:00am: This is the time usually the site starts to get online traffic, but the volume is small.

  • 10:00am: Traffic starts to increase, but is still relatively small. User requests come from the Internet, such as browsing a product catalog, placing an order, or tracking an existing order.

  • 12:00pm: Transaction peak hours start. The system is dedicated for handling online user requests. A lot of orders get generated at this point of time.

  • 10:00pm: Online traffic starts to slow down.

  • 11:30pm: A daily backup job starts to back up the database and filesystem.

  • 12:00am: A batch job starts to perform daily sales conciliations.

  • 12:30pm: Another batch job kicks in to process orders generated during the last 24 hours.

  • 2:00am: A multi-step batch job starts for processing back orders and sending the shop's order to suppliers.

  • 3:00am: As all outstanding orders have been processed, a backup job is started for backing up the database and filesystem.

  • 5:00am: A batch job generates and sends yesterday's sales report to the accounting department.

  • 5:15am: Another batch job generates and sends a stock on hand report to the warehouse and purchasing department.

  • 5:30am: A script gets triggered to clean up old log files and temporary files.

  • 7:00am: The system starts to hit by online traffic again.

In this example, programs for batch mode processing are set to run only when online transactions are low. This allows online processing to have the maximum system resources during its peak hours. During online processing's off peak hours, batch jobs can use up the entire system to perform resource-intensive processing such as sales conciliation or reporting.

In addition, because during the night time there are fewer changes to the data, batch jobs can have more freedom when manipulating the data and it allows the system to perform the backup tasks.

Time-based batch- and event-driven batch

What we have discussed so far Batch processing defined to run during a certain time is traditional time-based scheduling. Depending on the user's requirements, it could be a daily run, a monthly run, or a quarterly run, such as:

  • Retail store doing a daily sales consolidation

  • Electricity companies generating monthly bills

  • Banks producing quarterly statements

The timeframe allocated for batch processing is called a batch window. The concept sounds simple, but there are many factors that need to be taken into consideration before defining the batch window. Those factors include what time the resource and input data will be available for batch processing, how much data needs to be processed each time, and how long the batch jobs will take to process them. In case the batch processing fails to complete within the defined time window, not only does the expected batch output be delivered on time, but the next day's online processing may also get affected. Here are some of the scenarios:

  • Online requests started to come in at its usual time, but the backend batch processing is still running. As the system resource such as CPU, memory, and IO are still occupied by the over-running batch jobs, the resource availability and system response time for online processing are significantly impacted. As a result, online users see responses given slowly and get timeout errors.

  • Some batch processing needs to occupy the resource exclusively. Online processing can interrupt the batch processing and cause it to fail. In such cases, if the batch window is missed, either the batch jobs have to wait to run during the next batch window or online processing needs to wait until the batch processing is completed.

  • In extreme cases, online transactions are based on the data processed by the previously run batch. Therefore, the online transactions cannot start at all unless the pervious batch processing is completed. This happens with banks, as you often hear them say the bank cannot open tomorrow morning if the overnight batch fails.

A concept called event-triggered scheduling was introduced during the modern computing age to meet the growing business demand. An event can be as follows:

  • A customer submitted an order online

  • A new mobile phone SIM card was purchased

  • A file from a remote server arrived for further processing

Rather than accumulating these events and processing them during the traditional nightly batch window, a mechanism has been designed within the batch processing space to detect such an event in real-time and process them immediately. By doing so, the event initiators are able to receive an immediate response, where in the past they have to wait until the end of the next batch window to get the response or output. Use the online shopping example again; during the day, orders get generated by online users. These orders are accumulated on the system and wait to be processed against the actual stock during the predefined batch windows. Customers have to wait till the next morning to receive the order committed e-mail and back order items report. With event-triggered batch processing, the online business is able to offer a customer an instant response on their order status, and therefore, provide a better shopping experience.

On the other hand, as a noticeable amount of batch processing work is spared during event time, the workload for batch processing during a batch window (for example, a nightly batch) is likely to be reduced.

Is this the end for batch processing?

There have been talks about totally replacing time-based batch processing with real-time event-driven processing to build a so called real-time enterprise. A group of people argue that batch processing causes latency to business processes and as event-driven solutions are becoming affordable, businesses should be looking at completely shifting to event-driven real-time processing. This approach has been discussed for years. However, its yet to completely replace batch processing.

Shifting the business process into real-time can allow businesses to have quicker reaction to changes and problems by making decisions based on live data feeds rather than historical data produced by batch processing. For example, an online computer store can use a real-time system to automatically adjust their retail price for exchange rate sensitive computer components according to live feed currency exchange rate.

The business may also become more competitive and gain extra profit by having each individual event handled in real time. For example, mobile phone companies would rather provide each SIM card as soon as it is purchased, than let the customers wait until the next day (that is when the over-night batch processing finish processing the data) and lose the potential calls that could be charged during the waiting time.

Note

Case study: Commonwealth Bank of Australia shifting into real-time banking

"In October 2010, Commonwealth Bank of Australia announced their future strategy in IT, that is, progressively upgrading their banking platforms to provide real-time banking for customers. Real-time banking is the ability to open and close an account, complete credit and debit transactions, and change features immediately. Real-time banking removes the delay we experience now from batch processing, where transactions are not completed until the next working day. Real-time banking will gradually replace batch processing, as our new banking platform replaces existing systems over the next few years."

Reference from Commonwealth Bank of Australia Website.

However, real-time processing is not a silver bullet. Although purchasing real-time solutions is becoming cheaper, moving the organization's entire batch processing into real-time processing can be an extremely costly and risky project. Also, we may find that current computing technology is still not powerful enough to handle everything in real time. Some IT professionals did a test during a project. They designed the system to process everything in real time and when the workload increased, the entire system just collapsed. We know hardware is becoming cheaper these days, but machine processing power is not the only limitation. When we talk about limited computing resources, it also can be the number of established sessions to a database or the availability of information for access. In case of a large request, such as generating a report, it may require exclusive access to the database. During the processing of data within the table, it is not available for other requests to modify. Therefore, such requests should be considered as batch processing and configured to run at an appropriate time.

The bottom line is not all business processes are required to be completed in real time. For example, a retail store only needs to generate purchase orders once a week. The store has no reason to transfer this business process into real-time processing, because they want to accumulate individual stock requests over a period of time and send it to the supplier in a single order. In this case, the shop can receive goods in bulk and save shipping cost, and at the same time there is a possibility to receive special offers that are given based on large order quantity.

Note

ZapThink's Ron Schmelzer wrote:

Batch processing, often thought of as an artifact left over from the legacy days, plays a vital role in systems that may have real-time processing upfront. As he observed, "behind the real-time systems that power the real-time enterprise are regularly-updated back office business systems. Batch processes remain essential for one key reason: it is simply not efficient to regenerate a complete forecast or business plan every time the business processes a single event such as an incoming customer order."

Reference from ZapThink: http://www.zdnet.com/blog/serviceoriented/soas-final-frontier-can-we-should-we-serviceorient-batch-processing/1439

Either running the business process in real time or batches, the system designer should take many factors into consideration, such as:

  • Is the business process required to be running in real time?

  • What will the cost be to run the business process in real time?

  • What are the negative impacts on others, if running the business process in real time?

  • Will the benefit justify the cost and impact?

Moving IT into real time should be driven by actual business needs rather than the technology. The person who is going to design the system needs to carefully consider what needs to be processed in real time and what can be processed in batch to meet the business requirements and balance the system utilization. As a fact, large orgnisations today are still investing in batch processing and continuously trying to figure out how to make it run better.

Running batch processing tasks

To understand batch processing further we need to begin from it's father - the mainframe computers. Job Control Language (JCL) was introduced as a basic tool on the mainframe computers for defining how a batch program should be executed. A JCL is considered to be a job, also called a job card (inherited the name from punched cards). A job can have a single step or multiple steps (up to 255 steps in each JCL), and each step is an executable program or a JCL procedure (frequently used JCL statements are define into procedures for reusability). In JCL, a user needs to specify the name of the job, the program, or procedure to be executed during each step of the job, as well as the input and output of the step. Once the job is submitted for execution, the Job Entry Subsystem (JES) will interpret the JCL and send it to the mainframe operating system (MVS or Z/OS) for p rocessing (refer to the next diagram).

The system will read the submitted JCL to figure out what application to run, the location of the input, and where the output should go to.

Batch processing can be a standalone task, but in common cases business processes require many steps to deliver. Technically, all these steps can be combined within one processing unit (for example, steps with a JCL), but if a step fails during the processing, rerunning that specific step can be challenging. There are third-party products provided on the mainframe computers just for managing rerun JCL steps.

On the distributed systems, it is up to the program designer to design their own way in the program to detect the error and handle such rerun action. In most cases, we have to restart the entire process from the beginning. In order to have flexibility and ease of management in batch processing, these steps are normally broken down into individual processing units (also known as jobs) and are set to be executed in sequence. By doing so, when a step fails in the middle of the processing or towards the end of the entire processing, a rerun can be allowed from the point of failure easily by rerunning the problem job.

Here is a Hello World edition of batch processing. imagamingpc.com is an online computer shop specialized in build-to-order gaming computers. The owner implemented his own order processing application to handle online orders. The system works in the following way:

  1. During the day, customers visit the site. They choose each PC component to create a customized machine specification.

  2. Once the customer submits the completed customized machine specification, a flat file gets generated in the system and tells the customer that the order has been received.

  3. By 12:00am, let's say there are ten orders generated in total. A program is set to trigger around this time of the day to process these orders. For each order, the program first will check and make sure the parts the customer selected can be built into a complete working computer, whereas the order gets rejected if the components are incompatible with each other or some essential parts are missing. It is followed by sending an e-mail to the customer for correcting the order. For orders that passed the check, the program will scan the inventory to make sure each item is in stock, and sends an order confirmation e-mail to the customer. The program also generates and prints a daily build list. Sometimes there are orders that cannot be built due to missing parts, such as a specific graphics card being out of stock. In this case, the program will generate and print a backorder list and send the pending build e-mail to the customer to confirm their parts are being orde red and the system build is pending.

  4. The next morning, the technician comes in and builds each machine according to the printed daily build list. At the same time, the person in charge of purchasing and procurement will place orders to each supplier according to the backorder list.

This is a very simple batch processing example. It's not hard to figure out that step 2 is an interaction processing and step 3 is a batch processing.

One of the very early challenges was that the order processing program often fails somewhere during processing. Sometimes it fails during generating the daily build list and sometimes it fails to send out e-mails to customers for pending build notification. No matter at which step the program fails, the program needs to be re-run right from the beginning, even if the problem was at the last stage only. As a result, every time when a re-run happens, the customer will likely be getting two confirmation e-mails. Also, when the number of orders is large on a given day, rerunning the entire thing can take a lot of time.

The owner of the business realized the problem and requested an IT person to come up with a better way to run this problem. Pretty quickly, the IT person came up with an idea, that is, break down each processing step into a separate program. In this case, if any stage failed, you would only need to re-run the failed task. As an outcome, the batch processing became the following:

  • By 12:00am, there are ten orders generated in total. Program PROCESS_ORDER is set to trigger at this time of the day to process them. For each order, the program scans the inventory to make sure each item on the order is in stock and create the file called daily build list.

  • According to the daily build list, the program MAIL_CONFRIMED_ORDER sends out an e-mail to each customer to confirm if his or her order is ready to be built.

  • Another program called PRINT_DAILY_BUILD_LIST prints the daily build list for the technician who is going to build the machines on the following day.

  • Program GENERATE_BACKORDER_LIST will create a list of orders that cannot be built due to missing parts, the list also gets stored in a file called backorder list.

  • According to the backorder list, the program MAIL_BACKORDER sends out an e-mail to each customer to notify them that their order has been backordered.

  • Another program called PRINT_BACKORDER_LIST prints the backorder list for the purchasing and procurement officer.

By now, we have a series of programs linked together and executed in sequence to handle the business process, as opposed to having one program that does everything. In this case, if the processing stopped half way through, the IT person can easily re-run that particular step followed by the rest, rather than executing the entire process from the beginning. Therefore, duplicate e-mails are no longer sent and the time required for re-processing is significantly reduced.

Batch processing is about making the best use of computing resources to process data and deliver result on time. Schedule jobs so that they can run imminently one after another, and keep the computing resource occupied without idle time. But this may not be considered as highly efficient with today's high processing power machines that have the ability to handle multi-tasking. Dividing steps into jobs provides a possibility of parallel processing for job steps that are not inter-dependent, as a result, maximizing the system resource utilization as well as shortening the end-to-end processing time. Use our online computer shop example by breaking down processing into tasks and running tasks that are not interdependent in parallel, we can significantly shorten the processing time.

Another objective of batch processing is about delivering the processing result with minimal manual intervention. Technically, batch processing can be initiated by humans. In fact, back in the old days, companies employed operators on shifts to submit job streams at a set time of the day, check each job's execution result, and initiate restarts. However, running batch manually can be extremely challenging due to the nature of batch processing that it is complicated and detailed. The number of tasks for a business process can be large, and each task may require a complex input argument to run. Depending on the characteristics of the task an execution can take anything from minutes to hours.

In an environment that requires hundreds or thousands jobs to be scheduled each day, people who run the batch job not only need to work long hours to monitor each execution and ensure the right batch job gets triggered with correct input arguments at the correct time, but they also need to react to job failures, handle the complicated logic between jobs to decide which job to run next, and at the same time, keep in mind parallel processing, as well as ensure that each job is triggered as soon as its parent job(s) is/are completed to minimize machine idle time. Making mistakes is an unavoidable part of human nature, especially in a complex environment where everything is time-constrained. Businesses would rather invest in a batch automation tool than take the risk of having critical business problems due to batch processing delay and failures by human mistakes.