Any time you buy something with a credit card, make a phone call, pay for a doctor's visit, or pay your mortgage, there's a high probability that the transaction was processed in a nightly batch cycle using flat file FTP-based integration. The transaction is typically flat file FTP'd to your company's data center. This flat file FTP-based solution is data integration at its core, and the processing of these flat files is part of your overall business EII architecture. These batch processing information integration components can be part of a job scheduler, a BPEL flow, a workflow engine, or an ETL/ELT product. The information integration could also be custom coded using COBOL, Perl, Java, Oracle PL/SQL, and OS scripts that make database SQL calls.
As more companies adopt an enterprise approach to information integration and define SOA strategies, a fundamental question asked is: Do I put my current flat file batch processing into a job scheduler, an ETL/ELT tool, a workflow engine, or in BPEL? Is there a single product or tool that can manage both the business processing and information integration solutions that are required? If not, how can the various products and tools integrate? Legacy mainframe systems are typically straightforward, as business processing is typically achieved in batches through a job scheduler, and the remaining workflow and business processes will be part of the COBOL, PL/1, or Assembler application. On distributed systems, the question becomes even more complicated. Part of the processing may happen in a job scheduler, some in a business process manager, some in an ETL/ELT tool, and some in a workflow engine.
There's some industry discussion that BPEL engines can provide the same type of functionality as a job scheduler. Fundamentally, job schedulers and BPEL Process Managers are machine and human workflow engines. However, this greatly simplifies the number of duties job schedulers perform to provide you with enterprise-wide batch load scheduling and management. Job schedulers provide functionality that isn't core to BPEL:
Platform support: The ability to execute jobs on any popular hardware or operating system platform. This also includes integrated scheduling of z/OS, Linux on mainframe System z, iSeries, Tandem, Unix, Windows, and Linux workloads.
Job priority: Service classes (job priority) are built into mainframe Job Entry Subsystems (JES) from the outset to provide a mechanism for prioritizing batch processing, based on resource requirements and job priority.
Calendaring: This isn't just a simple system that starts a job at a specific time or date. This is a complex calendaring system with the ability to change dependencies and view daily, monthly, quarterly, and yearly job schedules. It offers easy configuration and maintenance of even the most complex schedules, letting users maintain a single job schedule calendar across the enterprise.
Proactive critical path analysis: Job schedulers automatically calculate and display job dependencies and analyze the critical path. Proactive notification, in the event it appears, means the critical path is in jeopardy of completing on time. Also, the ability to analyze the impact of a change in the job schedule, or a new job being added before it's put into production, are tasks the job scheduler performs automatically.
Interface to hardware devices: The job scheduler is proactive and can check the availability of peripheral devices, such as storage and printers, before it executes a job that's dependent on a device that's offline or not functioning properly.
Job scheduling systems served legacy systems well, when most of the processing happened in batches at night using flat file integration. New business processes must typically be web service-ready, because they're expected to communicate in a bidirectional manner with service-orientated technologies such as BPEL and ESBs. Today, BPEL Process Managers have become a core component for business process execution in service-orientated infrastructures on distributed systems and are taking the place of job scheduling products.
ETL/ELT products have both the capabilities of a BPEL Process Manager and job scheduler:
Workflow: The ability to graphically design the flow of information through the ETL/ELT process and then deploy this workflow to an integrated workflow engine. The workflows in ETL/ELT products are typically as 'smart' as job schedulers. Job schedulers have the built in capability to determine the best path to use.
Human interaction: ETL/ELT workflows offer the ability to add human intervention at any point in the processing of data.
Basic job scheduling: Basic job scheduling to including starting a ETL/ELT workflow at a specific time, looking for input files and then starting processing, and process dependencies are all part of ETL/ELT products.
Web services integration: The ability to expose any component or workflow as a web service (publish web services) and also the ability to consume web services.
Distributed systems job schedulers are available from several job scheduling companies, including ORSYP (Dollar Universe), UC4 (AppWorx), and ActiveBatch (Advanced Systems Concept, Inc). Oracle also has a free job scheduler that is part of the Oracle Database. As it is built into the Oracle Database, it is multi-platform and has all the reliability, availability, scalability, and performance that are part of the Oracle Database. Open systems business automation processing is typically achieved through BPEL Process Manager products. BPEL Process Managers let you orchestrate Web service implementations, human interaction, and system workflow quickly and easily using graphical, drag-and-drop techniques. The tools are end user-focused, allowing users to orchestrate their own systems. The execution language for orchestration is BPEL, and the runtime engine is Java EE. BPEL supports human and automated orchestration. Enterprises typically use BPEL Process Managers to streamline and rethink existing business processes.
The business information processing and information integration solution of the future will most likely be business service automation through the grid, cloud computing, dynamic workload automation, and job management as a service. This means workflow products, including job schedulers and BPEL Process Managers, will probably play an important role in batch-based information integration. Job schedulers, BPEL engines, and ETL/ELT products can be combined, leveraging the strengths of each to implement a batch-based information integration solution. ETL/ELT products will most likely continue to form the basis for the initial processing of flat files, data transformation, and database loading and extract flat files to and from relational databases.