Book Image

Oracle Data Integrator 11g Cookbook

Book Image

Oracle Data Integrator 11g Cookbook

Overview of this book

Oracle Data Integrator (ODI) is Oracle's strategic data integration platform for high-speed data transformation and movement between different systems. From high-volume batches, to SOA-enabled data services, to trickle operations, ODI is a cutting-edge platform that offers heterogeneous connectivity, enterprise-level deployment, and strong administrative, diagnostic, and management capabilities."Oracle Data Integrator 11g Cookbook" will take you on a journey past your first steps with ODI to a new level of proficiency, lifting the cover on many of the internals of the product to help you better leverage the most advanced features.The first part of this book will focus on the administrative tasks required for a successful deployment, moving on to showing you how to best leverage Knowledge Modules with explanations of their internals and focus on specific examples. Next we will look into some advanced coding techniques for interfaces, packages, models, and a focus on XML. Finally the book will lift the cover on web services as well as the ODI SDK, along with additional advanced techniques that may be unknown to many users.Throughout "Oracle Data Integrator 11g Cookbook", the authors convey real-world advice and best practices learned from their extensive hands-on experience.

Related Content you might be interested in

Current Title:

Small book image
Oracle Data Integrator 11g Cookbook
May, 2013 | 352 pages
No titles found
Table of Contents (19 chapters)
Oracle Data Integrator 11g Cookbook
Oracle Data Integrator 11g Cookbook
Credits
Credits
Foreword
Foreword
About the Authors
About the Authors
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Installation, Deployment, and Configuration
Installation, Deployment, and Configuration
Introduction
Deploying and configuring a standalone agent manually
Deploying a JEE ODI Agent
Configuring a standalone agent with OPMN
Deploying JDBC drivers with a JEE ODI Agent
2
Defining the Oracle Data Integrator Security
Defining the Oracle Data Integrator Security
Introduction
Setting up LDAP security for Oracle Data Integrator
Setting external authentication with Oracle Data Integrator
Creating users using generic profiles
Creating users using non-generic profiles
Creating new custom profiles in the Security navigator
3
Advanced Topology
Advanced Topology
Introduction
Creating a new technology
Modifying actions to get more from your diagrams
Modifying and expanding datatypes
Changing the case sensitivity for code generation
Best practice – using the Staging Area User to access your target schema
4
Using Variables
Using Variables
Introduction
Passing start-up parameters to a scenario using variables
Using table names that run in all contexts using getObjectName
Using variables to filter data based on a timestamp
Using variables in KM options (and reusing the variables in an interface, package, and so on)
Using variables in topology
Using variables to control loops inside packages
5
Knowledge Module Internals
Knowledge Module Internals
Introduction
Using the substitution passes
Using Java variables in KMs
Using Java for condition code generation
Invoking Java from the KMs
Using substitution methods in Java
Combining substitution methods in a KM
6
Inside Knowledge Modules – SCD and CDC
Inside Knowledge Modules – SCD and CDC
Introduction
Implementing Slowly Changing Dimensions (SCD) using ODI
Modifying a Slowly Changing Dimension KM to allow undefined behaviors
Using Changed Data Capture (CDC) - simple
Using Changed Data Capture (CDC) - consistent set
Using one single interface to load changes that occur in any dimensions
7
Advanced Coding Techniques
Advanced Coding Techniques
Introduction
Using diagrams to develop and maintain models
Generating DDL from data models
Generating interfaces from data models or diagrams
Creating a temporary interface (subquery)
Loading data from an SQL query
Performing a pivot
Loading data using partition exchange
8
Package Loops and File Processing
Package Loops and File Processing
Introduction
Defining packages and loops for near real-time execution using a hybrid loop
Using a file from a parameter variable
Detecting files with a variable name
Processing all files in a directory
Processing a large number of files in parallel
9
XML and Web Services
XML and Web Services
Introduction
Defining a connection to XML within ODI
Processing complex files with ODI
Processing XML data within an RDBMS not in memory
Invoking web services from ODI
Invoking asynchronous ODI web services with callbacks
Configuring container-based authentication with ODI web services
10
Advanced Coding Techniques Using the ODI SDK
Advanced Coding Techniques Using the ODI SDK
Introduction
Creating the Master and Work repositories using the SDK
Creating a project using the SDK
Automating the import of artifacts using the SDK
Creating models and datastores using the SDK
Creating an interface using the SDK
Invoking and monitoring a scenario using the SDK
11
More on ODI
More on ODI
Introduction
Invoking an external program that requires a password
Tuning a standalone ODI agent
Loading a file containing a LOB
Using ODI versioning
Performing a Smart Export/Smart Import
Accessing an Excel spreadsheet
Impacting the data flow by changing the staging area location
Automating Smart Export/Smart Import with the ODI SDK
Index
Index

Deploying a JEE ODI Agent

The ODI Agent can be deployed as a Java EE component within an application server. This installation type allows the ODI agent to take advantage of the benefits of an enterprise application server. When the ODI JEE Agent is deployed within Oracle WebLogic Server, the ODI JEE Agent can leverage the WebLogic's enterprise features, such as clustering and connection pooling for high availability and enterprise scalability. This Java EE Agent exposes an interface enabling lifecycle operation (start/stop) from the application server console and metrics that can be used by the application server console to monitor the agent activity and health.

The ODI 11g Java EE Agent can be deployed to an existing domain or deployed automatically when creating a new domain.

Note

Weblogic Server 10.1.3.6.0 is required for this recipe. Please review the latest Oracle Data Integrator Certification Matrix for the latest version compatibility.

This recipe will create a new WebLogic domain by deploying the Java EE Agent template.

Note

The standard ODI Java EE Agent template will be used to deploy our Java EE Agent. A template for any agent can also be generated from within ODI Studio and then used to deploy this agent.

How to do it...

  1. The Java EE Agent must exist in the ODI topology before the WLS domain server is started for the agent. Connect to ODI Studio, expand ODI Agents within the topology, and add an agent with the name OracleDIAgent and with the port 8001.

  2. To deploy and configure domains with WLS, execute config.bat or config.sh from within the ODI Home Install at Middleware_HOME\ODI_HOME\common\bin.

    Tip

    Downloading the example code

    You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

  3. Select the Create a new Weblogic domain option and click on Next.

  4. Select Oracle Data Integrator - Agent - 11.1.1.0 [Oracle_ODI1], which will additionally select the two options as shown in the following screenshot, and click on Next:

  5. For this recipe, accept the defaults and click on Next.

  6. Accept the default name as weblogic, set the password as weblogic1, and click on Next.

  7. Select SUN JDK and click on Next.

  8. Enter the appropriate connection information to connect to the Master repository and click on Next.

  9. Ensure that the test connection was successful and click on Next.

  10. Check the Managed Servers, Clusters and Machines option and click on Next.

  11. Set the port to 8001, accept the defaults, and click on Next.

  12. Click on the Next button on the Configure Clusters screen.

  13. Accept the defaults and click on Next on the Configure Machines screen.

  14. Click on Next on the Assign Servers to Machines screen.

  15. Review the Configuration Summary screen and click on Create.

  16. Do not click on the Start Admin Server check box. Then, click on Done.

  17. Start the ODI WLS admin server for this domain from the command shell. Execute startweblogic.cmd or startweblogic.sh from Middlware_Home\user_projects\domains\base_doman\bin.

  18. Security has to be set up for the JAVA EE application to have access to the ODI repository. For this access, an entry needs to be created within the credential store that will allow the JAVA EE Agent to authenticate itself and consume the resources that are needed. This user must already be set up in the ODI Security. To do this, we will do the following:

    1. Execute WLST, connect to our running admin server, and add the credential store.

    2. Start WLST from a command shell and change the directory to Middleware_home\odi_home\oracle_common\common\bin.

    3. Execute WLST.

  19. Execute the following command to connect to the running admin server:

    connect ('weblogic','weblogic1','t3://localhost:7001').

  20. Execute the following command to add the correct credential store for ODI Supervisor:

    createCred(map="oracle.odi.credmap", key="SUPERVISOR", user="SUPERVISOR", password="SUNOPSIS", desc="ODI SUPERVISOR Crendential")

    Note

    During runtime, the JAVA EE Agent will access this credential store to authenticate itself.

    1. Type exit() to close WLST.

    2. Start a command shell and change the directory to the user_projects directory of the Middleware_Home - Middleware_HOME\user_projects\domains\base_domain\bin.

    3. Execute StartManagedWeblogic odi_server1.

  21. Enter weblogic as the username and weblogic1 as the password.

  22. Verify base_domain is in the running mode and that there are no stack trace errors.

  23. Verify connectivity to Java EE Agent through ODI Studio Topology.

How it works...

Oracle has documented the overall process of installation and configuring the ODI JEE Agent. However, installation and deployment are broken into different sections of the documentation. This recipe gives a quick walk-through of the steps needed to easily install the JEE ODI Agent and also the steps necessary to configure the agent on the WebLogic server as well as the updates required within the ODI repository.

There's more...

The ODI JEE Agent can easily take advantage of the enterprise scalability features of Weblogic Server. Setting up clustering with the ODI JEE Agent is straightforward and follows the same setup as above. However, two agents would be created and then clustered. This is outlined in the Fusion Middleware High Availability Guide at http://docs.oracle.com/cd/E14571_01/core.1111/e10106/odi.htm.

There are also many more exciting ODI JEE components that are also easily installed and deployed, including the ODI Console, the ODI Admin Plug-in for the FMW EM Control, and the SDK Web Services, which are outlined at http://docs.oracle.com/cd/E21764_01/core.1111/e16453/configure.htm#autoId9.

Previous Section
End of Section 4
Next Section