Copyright 2010, Oracle. All rights reserved.

PTS Workshop: ODI 11g

<Insert Picture Here>

Oracle Data Integrator Architecture

Agenda
ODI Architecture Overview ODI Components Agents Console Repositories Data Flow Q&A
<Insert Picture Here>

Copyright 2010, Oracle. All rights reserved.

ODI Architecture

Copyright 2010, Oracle. All rights reserved.

ODI Architecture
OpenTools for adding OS functionality to any ETL job

Open Web Services for decoupled invocation of any ODI jobs

Desktop
JVM
Java Application ODI SDK

WebLogic 11g / Application Server

FMW Console
ODI Plug-in

ODI Studio
Designer Operator

Servlet Container
Java EE Application Runtime WS

Web Service Container

Data Services

Topology
Security

ODI Console
ODI SDK

Java EE Agent

Public WS

SDK APIs for Java generation of any ODI system metadata

Open Web Services for decoupled invocation of any ODI jobs

Data Sources Connection Pool

Repositories
ODI Master Repository ODI Work Repository Legacy

Sources and Targets

JVM
Runtime WS Standalone Agent

Metadata is an open relational model with built-in Flex Fields

Files / XML

DBMS

Applications ERP/CRM/PLM/SCM

DW / BI / EPM

Copyright 2010, Oracle. All rights reserved.

ODI Components

Copyright 2010, Oracle. All rights reserved.

Graphical Modules
ODI Studio

Designer Reverse-Engineer Develop Projects Release Scenarios Java Most Platforms DB - Most ISO-92

Operator Operate production Monitor sessions

Topology Manager Define the infrastructure of the IS

Security Manager Manage user privileges

Repository

Copyright 2010, Oracle. All rights reserved.

SOA / Java EE Run-Time Components

ODI Studio
Any Client App
Consume Data Control EL-T Operations

Designer Reverse-Engineer Develop Projects Release Scenarios

Operator Operate production Monitor sessions

Monitor sessions View Reports

Java - Any Platform

Invoke and manage sessions

Access Data

Submit Jobs

Repository Any ISO-92 RDBMS Data Services Access Data and Changed Data WebLogic Java EE server Public Web Services Manage sessions Access Data Java EE Agent Handles schedules Orchestrate sessions
Read sessions Write reports

Return Code

Execute Jobs

Information System

Copyright 2010, Oracle. All rights reserved.

Standalone Run-Time Components

Designer Reverse-Engineer Develop Projects Release Scenarios Java - Any Platform

Operator Operate production Monitor sessions

Monitor sessions View Reports

Submit Jobs

Repository

Any ISO-92 RDBMS Standalone Agent Handles schedules Orchestrate sessions Java - Any Platform

Read sessions Write reports

Lightweight Distributed Architecture

Return Code

Execute Jobs

Information System

Copyright 2010, Oracle. All rights reserved.

Enterprise Manager and ODI Console

Any Web Browser Control and monitor ODI production Browse metadata lineage

Repository Any ISO-92 RDBMS Enterprise Manager Fusion Middleware Console Java EE Agent Handles schedules Orchestrate sessions WebLogic Java EE server
Submit Executions

ODI Console Web access to the repository

Other components SOA Suite, WLS Servers, Web apps.

Return Code

Execute Jobs

Information System

Copyright 2010, Oracle. All rights reserved.

Components: a Global View

ODI Studio Any Client App
Consume Data Control EL-T Operations

Designer
Reverse-Engineer Develop Projects Release Scenarios

Operator
Operate production Monitor sessions

Topology Manager
Define the IS infrastructure

Security Manager
Manage user privileges

Any Web Browser Browse metadata lineage Operate production

Java - Any Platform

Invoke and manage sessions

Access Data

Repository Any ISO-92 RDBMS Data Services Access Data and Changed Data Java - Any Platform Information System Repository Access HTTP Connection Execution Query
Copyright 2010, Oracle. All rights reserved.

Public Web Services Manage sessions Access Data

Java EE Agent Handles schedules Orchestrate sessions

Enterprise Manager FMC ODI Console

Web access to the repository

Other components
SOA Suite, WLS Servers, Web apps.

Designer Navigator
Toolbar

Designer Projects editor Object tree

Other Designer editors

In ODI Designer, you can create, configure, and execute various ODI objects. Workspace to define ODI objects and design EL-T transformations

Copyright 2010, Oracle. All rights reserved.

Operator Navigator

Toolbar

In Operator Navigator, you can monitor execution of ODI objects.

Workspace to define ODI objects and design EL-T transformations

The monitoring session steps enable you to perform debugging.

Copyright 2010, Oracle. All rights reserved.

Topology Navigator

Toolbar

Technologies tree Workspace to define ODI Topology objects In ODI Topology Manager, you set the architecture of your information system, the technologies, servers, schemas, and repositories.

Copyright 2010, Oracle. All rights reserved.

Security Navigator

Object tree

In Security Navigator, you manage objects, profiles, users, and hosts.

Workspace to define objects, profiles, users, and hosts

Copyright 2010, Oracle. All rights reserved.

Agents

Copyright 2010, Oracle. All rights reserved.

What Is an Agent?
An agent is a run-time component of ODI that orchestrates the integration process. At design time, developers generate scenarios from the business rules that they have designed. The code of these scenarios is then retrieved from the repository by the agent at run-time. This agent then connects to the data servers and orchestrates the code execution on these servers.

Copyright 2010, Oracle. All rights reserved.

ODI Agents
Agents are lightweight Java processes that orchestrate the execution of objects at run time. Agents can: Execute objects on demand, or Execute according to predefined schedules

Copyright 2010, Oracle. All rights reserved.

Two Types of Agents

JEE and Standalone
Standalone agent
Standalone Java
Any Java App

JEE agent
JDeveloper
ODI Studio

Any Web App WebLogic 11g (or other JEE)

Servlet Container MBeans Server Registry Any Application Container Web Service Container

ODI Agent
Designer Operator Topology

ODI Agent

Any MBeans App ODI MBeans for ODI Agent

Any Application

ODI Public Web Service

ODI SDK APIs

Security

ODI Console

ODI SDK APIs

ODI Data Services

RDBMS ODI Repositories

ODI Master Repository Knowledge Modules ODI Work ODI Work Repository ODI Work Repository Repository

Data Sources Connection Pool

Repository Data Sources Sources Data Sources Repository Data Sources Sources Data Sources Repository Data Sources Sources Data Sources

Target Data Sources Target Data Sources Target Data Sources

Sources and Targets

---

----------ERP PLM CRM

----------Best-of-Breed Applications

-----------

Legacy

Copyright 2010, Oracle. All rights reserved.

Using the Two Types of Agents

Deploying an agent in a JEE Application Server (Oracle WebLogic Server): 1. Define the JEE Agent in the Topology Navigator 2. Create the WLS template for the JEE Agent 3. Deploy this template directly using the WLS Configuration Wizard Using a standalone agent: 1. Launch an Agent 2. Display Scheduling Information 3. Stop the Agent

Copyright 2010, Oracle. All rights reserved.

Example of Standalone Agent

Designer Operator

Reverse engineer Java - Any Platform

Operate production Monitor sessions

Monitor sessions View Reports

Develop projects
Submit Release Jobs scenarios

Repository Read sessions

Any ISO-92 RDBMS

Standalone Agent

Write reports

Orchestrates sessions

Lightweight Distributed Architecture

Java - Any Platform

Return Code

Execute Jobs

Information System

Copyright 2010, Oracle. All rights reserved.

Console

Copyright 2010, Oracle. All rights reserved.

ODI Console
Development

Agent

ODI Console
ODI Console

Copyright 2010, Oracle. All rights reserved.

Enterprise Manager Console

Development

Agent

ODI Plug-in for EM Console

Enterprise Manager Console

Copyright 2010, Oracle. All rights reserved.

ODI Repositories

Copyright 2010, Oracle. All rights reserved.

ODI Repositories
There are two types of repositories in ODI: Master repository Work repository Development repository Execution repository Work repositories are always attached to a Master repository.

Copyright 2010, Oracle. All rights reserved.

Master and Work Repositories

Security Topology Versioning Master Repository

Models Projects Execution Work Repository (Development) Execution Execution Repository (Production)

Two type of Repositories: Master and Work Work Repositories are always attached to a Master Repository

Copyright 2010, Oracle. All rights reserved.

Example of a Repository Set-Up

Security Topology Versioning
Create and archive versions of models, projects and scenarios Import released and tested versions of scenarios for production

Master Repository

Import released versions of models, projects and scenarios for testing

Models Projects Execution Work Repository (Development) Models Projects Execution Work Repository (Test & QA)

Execution Execution Repository (Production)

Development Test Production Cycle

Copyright 2010, Oracle. All rights reserved.

Repository Set-Up: Separate Networks

Security Topology Versioning
Create and archive versions of models, projects and scenarios (from the GUI) XML export / Import (manual)

Security Topology

Models Projects Execution Work Repository (Development) Models Projects Execution Work Repository (Test & QA)
XML export / Import (manual)

Firewall

Master Repository

Master Repository
Import released versions of models, projects and scenarios for testing (From the GUI)

Execution Execution Repository (Production)

Development Test Production Cycle

Copyright 2010, Oracle. All rights reserved.

ODI Data Flow

Copyright 2010, Oracle. All rights reserved.

Oracle Data Integrator

Architecture No Server: only an orchestrator (The Agent) that will send the appropriate SQL statements to the different databases.
Servers and Applications

User Interfaces Developer

Agent Data Flow Conductor

Execution

CRM

Data Warehouse

Legacy

Code
Execution Log

Return Codes
ESB Files / XML

ER P

Developer

Metadata Rules

Repositories

Developer

Developers each have the Java GUI on their machine and share a central repository to store the metadata and the transformation processes (no local storage).
Copyright 2010, Oracle. All rights reserved.

Oracle Data Integrator

Architecture

Servers and Applications

User Interfaces Developer

Agent Data Flow Conductor Code Execution Log

Execution

CRM

Data Warehouse

Legacy

Return Codes
ESB
Files / XML

ER P

Developer

Metadata Rules

Repositories

Developer

Processes are orchestrated (and possibly scheduled) by an ODI Java agent (no graphical component to allow for the largest possible set of platform support).

Copyright 2010, Oracle. All rights reserved.

Oracle Data Integrator

Architecture

Servers and Applications

User Interfaces Developer

Agent Data Flow Conductor Code Execution Log

Execution

CRM

Data Warehouse

Legacy

Return Codes
ESB
Files / XML

ER P

Developer

Metadata Rules

Repositories

Developer

Multiple agents can co-exist in any given environment. Typical locations for the agents are on the target system and/or on the system that hosts the ODI repository.
Copyright 2010, Oracle. All rights reserved.

Overview: 6 steps to Production

1. 2. 3. Retrieve/Enrich metadata Design transformations Orchestrate data flows
Development
Development Servers and Applications

4. Generate/Deploy data flows 5. Monitor executions 6. Analyze impact / data lineage

Production
Production Servers and Applications

CRM

Data Warehouse

CRM

Data Warehouse

Legacy

Legacy

ERP

ERP ESB Files / XML

ESB

Files / XML

ODI Design-Time Environment

User Interfaces Administrators Designers Design-time Design-time Repositories Repositories Agent Data Flow Conductor Runtime Repository

ODI Runtime Environment

Agent Data Flow Conductor User Interfaces Operator ODI Console

Copyright 2010, Oracle. All rights reserved.

Copyright 2010, Oracle. All rights reserved.

Copyright 2010, Oracle. All rights reserved.