Professional Documents
Culture Documents
SC18-9158-02
IBM DB2 Information Integrator
SC18-9158-02
Before using this information and the product it supports, be sure to read the general information under “Notices” on page 25.
This document contains proprietary information of IBM. It is provided under a license agreement and copyright law
protects it. The information contained in this publication does not include any product warranties, and any
statements provided in this manual should not be interpreted as such.
You can order IBM publications online or through your local IBM representative:
v To order publications online, go to the IBM Publications Center at www.ibm.com/shop/publications/order
v To find your local IBM representative, go to the IBM Directory of Worldwide Contacts at
www.ibm.com/planetwide
When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any
way it believes appropriate without incurring any obligation to you.
© Copyright International Business Machines Corporation 2003, 2004. All rights reserved.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
© CrossAccess Corporation 1993, 2003
Contents
Chapter 1. Overview . . . . . . . . . 1 DB2 Information Integrator
Product description . . . . . . . . . . . . 1 documentation . . . . . . . . . . . 15
Software prerequisites . . . . . . . . . . . 1 Accessing DB2 Information Integrator
Configuring support for Unicode . . . . . . . 1 documentation . . . . . . . . . . . . . 15
Documentation about replication function on z/OS 17
Chapter 2. Product architecture . . . . 3 Documentation about event publishing function for
Product architecture . . . . . . . . . . . . 3 DB2 Universal Database on z/OS . . . . . . . 18
Change-capture agents . . . . . . . . . . . 3 Documentation about event publishing function for
Introduction to recovery mode . . . . . . . 4 IMS and VSAM on z/OS . . . . . . . . . . 18
Correlation service . . . . . . . . . . . . 5 Documentation about event publishing and
Example . . . . . . . . . . . . . . . 5 replication function on Linux, UNIX, and Windows . 19
Location . . . . . . . . . . . . . . . 5 Documentation about federated function on z/OS 20
Recovery mode . . . . . . . . . . . . 5 Documentation about federated function on Linux,
Metrics reports . . . . . . . . . . . . 6 UNIX, and Windows . . . . . . . . . . . 20
Starting a correlation service . . . . . . . . 6 Documentation about enterprise search function on
Named servers . . . . . . . . . . . . 7 Linux, UNIX, and Windows . . . . . . . . . 22
Publication service . . . . . . . . . . . . 7 Release notes and installation requirements . . . . 22
Messaging . . . . . . . . . . . . . . . 8
Flow of data . . . . . . . . . . . . . . 9 Notices . . . . . . . . . . . . . . 25
Correlation service SQL descriptor areas Trademarks . . . . . . . . . . . . . . 27
(SQLDAs) . . . . . . . . . . . . . . 10
In-doubt transactions . . . . . . . . . . 10 Index . . . . . . . . . . . . . . . 29
Chapter 3. Requirements for Contacting IBM . . . . . . . . . . . 31
WebSphere MQ . . . . . . . . . . . 13 Product information . . . . . . . . . . . 31
Objects required when receiving applications use Comments on the documentation . . . . . . . 31
WebSphere MQ servers . . . . . . . . . . 13
WebSphere MQ objects required by Classic Event
Publisher when receiving applications use
WebSphere MQ clients . . . . . . . . . . . 14
Product description
DB2® Information Integrator Classic Event Publisher is a tool that you can use to
capture near-real-time changes in CA-IDMS, IMS™ and VSAM databases. Classic
Event Publisher uses what is commonly referred to as “push” technology to
perform actions in response to changes in your database. When Classic Event
Publisher captures changes made to a monitored database, Classic Event Publisher
publishes those changes as XML messages on WebSphere® MQ message queues.
Software prerequisites
DB2 Information Integrator Classic Event Publisher works on z/OS®, Version 1.4 or
later and with the following software:
| v CA-IDMS r14 and r15
v IMS, Version 7.1 or later
v VSAM managed by CICS® Version 1.3 or later
v WebSphere MQ for z/OS, Version 5.3
The application that receives messages from Classic Event Publisher must use one
of the following applications:
v WebSphere MQ, Version 5.3
v WebSphere Business Integration Event Broker, Version 5.0
The conversion image environment is defined by the image generator. The image
generator utility runs as a batch job on z/OS.
For information about service info entries for publication services and about the
SERVER CODEPAGE parameter, see DB2 Information Integrator Operations Guide for
Classic Event Publishing.
Product architecture
DB2 Information Integrator Classic Event Publisher is a change capture product
with three components:
Change-capture agents
The change-capture agent is a lean application module that screens
changes to the database, and then sends to the correlation service
information about changes being made to monitored tables. There are 2
change-capture agents per database environment: one for active change
capture and one for recovery change capture. This module can be
configured to run either in the database, where the recovery agent will be
run as a batch job, or as a service that runs within the correlation service.
The configuration is determined by the databases that you are monitoring.
Correlation service
The correlation service receives raw data for database changes from the
change-capture agents, matches the changes against user-mapped tables in
the DB2 Information Integrator Classic Event Publisher metadata catalog,
and then sends the data to the publication service after the changes are
committed. This processing is handled in the correlation service so that the
change-capture agents can be minimally intrusive to the database
environment.
Publication service
The publication service receives information from the correlation service
that indicates that changes were made to one or more tables that are
mapped in the DB2 Information Integrator Classic Event Publisher
metadata catalog. The publication service determines which changes
correspond to your publications, generates XML messages that contain
information about the changes, and puts the messages on the WebSphere
MQ message queues used by your publications.
Change-capture agents
A change-capture agent is a module that runs on the same machine as your
database. A change-capture agent monitors the database, looking for actions that
affect data. When a table is affected by an INSERT, DELETE, or UPDATE
operation, the change-capture agent checks the DB2 Information Integrator Classic
Event Publisher metadata catalog to determine if the data affected is in monitored
tables, and if so sends the raw data for the candidate change to the correlation
service module. The information that is contained in that data is explained in the
section “Flow of data” on page 9.
On z/OS, the change-capture agents and the correlation service must be on the
same LPAR.
These failures might not be recoverable if they are driven by change data that will
produce the same error if recovery is attempted. For example, if the data captured
and forwarded to the publication service caused the publication service to reject
the message, then the publication service will reject that message every time that
the message is resent.
The correlation service is responsible for detecting failure and returning messages
stating that the system entered recovery mode. At this point, the recovery
change-capture agent should start or be started. Depending upon the
configuration, recovery might start automatically, or might require you to run a job
or otherwise start the recovery agent manually.
When the recovery of data catches up with the active server, with some databases
you need to restart the database to move back into active mode with no changes
lost. However, often, it is not practical to stop the monitored database to complete
the recovery process. For these situations, Classic Event Publisher provides
methods to exit recovery mode and resume the active capture of changes from the
database system. Check the chapter of the DB2 Information Integrator Operations
Guide for Classic Event Publishing for your database for the steps that you must
take.
Correlation service
The correlation service verifies the captured change data against the metadata that
is stored in the DB2 Information Integrator Classic Event Publisher metadata
catalog. If the table that was affected is mapped in the metadata catalog and is
flagged for data capture, the correlation service stores the data until it receives a
COMMIT statement.
You add information to the metadata catalog using the Data Mapper and the
metadata utility. For information about the Data Mapper, see the DB2 Information
Integrator Data Mapper Guide for Classic Federation and Classic Event Publishing.
Each piece of change information that the correlation service stores is kept until a
COMMIT or ROLLBACK action is performed on the unit of work. At COMMIT,
the correlation service creates one or two SQLDAs for each change to a monitored
table. These SQLDAs are sent to the publication service. The information contained
in the SQLDAs that are passed to the publication service is described in “Flow of
data” on page 9.
If a ROLLBACK is detected for a unit of work, all data currently stored for that
unit of work is deleted.
Example
If the change-capture agent determines that an UPDATE was performed on the
Employee database table, the change-capture agent checks the DB2 Information
Integrator Classic Event Publisher metadata catalog to determine if the Employee
database table is mapped to a logical table and flagged for change capture.
If the table is flagged for change capture, the change-capture agent passes change
information to the correlation service indicating that the table was updated. The
correlation service stores the change information until it receives information from
the change-capture agent that indicates that the UPDATE was committed. At that
point, the change-capture agent creates and sends two SQLDAs that provide the
publication service with the before and after image of the data that was updated.
Location
The correlation service resides on the same machine as the change-capture agent,
in a different address space. On z/OS, the change-capture agents and the
correlation service must be on the same LPAR.
Recovery mode
The correlation service tracks change-capture agents in recovery mode and
maintains a persistent data store of the recovery points. The failure of a correlation
service usually requires restarting a recovery process at a point other than where
the existing recovery process is running, because changes might be in transit
between the recovery process and correlation service, or changes might be waiting
to be forwarded to the publication service when the correlation service is stopped.
Restarting the recovery process ensures that no changes are lost.
The correlation service, as the focal point of change capture, tracks the point of
recovery. It determines where the recovery agent was. Recovery agents query the
restart information from the correlation service when they are started.
The correlation service continues sending data to the publication service until the
correlation service is stopped or an error occurs. When the correlation service
stops, it halts processing any cached transactions and waits for confirmation of any
outstanding messages that were sent to the publication service. See “Introduction
to recovery mode” on page 4, for more information.
Metrics reports
Classic Event Publisher maintains information about the correlation service. It
generates a report each time a publication service disconnects from the correlation
service, and each time the correlation service is shut down.
When warm-starting, you start the correlation service so that it will retain recovery
start information, so that no data loss occurs from changes that were made during
the time that the correlation service was unavailable.
Cold-starting requires that you change a setting in the configuration file that tells
the correlation service to disregard any lost changes by discarding the recovery
start point, and to let any change-capture agents continue running in active mode.
Named servers
You can use named correlation services in your deployment if you will need more
than one correlation service. For example, you might choose to have more than one
correlation service if you have both a test and production database on the same
system. If you were to use only an unnamed correlation service, and you had to
stop the correlation service for the test system, then it would put both the test
system’s change-capture agents into recovery and the production system’s
change-capture agents into recovery mode. Using unnamed servers limits you to
one correlation service per database class.
Setting up a named server environment requires that you configure both the
correlation service and its corresponding change-capture agents with the names
that you set up. To configure a correlation service, you add a ’NAME=’ parameter to
the Service Info Entry. See “Configuring a correlation service,” on page 10 in DB2
Information Integrator Operations Guide for Classic Event Publishing. For information
about configuring change-capture agents for a named server environment, see the
chapter for the corresponding data source in the DB2 Information Integrator
Operations Guide for Classic Event Publishing.
Publication service
The publication service is a multi-threaded server that supports full logging. Each
correlation service has one corresponding publication service. A publication service
performs the following tasks:
1. Receives data in SQLDA format about changes to your source tables from the
correlation service
2. Matches the data against the publications that you configured
3. Converts the SQLDA data into messages in an XML format
4. Puts the XML messages on the local, persistent WebSphere MQ message queues
specified for your publications.
Information about the last unit of recovery (UOR) to be sent as a message and
confirmation from WebSphere MQ that messages are accepted is stored in a local,
persistent message queue referred to as the in-doubt resolution queue. This queue
ensures that if the correlation service is shut down, recovery point information will
not be lost.
For information about configuring publication services, see the DB2 Information
Integrator Operations Guide for Classic Event Publishing.
Common storage
1 1
2 2
3 4 6 7
5
Metadata catalog 8
Correlation service
B-tree 9
10 11
Publication
service
WebSphere MQ 12 13
queue manager
1. The active and recovery change-capture agents both intercept database change
data in common storage, and check to see if tables affected by a change are
flagged for data capture.
2. The active and recovery change-capture agents send raw database change data
and syncpoint records to the common storage.
3. The active change-capture agent sends raw database change data and
syncpoint records to the correlation service.
4. The recovery change-capture agent sends raw database change data and
syncpoint records to the correlation service.
5. The correlation service sends restart information to recovery change-capture
agents.
Flow of data
The information generated by a captured change is passed from the change-capture
agent to the correlation service, where it is stored until the captured change is
committed. If other changes are captured on the same unit of work, they are stored
along with the first captured change, also until the unit of work is committed.
After the data is committed, the correlation service creates either one or two
SQLDAs for each change, with information about the changes that were made to
the database tables, and forwards them to the publication service.
The publication service determines if the tables specified in the SQLDAs have
corresponding publications. The publication service then converts the SQLDAs to
messages in an XML format and puts the messages on local, persistent WebSphere
MQ message queues specified by the publications.
When making database changes, you can either commit a change, or ROLLBACK
(undo) a change. DB2 Information Integrator Classic Event Publisher only acts
upon committed data. This ensures that the database makes the requested updates.
The correlation service is responsible for maintaining all the syncpoint records to
allow monitoring of COMMIT operations. This is discussed in more detail in
“Messaging” on page 8.
Message structure
The data that is sent to the publication service from the correlation service has the
following elements:
Action
This value indicates whether the action performed was an INSERT,
UPDATE, or DELETE.
Table name
This value provides the name of the table that was affected by the action.
This table name corresponds to an entry in the DB2 Information Integrator
Classic Event Publisher metadata catalog.
Timestamp
This value provides the time that the action occurred.
SQLDA1, SQLDA2
These hold the before and/or after images of the data records that were
modified.
In-doubt transactions
Transactions are in-doubt after they are sent from the correlation service to the
publication service, and before the publication service sends a confirmation or
rejection message back to the correlation service.
When a transaction is in-doubt at the time that the correlation service that sent it
terminates, the change-capture agents that connect to that correlation service are
halted, and are set to an in-doubt state.
When the correlation service is restarted and it detects that the system is in-doubt,
the correlation service puts any active change-capture agents into recovery mode.
The recovery agents then detect the in-doubt state of the system, and terminate
with an error message.
The publication service then attempts to clear the in-doubt state by sending any
pending confirmations or rejections for past transactions. If the publication service
is unable to do so, then you must either manually override the in-doubt status, or
you must cold start the correlation service.
Channel objects:
v One Message Channel Agent (MCA) at the WebSphere MQ server where Classic
Event Publisher is running (Server 1).
v One MCA at the WebSphere MQ server where the receiving application is
running (Server 2).
v One message channel from WMQ Server 1 to WMQ Server 2.
v One message channel from WMQ Server 2 to WMQ Server 1.
Channel objects:
v Server-connection channel.
An MQ listener must be listening on the server where you define the
server-connection channel. Otherwise, the channel will not be enabled.
For information about how to configure and use WebSphere MQ clients with your
receiving applications, see the WebSphere MQ Clients guide at
www.ibm.com/software/integration/mqfamily/library/manualsa/manuals/
crosslatest.html.
To access the latest DB2 Information Integrator product documentation, from the
DB2 Information Integrator Support Web site, click on the Product Information
link, as shown in Figure 2 on page 16.
You can access the latest DB2 Information Integrator documentation, in all
supported languages, from the Product Information link:
v DB2 Information Integrator product documentation in PDF files
v Fix pack product documentation, including release notes
v Instructions for downloading and installing the DB2 Information Center for
Linux, UNIX, and Windows
v Links to the DB2 Information Center online
Scroll though the list to find the product documentation for the version of DB2
Information Integrator that you are using.
You can also view and print the DB2 Information Integrator PDF books from the
DB2 PDF Documentation CD.
To view the installation requirements and release notes that are on the product CD:
v On Windows operating systems, enter:
x:\doc\%L
x is the Windows CD drive letter and %L is the locale of the documentation that
you want to use, for example, en_US.
v On UNIX operating systems, enter:
/cdrom/doc/%L/
cdrom refers to the UNIX mount point of the CD and %L is the locale of the
documentation that you want to use, for example, en_US.
IBM may have patents or pending patent applications covering subject matter
described in this document. The furnishing of this document does not give you
any license to these patents. You can send license inquiries, in writing, to:
IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
U.S.A.
For license inquiries regarding double-byte (DBCS) information, contact the IBM
Intellectual Property Department in your country/region or send inquiries, in
writing, to:
IBM World Trade Asia Corporation
Licensing
2-31 Roppongi 3-chome, Minato-ku
Tokyo 106-0032, Japan
The following paragraph does not apply to the United Kingdom or any other
country/region where such provisions are inconsistent with local law:
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS
PUBLICATION “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER
EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS
FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or
implied warranties in certain transactions; therefore, this statement may not apply
to you.
Any references in this information to non-IBM Web sites are provided for
convenience only and do not in any manner serve as an endorsement of those Web
sites. The materials at those Web sites are not part of the materials for this IBM
product, and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it
believes appropriate without incurring any obligation to you.
The licensed program described in this document and all licensed material
available for it are provided by IBM under terms of the IBM Customer Agreement,
IBM International Program License Agreement, or any equivalent agreement
between us.
All statements regarding IBM’s future direction or intent are subject to change or
withdrawal without notice, and represent goals and objectives only.
This information contains examples of data and reports used in daily business
operations. To illustrate them as completely as possible, the examples include the
names of individuals, companies, brands, and products. All of these names are
fictitious, and any similarity to the names and addresses used by an actual
business enterprise is entirely coincidental.
COPYRIGHT LICENSE:
© (your company name) (year). Portions of this code are derived from IBM Corp.
Sample Programs. © Copyright IBM Corp. _enter the year or years_. All rights
reserved.
Trademarks
The following terms are trademarks of International Business Machines
Corporation in the United States, other countries, or both:
IBM
CICS
DB2
IMS
Language Environment
WebSphere
z/OS
Java and all Java-based trademarks and logos are trademarks or registered
trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of
Microsoft Corporation in the United States, other countries, or both.
Intel, Intel Inside (logos), MMX and Pentium are trademarks of Intel Corporation
in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other
countries.
Notices 27
28 DB2 II Planning Guide for Classic Event Publishing
Index
C
change-capture agent 3
messaging failure 4
recovery mode 4, 5, 6
starting 4
correlation service 6
cold-starting 6
confirming transactions 7
errors 6
location 5
named servers 7
shutting down without stopping
database 4
SQLDAs 5, 10
starting 6
starting a database without 4
warm-starting 6
D
DB2 Information Integrator Classic Event
Publisher
uses 1
I
in-doubt transactions 6
M
messaging 8
metadata catalog 3, 5
metadata utility 5
metrics reports 6
N
named servers 7
P
point of recovery 6
publication service 7
R
recovery mode 4, 5
S
SQLDAs 5
T
THROTTLE parameter 5
To learn about available service options, call one of the following numbers:
v In the United States: 1-888-426-4343
v In Canada: 1-800-465-9600
To locate an IBM office in your country or region, see the IBM Directory of
Worldwide Contacts on the Web at www.ibm.com/planetwide.
Product information
Information about DB2 Information Integrator is available by telephone or on the
Web.
If you live in the United States, you can call one of the following numbers:
v To order products or to obtain general information: 1-800-IBM-CALL
(1-800-426-2255)
v To order publications: 1-800-879-2755
Printed in USA
SC18-9158-02