Siebel EIM Best Practice

Contents
1 Overview ------------------------------------------------------------------------------- 2
1.1 Purpose -------------------------------------------------------------------------------------2
1.2 What’s not included ---------------------------------------------------------------------------2

2 EIM Best Practices --------------------------------------------------------------- 3

2.1 Introduction ----------------------------------------------------------------------------------- 3
2.2 EIM Configuration ----------------------------------------------------------------------------- 3
2.2.1 Overview ---------------------------------------------------------------------------------- 3
2.2.2 IFB Generic Parameters ------------------------------------------------------------------ 5
2.2.2.1 Only Base Tables -------------------------------------------------------------------- 5
2.2.2.2 Ignore Base Tables ------------------------------------------------------------------ 5
2.2.2.3 Ignore Base Columns ---------------------------------------------------------------- 6
2.2.2.4 Default Columns --------------------------------------------------------------------- 6
2.2.2.5 Fixed Columns ----------------------------------------------------------------------- 6
2.2.2.6 Use Index Hints, Use Essential Index Hints ---------------------------------------- 7
2.2.2.7 Session SQL -------------------------------------------------------------------------- 8
2.2.2.8 Using Synonyms --------------------------------------------------------------------- 8
2.2.2.9 Rollback On Error, Commit Each Pass, Commit Each Table ----------------------- 9
2.2.2.10 Update After Failed Insert --------------------------------------------------------- 10

2.2.3 IFB Import Process Specific Parameters --------------------------------------------------- 11

2.2.3.1 Only Base Columns --------------------------------------------------------------------- 11
2.2.3.2 Insert Rows, Update Rows ------------------------------------------------------------- 11
2.2.3.3 Net Change ----------------------------------------------------------------------------- 12
2.2.3.4 Oracle Insert Append Mode ------------------------------------------------------------ 13
2.2.3.5 Trim Spaces ----------------------------------------------------------------------------- 14

2.2.4 EIM Delete Process Configuration ---------------------------------------------------------- 15

2.2.4.1 Delete All Rows -------------------------------------------------------------------------15
2.2.4.2 Delete Exact ---------------------------------------------------------------------------- 15
2.2.4.3 Delete Matches ------------------------------------------------------------------------- 16
2.2.4.3.1 Important Information from Siebel Support -------------------------------------- 17

2.2.5 EIM Merge Process Configuration ----------------------------------------------------------- 18

2.2.6 Other Parameters ---------------------------------------------------------------------------- 20

3 Running EIM ---------------------------------------------------------------------- 22

3.1 Data -------------------------------------------------------------------------------------------- 22
3.2 IFB File and EIM Processes ------------------------------------------------------------------- 23
3.3 Scheduling ------------------------------------------------------------------------------------- 24
3.3.1 Degree of Parallelism/Layout ------------------------------------------------------------ 24
3.3.2 Batch size --------------------------------------------------------------------------------- 24
3.3.3 EIM Streaming Configurations ----------------------------------------------------------- 24
3.3.4 Control-M Scheduling --------------------------------------------------------------------- 25
3.4 EIM Table Maintenance ----------------------------------------------------------------- 26

Page 1 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

1 Overview

1.1 Purpose

This document is intended to be used as a guide for the design and deployment of EIM
interfaces to support the delivery of Siebel based projects at BT. It will be of particular
relevance to the delivery of OneSiebel (aka Account Managed Platform, AMP) and OneView (aka
Volume Siebel Release 6).

This document is split into two main sections as follows:

1. Siebel EIM Best Practice

 This section details techniques to be adopted to ensure that EIM interfaces and their
configurations adhere to Siebel and BT Best Practices.
2. OneView Release 4 data load Lessons Learned
 This section details the key lessons learned from the OneView (aka Volume Siebel, Big
Data, CCC) bulk data load of Billing Accounts, Assets and Consumer Updates into the
OneView Siebel instance.

These applications are fed data via ODM controlled data loads. Where processes other than
ODM will drive EIM interfaces (e.g. Converge to OneSiebel migration jobs) then some ODM
related content will still be relevant and so all sections should be read.

1.2 What’s not included

 This document is intended to share Best Practice and Lessons Learned and is neither a
High Level nor a Low Level Design. This document should be used to feed into the above
mentioned designs.
 A detail of Oracle-Level tuning is not included although some techniques may be
mentioned in the context of other EIM specific parameters or configurations.
 This is not a ‘One Stop Shop’ from which to learn EIM. Instead, this document is
intended to communicate EIM Best Practice.

1.3 Document Conventions

Key points within this document have been included in call-out boxes as follows:

This indicates a convention that should be followed by Development Teams within


BT.

A warning sign indicates that specific care should be taken, that a piece of work is
required to determine the best configuration or that a convention or parameter
should not be followed.

An information symbol is used to indicate additional points of interest, not

necessarily relevant to specific implementations at BT.

These Document Conventions appear for most key point (in the EIM Best Practice section) and
so the document’s recommendations could be summarised by reading just the illustrated points.

Page 2 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2 EIM Best Practice

2.1 Introduction

Siebel’s Enterprise Integration Manager (EIM) operation is determined by a configuration file.

This file is called the IFB file and, coupled with loading data into a Siebel EIM/Interface table (a
staging table that Siebel uses so that it can build the logical data relationships as defined within
Siebel’s Repository) determines what operations EIM carries out.

As a generic, out-of-the-box interface mechanism, EIM’s performance can be streamlined by

reducing the amount of work it has to do. This is done by imposing strict IFB guidelines.

Not all parameters are discussed in detail.

A single Interface Table maps to multiple Base Tables (to closely follow the nature of Siebel
Business Objects and Business Components).

All tables and related foreign keys are processed by default unless parameters are specified in
the ifb file.

This document describes how to alter EIM’s default operation by specifying parameters in the ifb
file. Some of these changes make functional impacts (i.e. change the way in which how EIM
processes the data) whereas others can improve performance (normally by limiting the amount
of work EIM and the underlying database has to do).

2.2 EIM Configuration

2.2.1 Overview

EIM supports the following 4 modes of operation:

o Import
o Delete
o Merge
o Export

The EIM Import job is actually an Upsert operation and so this document will refer to both
Import (Insert) and Update EIM operations.

There are a number of generic ifb parameters that are valid for all modes of operation as well as
those that are operation specific.

This document will explore Import, Delete and Merge jobs in more detail. EIM Exports will not
be covered, as they are not (generally) used within BT – instead, PL/SQL (often controlled by
Ab Initio graphs) is used to query the Siebel data tables directly in order to export information.

Page 3 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

So that some of the terminology within this document is clear, here’s an example IFB file (…
indicates lines removed for clarity):

[Siebel Interface Manager]

LOG TRANSACTIONS TO FILE = FALSE header
USING SYNONYMS = FALSE section

[Update_Volume_Bill_Accnt]
TYPE = IMPORT
BATCH = $BatchRange
TABLE = EIM_ACCOUNT
ONLY BASE TABLES = S_PARTY, S_ORG_EXT
INSERT ROWS = S_PARTY, FALSE
UPDATE ROWS = S_PARTY, FALSE Process
INSERT ROWS = S_ORG_EXT, FALSE section
NET CHANGE = FALSE
ONLY BASE COLUMNS = S_PARTY.PARTY_UID, \
S_PARTY.PARTY_TYPE_CD, \
S_ORG_EXT.NAME, \
S_ORG_EXT.LOC, \
S_ORG_EXT.BU_ID, \
…

Note: a backslash (‘\’) should be used to escape new lines for readability.

Many parameters can be specified in the Header section, the Process section or both. The rules
of inheritance are as follows:

 If a parameter is set in the Header section, it is included in all Process sections.

 If a parameter is used in a Shell process, it is included in all Shell Sub processes
(detailed later).
 If a parameter is in the Header section and also in the Process section, then the value in
the Process section over-rides that of the Header section. The same is true for Shell
process and Shell Sub process.
 If a parameter exists only in a Process section and not in the Header section, then it is
only used in that Process.

Page 4 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.2 IFB Generic Parameters

The parameters here apply to all EIM job types, with process specific (i.e. Import, Delete or
Merge) parameters and techniques being specified in subsequent sections.
2.2.2.1 Only Base Tables
This is a Process section parameter.

Usage:

ONLY BASE TABLES = BASE_TABLE1, BASE_TABLE2

…where BASE_TABLE1 & BASE_TABLE2 are tables mapped to by the EIM table in question.

By only including the Base Table relationships that you need to resolve, EIM’s work is reduced
and so Performance can be improved.

Example: In OneView, a Consumer Update job incorrectly carried out updates to S_PARTY,
S_ORG_EXT, S_ACCNT_POSTN and S_ORG_BU despite the fact that the data in
S_ACCNT_POSTN and S_ORG_BU was being defaulted by the Consumer Adapter (Ab Initio)
each time. By removing these tables from the ONLY BASE TABLES parameter, the performance
of the Volume Siebel Release 4 Consumer Update job was greatly improved. So that
performance gains such as these can be made, detailed functional analysis of the EIM interface
must be carried out in order to fully understand the data source.

 This parameter should be used – ONLY BASE TABLES

2.2.2.2 Ignore Base Tables

This is a Process section parameter.

Usage:

INGORE BASE TABLES = BASE_TABLE1, BASE_TABLE2

…where BASE_TABLE1 & BASE_TABLE2 are tables mapped to by the EIM table in question.

As an alternative to ONLY BASE TABLES, this parameter can be used to detail specific tables
that should be ignored by the EIM process.

For clarity, the only instance in which this parameter would be used instead of ONLY BASE
TABLES is where the majority of the tables must be included. By using this parameter, it will be
clear which tables will not be used.

 This parameter will generally not be used – IGNORE BASE TABLES

Page 5 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.2.3 Ignore Base Columns

This is a Process section parameter.

Usage:

IGNORE BASE COLUMNS = BASE_TABLE1.BASE_COLUMN1,\

BASE_TABLE1.BASE_COLUMN2,\
BASE_TABLE2.BASE_COLUMN1

…where BASE_TABLE1 and BASE_TABLE2 are mapped to by the EIM table in question and
BASE_COLUMN1 & BASE_COLUMN2 are columns on each Base Table.

This parameter can be used to instruct EIM to ignore the processing of columns specified. This
is most useful for ignoring unused foreign key. This would result in EIM not attempting to
resolve the foreign key in question.

This parameter would generally not be used on OneSiebel with the use of ONLY BASE COLUMNS
being preferred.

 This parameter will generally not be used – IGNORE BASE COLUMNS

Not supported for Merge processes.

2.2.2.4 Default Columns

There are a number of EIM columns where it may be desirable to set a default value for a
column, if no value is specified.

There is an option to specify these via the ifb file (DEFAULT COLUMN = COLUMN_NAME, ‘value’)
however this gives EIM an additional SQL statement to execute.

In tuning EIM, the objective is to reduce the work that EIM has to do, therefore fixed & default
columns should be set by the external interface architecture i.e. Ab Initio.

 This parameter should not be used – DEFAULT COLUMN

2.2.2.5 Fixed Columns

There are a number of EIM columns where it may be desirable to set a fixed value for a column.

There is an option to specify these via the ifb file (FIXED COLUMN = COLUMN_NAME, ‘value’)
however this gives EIM an additional SQL statement to execute.

In tuning EIM, the objective is to reduce the work that EIM has to do, therefore fixed & default
columns should be set by the external interface architecture i.e. Ab Initio.

 This parameter should not be used – DEFAULT COLUMN

Page 6 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.2.6 Use Index Hints, Use Essential Index Hints

These are Process section ifb file parameter.

Usage:

USE INDEX HINTS = TRUE|FALSE

USE ESSENTIAL INDEX HINTS = TRUE|FALSE

The default values of these parameters are indicated in bold, meaning that EIM adds index
hints to certain SQL statements. There are two levels of granularity available. By specifying USE
INDEX HINTS = TRUE, then additional index hints are used from the default. If no hints are
desired, then set both parameters to FALSE.

EIM does not always suggest the most appropriate execution path and so to determine whether
the (hard coded) hints improve performance or not, EIM jobs should be performance tested
with and without this set to True in order to gauge performance characteristics. Within Volume
most interfaces (but not all) have been found to perform better with no index hints in place
when the Oracle Cost Based Optimizer (CBO) is used.

The actual index hints themselves cannot be specified – just an ability to turn on|off index hints
is provided.

In Volume Release 4 this parameter was used in conjunction with the SESSION SQL = "Alter
session set optimizer_mode = all_rows" although as OneSiebel have chosen the Siebel 7.7
version, then the database will already be running in Cost Based mode. It is important that
OneSiebel performance testing explores this area.

As Siebel 7.7’s now supports (in fact, mandates) the Oracle Cost Based Optimizer, then
the USE INDEX HINTS, USE ESSENTIAL INDEX HINTS must be thoroughly examined by
Performance Testing.

On Volume it has been observed that Siebel often adds inappropriate index hints such
as hinting the use of S_ORG_EXT_U2 to resolve foreign keys to S_ORG_EXT. As this
index is led by BU_ID which is non-selective in the Volume application (all Accounts
belong to the ‘BT Volume’ organisation), this is an inappropriate index to use. By
turning hints off, the more appropriate S_ORG_EXT_U1 index (or better still a custom
index) may be chosen by Oracle.

Page 7 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.2.7 Session SQL

This parameter has already been introduced in the section on USE INDEX HINTS, USE
ESSENTIAL INDEX HINTS.

Usage:

SESSION SQL = “sql statement to be executed”

Note that no terminating semi-colon (;) should be used eg.

SESSION SQL = "Alter session set optimizer_mode = all_rows"

This parameter can be used to execute custom SQL at the beginning of EIM’s session (EIM is a
single threaded, single database connection component). Only a single statement can be issued.

Session SQL should NOT be used to modify data in the Siebel schema directly as Siebel does
not support direct SQL (except for querying).

Session SQL should NOT be used to modify data in the Siebel schema directly as Siebel
does not support direct SQL (except for querying).

As Siebel 7.7’s now supports (in fact, mandates) the Oracle Cost Based Optimizer, then
setting of optimiser mode via SESSION SQL may no longer be appropriate. The
Performance Team will investigate this area.

2.2.2.8 Using Synonyms

This is an EIM Process section parameter.

Usage:

USING SYNONYMS = TRUE|FALSE

EIM’s default operation is to look up Account Synonyms in order to resolve foreign keys to
S_ORG_EXT. This requires that EIM does an additional lookup and so setting this parameter to
False can reduce overall run time. The default is True.

Although OneSiebel may have a requirement to support Synonyms (Trading As names), ODM
will supply Siebel User Keys of the Account as defined in S_ORG_EXT and so setting this
parameter to FALSE will not impact this functionality.

 This parameter should be used – USING SYNONYMS = FALSE

Page 8 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.2.9 Rollback On Error, Commit Each Pass, Commit Each Table

These parameters determine what happens when a low level EIM error occurs (such as an
Oracle error).

By default, EIM commits each table and each pass, meaning that if an error occurs, the result
may be partially imported data.

Example – The Volume Consumer entity consists of an entry in S_PARTY, S_ORG_EXT,

S_ACCNT_POSTN, S_ORG_BU. If a Low Level occurs during EIM Import, an entry may be
inserted into S_PARTY, S_ORG_EXT but not into other tables, making this record logically
invalid within OneView.

In order to properly rollback transactions in the event of a Low Level error, then this parameter
should be used in conjunction with COMMIT EACH PASS = FALSE, COMMIT EACH TABLE =
FALSE.

Usage:

ROLLBACK ON ERROR = TRUE|FALSE

COMMIT EACH PASS = TRUE|FALSE
COMMIT EACH TABLE = TRUE|FALSE

The defaults are indicated in bold.

As per Siebel Alert 290, these settings should be set to perform a single commit (ROLLBACK ON
ERROR = TRUE, COMMIT EACH PASS = FALSE, COMMIT EACH TABLE = FALSE) for all Merge
and Delete Operations.

Whilst it may immediately appear beneficial to use these settings for Insert and Update
operations, there is a high risk that this would impact competing processes (like Siebel
Analytics) by causing an Oracle ‘Snapshot too old’ error. This combination of parameters should,
therefore, only be used for EIM Merge and Delete processes.

As it does not make sense to use ROLLBACK ON ERROR without using both COMMIT EACH PASS
= FALSE and COMMIT EACH TABLE = FALSE, then this parameter should not be included in the
header section of the ifb file (as is the case currently on Volume).

Page 9 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.2.10 Update After Failed Insert

If single commit is not implemented (and it should generally not be used for EIM Import
processes) in OneView, ODM’s TrueDiff process should check all tables in a multi-table entity to
ensure that the Update after failed Insert operation performs properly.

EIM Import jobs intended to Insert-only into the Base Tables are not generally configured with
UPDATE ROWS = FALSE, which may otherwise be expected (to avoid a Data Quality Issue
where an anomaly has occurred). This is so that if the initial insert fails, then sending the record
again will result in an update to a partial record. In Volume at present, this is not implemented
properly because the TrueDiff process (the ODM process that identifies which records to insert,
update or delete from Siebel based on comparing the data within Siebel with that within ODM’s
datastores) does not compare columns in all tables for every entity.

As per Siebel Alert 290, these settings should be set to perform a single commit
 (ROLLBACK ON ERROR = TRUE, COMMIT EACH PASS = FALSE, COMMIT EACH TABLE =
FALSE) for all Merge and Delete Operations.

If single commit is not used (and it should not be), ODM’s TrueDiff process should be
capable of including all columns in all Base tables involved in each EIM process.

Page 10 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.3 IFB Import Process Specific Parameters

This section details those parameters that are specific to EIM jobs of type ‘Import’ (both Insert
and Update operations).

As well as the parameters detailed within this section, those parameters in the Generic section
can also be applied.
2.2.3.1 Only Base Columns
This is a Process section parameter.

Usage:

IGNORE BASE COLUMNS = BASE_TABLE1.BASE_COLUMN1,\

BASE_TABLE1.BASE_COLUMN2,\
BASE_TABLE2.BASE_COLUMN1

…where BASE_TABLE1 and BASE_TABLE2 are mapped to by the EIM table in question and
BASE_COUMN1 & BASE_COLUMN2 are columns on each Base Table.

Whereas the generic parameter ‘Ignore Base Columns’ instructs EIM to ignore specific columns,
‘Only Base Columns’ instructs EIM to only process those columns specifically listed.

It does not make sense to use both ‘Ignore Base Columns’ and ‘Only Base Columns’ in the same
job. Either specify one or the other.

ONLY BASE COLUMNS should be used for every interface to achieve best performance

You must include Foreign Key columns (Insert and Update), required fields (Insert operation)
and any data columns to be loaded.

 ONLY BASE COLUMNS should be used for every interface to achieve best performance.

2.2.3.2 Insert Rows, Update Rows

These two Process section parameters impact upon both Data Quality and Performance.

Usage:

INSERT ROWS = BASE_TABLEN, TRUE|FALSE

UPDATE ROWS = BASE_TABLEN, TRUE|FALSE

As has already been stated, an EIM table maps to multiple base tables. The use of the above
parameters controls what type of operation is allowed on each specified table. The default for
both parameters is True.

As an EIM Import job is actually an Upsert operation. i.e. if the User Keys populated in the EIM
table matches those of an existing row in the Base Tables, then the operation is an Update. If
the User Keys populated in the EIM table do not match those of an existing row in the Base
Tables, then the operation is an Insert operation.

BT has invested heavily in the ODM layer’s capability of creating separate Insert and Update
transactions, based around the unload functionality which is necessary due to multi-mastering.
As such, ODM determines whether the operation should be an insert or an update to Siebel’s

Page 11 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

data. Therefore, the author believes that every EIM job should specifically use INSERT ROWS
and UPDATE ROWS to specify the operations that EIM is allowed to perform. This will prevent
any rows from being updated (where User Keys match in error) or inserted (when User Keys are
not located) unintentionally.

Performance can be improved by preventing redundant updates.

INSERT ROWS and UPDATE ROWS should be used to tighten Data Quality and improve

performance.

If single commit is not used, ODM’s TrueDiff process should be capable of checking all
tables involved in an EIM interface. This would mean, optimally, that an Insert interface
could specify UPDATE ROWS = FALSE on base tables so that no erroneous update
would be allowed. If TrueDiff is not fully aware of each Siebel entity, then Insert jobs
would need to be set to allow updates although this wouldn’t always fix records as
intended (self-healing capability).

Although INSERT ROWS = TRUE is the default, it is a good idea to state this specifically
so that the interface’s intention is not ambiguous.

Full use of INSERT ROWS and UPDATE ROWS will mean that any occurrences of
PARTIALLY_IMPORTED should be treated as rightful errors.

2.2.3.3 Net Change

This parameter is used to specify whether columns can be updated to NULL.

Usage:

NET CHANGE = TRUE|FALSE

The default value is shown in bold.

When set to FALSE, Base Table columns will be updated to NULL if the corresponding EIM table
column is NULL and ONLY BASE COLUMNS is used. This is the only way to update a column to
NULL.

EIM’s default behaviour is to skip processing of NULL columns in the EIM table (except where
ONLY BASE COLUMNS is used).

Because of the way in which ODM adapters are designed and owing to the fact that
 ONLY BASE COLUMNS is used throughput, NET CHANGE = FALSE is used for all Update
processes.

Whilst the underlying SQL generated by NET CHANGE = FALSE implements this by use
of a large NVL based statement, this has not been seen to cause a significant
processing overhead.

Page 12 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.3.4 Oracle Insert Append Mode

This is a new feature in Siebel 7.5x.

Usage:

ORACLE INSERT APPEND MODE = TRUE|FALSE

The default is shown in bold.

Using this means that when inserting data into the Base tables, Oracle’s Insert Append Mode is
used. This inserts data into new blocks above the High Water Mark.

Whilst this may improve general insert performance, use of this parameter will lead to increased
fragmentation and increased storage requirements. The Volume Production DBA team do not
support use of this parameter.

Do not use this parameter as it can lead to increased Base Table fragmentation as well

as increased storage requirements.

Page 13 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.3.5 Trim Spaces

This is an EIM Process section parameter.

Usage:

TRIM SPACES = TRUE|FALSE

EIM’s default operation is to right trim white space around data fields:

UPDATE SIEBEL.EIM_ACCOUNT
SET IF_ROW_STAT_NUM = 0,
IF_ROW_STAT = 'IN_PROGRESS',
T_ORG_EXT__STA = 0,
T_ORG_EXT__RID = NULL,
T_ORG_EXT__EXS = 'N',
T_ORG_EXT__UNQ = 'N',
T_ORG_EXT_BU_ID = NULL,
T_ORG_EXT_CURRPR = NULL,
...
...
END_DT = DECODE(END_DT,NULL,TO_DATE(NULL),
TO_DATE(TO_CHAR(END_DT,'YYYY-MM-DD')||' 00:00:00','YYYY-MM-DD
HH24:MI:SS')),
START_DT = DECODE(START_DT,NULL,TO_DATE(NULL),
TO_DATE(TO_CHAR(START_DT,'YYYY-MM-DD')||' 00:00:00','YYYY-MM-DD
HH24:MI:SS')),
NAME = RTRIM(NAME),
LOC = DECODE(LOC,NULL,NULL,RTRIM(LOC)),
ALIAS_NAME = DECODE(ALIAS_NAME,NULL,NULL,RTRIM(ALIAS_NAME)),
ASGN_PH_AC = DECODE(ASGN_PH_AC,NULL,NULL,RTRIM(ASGN_PH_AC)),
ASGN_PH_CC = DECODE(ASGN_PH_CC,NULL,NULL,RTRIM(ASGN_PH_CC)),
BASE_CURCY_CD = DECODE(BASE_CURCY_CD,NULL,NULL,RTRIM(BASE_CURCY_CD)),
...
...
URL = DECODE(URL,NULL,NULL,RTRIM(URL)),
VAT_REGN_NUM = DECODE(VAT_REGN_NUM,NULL,NULL,RTRIM(VAT_REGN_NUM)),
ROW_STATUS = DECODE(ROW_STATUS,NULL,NULL,RTRIM(ROW_STATUS)),
COMMENTS = DECODE(COMMENTS,NULL,NULL,RTRIM(COMMENTS)),
STATUS = DECODE(STATUS,NULL,NULL,RTRIM(STATUS))
WHERE (IF_ROW_BATCH_NUM = 1)

Not all columns are trimmed but this cannot be controlled.

By setting this parameter to False, then this operation is not performed during the ‘Initialize
Interface Tables’ EIM pass. Whilst this theoretically slims down EIM processing, the gains to by
had by setting TRIM SPACES = FALSE will be negligible when compared to the impact of other
EIM parameters.

Additionally, both the eVolve and Volume development teams have expressed that they prefer
to retain the default of TRUE.

TRIM SPACES should be left at the default value of TRUE. It adds no value to specify

this explicitly in the ifb.

Page 14 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.4 EIM Delete Process Configuration

This section details parameters and techniques as well as important Technical Information from
Siebel Support, which is specific to EIM jobs of type ‘Delete’ and ‘Merge’.

This section is structured differently than other like sections within this document as this walks
the reader through each type of Delete job, namely:

o Delete All Rows

o Delete Exact
o Delete Matches
2.2.4.1 Delete All Rows
This Process section parameter is used to specify that all rows mapped to by the interface table
should be deleted. This parameter should therefore be used with extreme caution!

Usage:

DELETE ALL ROWS = TRUE|FALSE

The default value is False.

No other parameters are really be valid as all data in the specified Base Table will be deleted.

Once run, the User Keys of records deleted by the process will be loaded into the EIM tables
and can be archived or deleted.

This parameter should be used with extreme caution! It is not particularly relevant to
Production Processes.

2.2.4.2 Delete Exact

This Process section parameter is used to specify that EIM should delete rows in the Base Tables
whose User Keys match those of records loaded into the Interface Tables.

Usage:

DELETE EXACT = TRUE|FALSE

As EIM only needs the User Keys, then only the User Keys should be loaded.

This operation is primarily concerned with deleting rows from the primary Base Tables mapped
to by the given EIM table. Deletion of Child Rows can be controlled by use of ONLY BASE
TABLES.

DELETE ROWS can be used to specify from which tables deletes are allowed. The default value
is True.

Usage:

DELETE ROWS = BASE_TABLEN, TRUE|FALSE

…where BASE_TABLEN is a Base Table mapped to by the specific EIM table.

UPDATE ROWS is also valid for this job.

Page 15 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

Setting DELETE ROWS and/or UPDATE ROWS to FALSE should be done with extreme caution as
it may result in dangling foreign keys i.e. foreign keys that no longer point to a valid record.

Once run, no additional data is populated into the interface tables and so the original rows that
were loaded into the EIM table may be archived or deleted.

Setting DELETE ROWS and/or UPDATE ROWS to FALSE should be done with extreme
caution as it may result in dangling foreign keys.

DELETE EXACT is the safest method of deleting data in bulk from Siebel and so is the

adopted delete method at BT.

2.2.4.3 Delete Matches

This Process section parameter is used to specify that EIM should delete data from the Base
Tables that matches the where clause specified in the ifb.

e.g.

DELETE MATCHES = EIM_ACCOUNT, (CUST_STAT_CD = ‘Inactive’)

In the example above, all rows whose CUST_STAT_CD = ‘Inactive’ would be deleted.

Once run, this EIM job will have written the User Keys of the records deleted into the EIM
tables. This data can then be either archived or deleted.

There is no concept of Batchsize in a Delete Matches job – all rows that meet the criteria are
processed in a single run.

There is no concept of batchsize in a Delete Matches job. Therefore, several jobs may
be required to delete the intended records, perhaps grouping them by a data range of
range of Unique Identifiers.

Page 16 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.4.3.1Important Information from Siebel Support

There are two important Siebel Alerts need to be kept in mind for all Delete and Merge jobs.
These Alerts should be implemented (subject to confirmation/approval from Siebel) in all Delete
and Merge jobs.

o There is a Siebel Alert (Alert 306: Specify $NUM_IFTABLE_LOAD_CUTOFF = 0 for

EIM merge and delete) that states that you must specify a parameter
($NUM_IFTABLE_LOAD_CUTOFF=0) otherwise EIM will be blind to some metadata
relating to foreign keys. This could clearly cause DQ issues as it may leave dangling
foreign keys. Whilst this parameter should be able to be specified in the IFB Header
section as detailed within the Alert, this was found to repeatedly cause a core dump on
the Solaris Application Server. A Service Request was logged on Siebel SupportWeb and
the workaround of specifying this via Extended Parameters on the command line was
found.
 
o Orphaned records may be created in the current configuration in the event that an
Oracle error (or other similarly severe error) occurs whilst the EIM job is running. This is
as a result of another Siebel Alert (Alert 209: EIM Merge and Delete processes
should use only one commit per batch to avoid data integrity issues) and, again,
may cause DQ issues. To prevent orphans from being created the following settings
should be specified in the Process section:
 
COMMIT EACH TABLE = FALSE
COMMIT EACH PASS = FALSE
ROLLBACK ON ERROR = TRUE
 
(If a severe error does occurs, the entire batch is rolled back but records are left in the
EIM table with an IF_ROW_STAT of ‘IN_PROGRESS’ because Siebel does a commit after
preparing the interface tables (see Trim Spaces section within this document for an
example EIM log file showing this). Therefore, if records are left with an IF_ROW_STAT =
‘IN_PROGRESS’ after all EIM jobs have completed, then it is clear that a low-level error
has occurred.)

Use the following parameters for all Delete and Merge jobs:

 COMMIT EACH TABLE = FALSE

COMMIT EACH PASS = FALSE
ROLLBACK ON ERROR = TRUE

$NUM_IFTABLE_LOAD_CUTOFF = 0 should be specified for all Delete and Merge jobs.

ODM’s GSL (or any other calling process) must be able to specify this parameter and, in
fact, it would be wise that the GSL could build a flexible Server Manager command line

mode call so that Extended Parameters and other specialised attributes can be
included. This would require that the current ODM EIM_DETAIL table and surrounding
functionality be enhanced.

Including $NUM_IFTABLE_LOAD_CUTOFF = 0 within the IFB file’s Header section may

cause core dumps on the Solaris Application Server. To workaround this, specify the
parameter via Extended Parameters on the command line.

Page 17 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.5 EIM Merge Process Configuration

In a Merge job, victim and survivor records are specified in the EIM tables. The
IF_ROW_MERGE_ID on the victim record is populated with the row_id (in the Interface Table) of
the survivor record e.g.

IF_ROW_BATCH_NUM ROW_ID NAME IF_ROW_MERGE_ID

100 1 OneSiebel
100 2 OneView
100 3 Volume 2
100 4 Account Managed Platform 1
100 5 Big Data 2

In the example above:

 OneSiebel and OneView are survivor records (no IF_ROW_MERGE_ID specified).

 Volume and Big Data are merged into OneView.
 Account Managed Platform is merged into OneSiebel.

A Merge job does not have any specific parameters, however there are a number of
recommendations and observations:

o A Merge job only needs the User Keys and IF_ROW_MERGE_ID on the victim record to
be specified.
o Siebel Alerts 306 and 209 (as detailed in the section entitled Important Information from
Siebel Support) should be implemented.
o UPDATE ROWS is the only other relevant setting. This controls whether foreign keys in a
child table are updated to point towards the survivor or not. Setting this parameter to
FALSE can cause dangling foreign keys and is not recommended.
o The batchsize used should be considerably smaller than the batchsize usually adopted
e.g. on Volume the standard EIM batchsize is 10,000 but a batchsize of just 2000 is used
for Address Merge operations.
o Whilst EIM Merge can theoretically be run in parallel if records are victim records are
mutually exclusive across all batches, this is not recommended – Oracle locks occurred
during Performance Testing on a number of occasions and as database resource is
considerable even for a single EIM stream, then running EIM merge in parallel is likely to
saturate IO resources very quickly and result in an own goal.
o As in the example above if more than 1 victim is to be merged into survivor, this should
be done in the same batch (if mutual exclusivity can be guaranteed). Merge operations
should NOT be split into batches of 2 containing just a survivor and a victim (as was the
case on Volume at one point).

Page 18 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

o Volume is currently experiencing difficulties with EIM Merge jobs due to Performance and
Data Quality Issues.
o Performance
 Due to the large number of relationships within the Siebel database, not
all relevant foreign key columns on the database have been indexed, out
of the box. This invokes Full Table Scans which are not viable against
OneView’s largest tables.
 Siebel Expert services are currently engaged to resolve the issues and a
resolution is forthcoming.
o Data Quality
 A Siebel documented issue whereby duplicate columns are left behind in
intersection tables was recently experienced by the Volume team. Whilst
this issue is reportedly resolved in 7.7, it merits that due care be taken.

Use the following parameters for all Delete and Merge jobs:

 COMMIT EACH TABLE = FALSE

COMMIT EACH PASS = FALSE
ROLLBACK ON ERROR = TRUE

$NUM_IFTABLE_LOAD_CUTOFF = 0 should be specified for all Delete and Merge jobs.

ODM’s GSL must be able to specify this parameter and, in fact, it would be wise that
the GSL could build a flexible Server Manager command line mode call so that

Extended Parameters and other specialised attributes can be included. This would
require that the current Volume EIM_DETAIL table and surrounding functionality be
enhanced.

The Volume team encountered Data quality issues due to an out-of-the-box Siebel
issue. Whilst this has been reportedly fixed in 7.7, due care needs to be taken.

Page 19 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

2.2.6 Other Parameters

The following table includes details on other EIM parameters for completeness. Parameters
requiring attention include description text in bold.

N.B. This guide was prepared with reference to both Siebel 7.5.2 and Siebel 7.7.

Parameter Description
Header Section Only Parameters
Other parameters can generally be specified in either the header or the process section.
CONNECT ODBC data source – not required as this is set using data
retrieved from the Gateway Server
LOG TRANSACTIONS TO A Siebel Remote related parameter – this should be set to
FILE FALSE in the header section of all IFB files whilst Siebel
Remote is not being used. All Siebel Remote related
parameters will need to be reviewed if Siebel Remote is to be
deployed.
USERNAME This parameter should not be set. Rather this information is
retrieved from the Gateway.
PASSWORD This parameter should not be set. Rather this information is
retrieved from the Gateway.
TABLEOWNER This parameter should not be set. Rather this information is
retrieved from the Gateway.
PROCESS This defines the process to invoke when EIM is started with
the IFB file. This should not be used as it is preferred that
the process name be specified externally i.e. specified on the
command line when invoking EIM.
Generic Process Parameters
BATCH The batch or batch range to be processed.
INCLUDE Used in Shell Processes to automate the running of a number
of processes automatically. Shell Processes should generally
not be used, as scheduling of jobs is generally mastered out-
with the EIM environment and specific EIM processes to
invoke are specified on the Server Manager command line.
Default.ifb, which can be found within the Application
Server’s admin directory, contains examples of calling Shell
Processes.
LOG TRANSACTIONS A Siebel Remote related parameter. This parameter is only
relevant if Docking:Transaction Logging System Preference is
set to True. Not relevant to either OneSiebel r1.0 or OneView
r6.0.
SKIP BU_ID DEFAULT The default value of False should be retained to specify that
EIM uses the default value defined within the Repository
when setting virtual null keys.
TABLE This parameter specifies which EIM table is to be processed
by the EIM job.
TRANSACTION SQL The optional parameter can be used (similarly to SESSION
SQL) to run ad-hoc SQL after each EIM SQL statement is
committed. To date no specific use has been found for this
parameter.
TYPE Specifies whether the EIM process is of type IMPORT
(includes BT Update and Import processes), EXPORT,
DELETE, MERGE or SHELL. Shell processes should not
generally be used. Default.ifb, which can be found within the
Application Server’s admin directory, contains examples of
Page 20 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

Parameter Description
calling Shell Processes.
UPDATE STATISTICS As this parameter only applies to DB2, it should not be
used within BT (BT uses Oracle).
Import Process Parameters
COMMIT OPERATIONS A Siebel Remote related parameter. Not relevant to either
OneSiebel r1.0 or OneView r6.0.
FILTER QUERY This option can be used to specify which rows in the EIM
table are to be processed e.g. FILTER QUERY =
(CUST_STAT_CD = “Active”). As BT has invested heavily in
external preparation of data for EIM (Ab Initio graphs), then
there should be no need to use a parameter like this.
MISC SQL This parameter should be used for initial loads only – the
author will review its applicability for OneSiebel at a later
date.
ATTACHMENT DIRECTORY This parameter is used to specify the location of attachments
to be loaded into Siebel. EIM must have both read and write
access to this directory. (Default = SIEBEL_HOME/INPUT)
Export Process Parameters
Note: The EIM Export process is not generally used within BT but
CLEAR INTERFACE TABLE The default for this parameter is True meaning that the EIM
export process deletes all existing rows in the interface table
prior to starting the export process. Use of this parameter is
only possible where no other users/processes are sharing the
specific EIM table to be used.
EXPORT ALL ROWS Synonymous with DELETE ALL ROWS
EXPORT MATCHES Synonymous with DELETE MATCHES
Delete Process Parameters
CASCADE DELETE ONLY Used to specify whether child records are deleted (TRUE) or
their FK is updated to NULL (TRUE). The default is False.
DELETE SKIP PRIMARY This parameter specifies whether EIM should perform a
cascade update to the primary child column. The Default
value is TRUE.
Merge Process Parameters
SET BASED LOGGING A Siebel Remote related parameter. This parameter is only
relevant if Docking:Transaction Logging System Preference is
set to True. Not relevant to either OneSiebel r1.0 or OneView
r6.0.

Page 21 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

3 Running EIM

3.1 Data

A review of several Siebel implementations at BT has revealed that there has, on occasion, been
some confusion as to what data needs to be populated for each different job type. This section
clarifies what data is required for each type of EIM job.

 Import (Insert) – To insert a new record (into Primary or Child Base Tables), User
Keys, Required Fields, Data columns and System Columns need to be populated.
 Import (Update) – To update an existing record just the User Keys, specific Data
Columns to update and System Columns need to be populated.
 Delete – To Delete a record using Delete Exact, just the User Keys and System Columns
need to be populated.
 Merge – To Merge records, just User Keys and System fields need be populated.

The following definitions apply to the above:

 User Keys – the combination of columns used by EIM to uniquely identify a row
(synonymous with ‘Compound Keys’).
 Required Fields – those fields that must be populated when creating a new record.
 Data Columns – regular columns in the EIM table that map to columns in the Base
Tables.
 System Columns – CREATED, CREATED_BY, LAST_UPD, LAST_UPD_BY, ROW_ID,
IF_ROW_BATCH_NUM, IF_ROW_MERGE_ID, IF_ROW_STAT, MODIFICATION_NUM,
CONFLICT_ID.

The following are mandatory System Columns:

 IF_ROW_BATCH_NUM – controls the Batch to which the record belongs

 ROW_ID – used to uniquely identify records within a batch. The combination of
IF_ROW_BATCH_NUM and ROW_ID must be unique.
 IF_ROW_MERGE_ID (for EIM Merge processes only) – set this to the ROW_ID within
the batch that the record will be merged into.
 IF_ROW_STAT – set with a meaningful status, such as ‘FOR_IMPORT’, ‘FOR_UPDATE’.
This will be set to ‘IN_PROGRESS’ by EIM during processing.

The other System Columns are defaulted on a table level (Oracle default value) in some
instances – not all instances. It is, however, good practice to set these fields explicitly so that
records in EIM tables can be easily identified. Note that the values populated in these other
System Columns will not influence the corresponding Base table records – these columns are
set by the EIM process.

 Only load the data required by the interface.

Page 22 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

3.2 IFB File and EIM Processes

This section details guidelines that should be adopted by BT Siebel interface development teams
relating to how to logically collate multiple EIM processes into ifb files.

In general, there should be a single ifb file per entity, with multiple processes being maintained
within that ifb file e.g.:

A file called onesiebel_customer.ifb should contain all Customer related EIM processes for
OneSiebel, such as:

 Import_OneSiebel_Customer
 Update_OneSiebel_Customer
 Cease_OneSiebel_Customer
 Delete_OneSiebel_Customer
 Merge_OneSiebel_Customer

Note: the Cease process is a logical delete of the Customer whilst the Delete process is a
physical delete.

By including multiple EIM processes for a given entity within a single IFB file, then configuration
management can be controlled at an appropriate level.

The calling process (Ab Initio adapter, script etc) should specify the process to run (and other
parameters) via the command line e.g.:

$SIEBEL_ROOT/bin/srvrmgr -g $GATEWAY:$GPORT -e $ENTERPRISE -s $APPSERVER -u

$SUSER -p $SPASS -c "run task for component eim with config="$IFB", traceflags=3,
errorflags=1, SQLFlags=8, ExtendedParams="BatchRange=$BATCH", Process="$PNAME""

(UNIX escape sequences have been omitted for clarity).

 $SIEBEL_ROOT – the path where the Siebel Application Server is installed

 $GATEWAY – the IP address (or hostname) of the Siebel Gateway Server
 $GPORT – the port on which the Gateway Server is running
 $ENTERPRISE – the name of the (Logical) Siebel Enterprise Server
 $APPSERVER – the name of the (Logical) Siebel Application Server
 $SUSER - the Siebel application user to be used to run the EIM process under
 $SPASS – the password of the aforementioned user
 $IFB – the full path and file name of the IFB file to be used
 $BATCH – the batch or batch range to be processed by this EIM process (the use of
batch ranges minimizes the overheads associated with starting an EIM process)
 $PNAME – the name of the EIM Process within the IFB file to invoke

As EIM name-value parameter pairs and extended parameters may need to be controlled on an
ad-hoc basis, then the calling process should be configurable in the way in which the command
line is structured and sent to Server Manager.

By including multiple EIM processes for a given entity within a single IFB file, then
 configuration management can be controlled at an appropriate level. This guideline
should be adopted.

Page 23 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

3.3 Scheduling

This section discusses some factors effecting the running of EIM as follows:

 As EIM operates on batches of data, then use of database server resources (in terms of
redo, archiving and undo) can be optimised by use of different batch sizes.
 EIM runs as a single threaded process against the database server, therefore it can be run in
parallel to increase throughput.
 Different scheduling options.

3.3.1 Degree of Parallelism/Layout

The GSL supports the running of EIM batches in parallel. This means that concurrent EIM
streams are fired off at the same time, working against different ranges of batches
simultaneously.

During Volume 4 Performance Testing determined the optimal degree of parallelism by varying
the number of concurrent streams and analysing throughput. A limit comes about when the
competition for I/O results in a degraded throughout as more and more streams are added. If a
job is more CPU bound than IO bound, then parallelism can generally be increased (e.g. to 8-
way).

By determining the optimal degree of parallelism, then better EIM throughput can be achieved
although competing processes also need to be catered for.

 Performance Testing should vary the Layout to achieve the highest throughput.

3.3.2 Batchsize

Similarly to Layout, EIM’s best throughput can be found by varying batchsize during testing.

For Volume, the optimal batchsize is generally 10,000 records per batch although smaller
batches must be used for Merge and Delete jobs to reduce the risk of causing competing
process to fail (e.g. “Snapshot too old” errors). Such a large optimal batchsize is possible when
the buffer cache is large.

 Performance Testing should vary the Batchsize to achieve the highest throughput.

3.3.3 EIM Streaming Configurations

Whilst Converge determined that when spawning more than 5 EIM stream this needed to be
spread across multiple Siebel Application Servers, such behaviour has yet to be exhibited on
Volume.

Volume runs all EIM processes from a single Siebel Application Server.

Volume 4’s Performance was consistent in determining that I/O on the database server was the
single biggest bottleneck (for loads into tables not populated, the single biggest bottleneck is

Page 24 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

generally with CPU). This therefore did not give rise to performing testing the spawning of EIM
from multiple Siebel Application Servers.

Performance Testing should determine whether increasing the number of physical

Siebel Application Servers enhances EIM performance. If this behaviour is not verified,
the multi-server EIM streaming configuration should not be adopted.

3.3.4 Control-M Scheduling

In Volume, the current schedule (maintained and operated from Control-M) is based upon 3
streams and looks something like this:

Stream 1 Stream 2 Stream 3

Process 1 Process 4

Unload
Process 2 Process 5 Entity 1

Unload
Process 3 Process n Entity 2

Unload
Process n Entity n

In this schedule, EIM table loads happen at the tail end of the on-line day (8pm onwards) – this
has not yet caused any performance impact upon the on-line processes.

In the excerpt above, multiple streams run in the schedule:

 Stream 1 runs EIM processes (each of which is running in parallel). There are dependencies
between the processes between the streams
 Stream 2 runs EIM processes as in Stream 1.
 Stream 3 runs unloads with dependencies with the EIM processes in the other two streams –
a table isn’t unloaded until all EIM processing against that table is complete (S_ORG_EXT is,
therefore, unloaded last in Stream 3 as it is accessed by most EIM jobs).

In this approach jobs that process different EIM and Base tables can be run simultaneously,
with unloads going on in parallel – dependencies between EIM streams and unloads must be
enforced. This maximises use of the batch window, as long as the database server can cope
with the demands put upon it.

Where a process is determined to run in isolation with maximum throughput at a certain degree
of parallelism during testing, this may need to be reduced if run in parallel with other EIM
processes and alongside unloads.

Page 25 of 26
IN CONFIDENCE
 Siebel EIM Best Practice

3.4 EIM Table Maintenance

Due to the number of operations executed against the EIM tables, then the following table
maintenance is recommended:

 Truncate EIM tables regularly – This has many advantages including releasing unused
storage and removing fragmentation.
 Analyse EIM tables after each table load – As EIM executes many SQL statements in
Cost Based Mode, then table (and index) statistics assist Oracle in choosing an
appropriate execution path. Regular statistics gathering on the base tables should also
be done for the same reasons.
 Storage Parameters – given that Row Chaining can occur on some EIM tables, then
the project DBAs should consider setting a high PCTFree value.

Page 26 of 26
IN CONFIDENCE