International Turnkey Systems

SDG Telco (Service Delivery Group)

User Acceptance
For
10g Upgrade at BGL

Project/Application:
CCBS

Prepared For:
BGL

Version Number: 1.0

Page 1 of 21
Document Control
Change Record

Date Author Version Change Reference

20/11/2008 Umashankar Barua 1.0 BGL10gUpgrade

Reviewers

Name Department / Position Review Date

Nazir Akhtar Team Lead

28-Nov-2008

Distribution List

Copy No. Name Location

References

Document Customer / Author Document Type Document Name Date

Version Name
Number

1.0 BGL Rollout/Rollback

BGL10gUpgrade 20/11/2008

Contact Persons
S No. Person Email Telephone/Ext

1 Umashankar Barua uma.barua@its.ws

uma.barua@its.ws
2 Nazir Akhter mnazir.akhter@its.ws

Page 2 of 21
 TABLE OF CONTENTS

INTRODUCTION 3
OVERVIEW ... 4
Existing Environment .. 4
Target Environment .. 5
Database Upgrade Methods 6
HARDWARE AND SOFTWARE REQUIREMENTS .. 7
ADOPTED UPGRADE METHODS 8
BACKUP STRATEGIES .. 9
ROLLBACK PLANS . 10
UPGRADING DATABASES......12
TABS . 12
TABSMED .14
CONFIGURING THE APPLICATION SERVERS ..16
CONFIGURING WEBUTIL .18
CUTOVER PLAN 19

Page 3 of 21
INTRODUCTION

This document intended to cover the aspects of the upgrade principals for Banglalink Billing environment.

In this document following topics will be covered

1. Upgrade methodology for each type of Database

2. Reconfiguration of application servers
3. Hardware and Software requirements for upgrade.
4. Test cases for Database and application server related components
5. Risk assessment
6. Backup strategy for the upgrade
7. Fallback planning and methods in case of abandoning the upgrade.
8. Common pitfalls for the operation
9. Step by step guide for upgrade

Page 4 of 21
OVERVIEW

EXISTING ENVIRONMENT

Databases

Database Name: TABS

Database Type: Oracle 9i RAC (Real application Cluster)
Instances: 2 (TABS1, TABS2)
Database Version: 9.2.0.6
Cluster Ware: MC/SeviceGuard (A 11.16)
Operating System: Hp-ux 11i (PA-RISC)
Machine: HP 9000 Series servers
TABS version: 6.5.3

Database Name: TABSMED

Database Type: Single Instance (non-clustered)
Instances: 1 (TABSMED)
Database Version: 9.2.0.8
Operating System: Hp-ux 11i (PA-RISC)
Machine: HP 9000 Series servers
TABS version: 6.5.3

Application Servers

Application Server Version: Oracle 10g R1 (9.0.4)

Installation Type: Mixed (2 with FRServices, 2 with Infrastructure)
TABS version: 6.5.3
Operating System: Windows 2003 server
Machine: HP R4440

Page 5 of 21
OVERVIEW

TARGET ENVIRONMENT

Databases

Database Name: TABS

Database Type: Oracle 10g R2 RAC (Real application Cluster)
Instances: 2 (TABS1, TABS2)
Database Version: 10.2.0.4
CRS version: 10.2.0.4
Cluster Ware: MC/SeviceGuard (A 11.16)
Operating System: Hp-ux 11i (PA-RISC)
Machine: HP 9000 Series servers
TABS version: 6.5.3
Server Role: Cater frontend application and post-paid mediation, rating.

Database Name: TABSMED

Database Type: Single Instance (non-clustered)
Instances: 1 (TABSMED)
Database Version: 10.2.0.4
Operating System: Hp-ux 11i (PA-RISC)
Machine: HP 9000 Series servers
TABS version: 6.5.3
Server Role: Cater pre-paid loading

Application Servers

Application Server Version: Oracle 10g R2 (10.1.2)

Installation Type: Mixed (All with Infrastructure)
TABS version: 6.5.3
Operating System: Windows 2003 server
Machine: HP R4440

Page 6 of 21
OVERVIEW

DATABASE UPGRADE METHODS

Oracle Database 10g supports the following tools and methods for upgrading a database to the new Oracle
Database 10g release:

Database Upgrade Assistant (DBUA)

Provides a graphical user interface (GUI) that guides you through the upgrade of a
database. The DBUA can be launched during installation with the Oracle Universal Installer,
or you can launch the DBUA as a standalone tool at any time in the future.

Manual upgrade using SQL scripts and utilities

Export and Import utilities
CREATE TABLE AS SQL statement

All the methods have their own advantages and disadvantages. Oracle recommends to use DBUA, as its
automate almost all the aspects of the Upgrade process.

However, by using DBUA, there is very little control over the overall upgrade process. In manual upgrade,
Many steps can be controlled by the DBA or system administrators. And in Export/Import method we can
have a fresh database, specially a brand new data dictionary.

But comparatively export/import method can take a lot of time to accomplish depending on the size of the
database. We have chosen mixed type of upgrade method for different database, according to the
business needs.

Page 7 of 21
HARDWARE AND SOFTWARE REQUIREMENTS

Hardware

No additional Hardware required for the upgrade activity.

Storage

For TABS database the activity will require exactly double amount of space that allocated to the
existing environment.

For TABSMED database Enough space should be provided to keep a full cold backup of database.

Software

Operating system

Following patches need to be installed

Quality Pack Bundle:

HP-UX 11i Quality Pack (GOLDQPK11i)

HP-UX 11.11 Patches:

PHNE_31097: ONC/NFS general release/performance patch

PHSS_31221: HP aC++ -AA runtime libraries (aCC A.03.60)
PHSS_32508: HP aC++ Compiler (A.03.63)
PHSS_32509: ANSI C compiler B.11.11.12 cumulative patch
PHSS_32510: +O4/PBO Compiler B.11.11.12 cumulative match

Serviceguard and OS Patches:

PHSS_32731: Serviceguard A.11.16.00 (replaces PHSS_31075)
PHSS_32658: Serviceguard Extension for RAC A.11.16.00 (replaces PHSS_31079)
PHSS_32732: COM B.03.00.00 (replaces PHSS_31077)

Page 8 of 21
 Patches for JDK on HP-UX 11.11:

PHSS_30970: ld (1) and linker tools cumulative patch

Database

Oracle 10g Release 2 (10.2.0.1)

Oracle 10g Patchset 4 (10.2.0.4)

Application server

Oracle Application server 10g Release 2 (10.1.2) with infrastructure for windows.
Webutil 106
Jacob 1.8

ADOPTED UPGRADE METHOD

For TABS database

The upgrade method chosen for TABS database is Export/Import

Considerations taken to choose this method

To have a fresh data dictionary

To have an standby instance to fallback with minimum efforts and downtime
To redistribute the loads on storage

Risks associated

The Export procedure can take long time due the data dictionary problem with current
installation
Import can take longer then expected time due to performance issue on newly allocated
storage

For TABSMED database

The upgrade method chosen for TABSMED database is DBUA

Consideration taken to choose this method

Reliability of DBUA
Lesser business criticality
No points to have a fresh data dictionary
Upgrade timing

Risks associated

Page 9 of 21
 DBUA may fail to accomplish task, which will require redoing everything as DBUA provides very
less control over upgrade process
In case of fallback it might take longer time then expected to restore the backup

For APPLICATION SERVERS

The application servers will be re-installed and reconfigured

Consideration taken to choose this method

different file structures

Different combinations of softwares
Problem with report servers

Risks associated

Client side browser and jinitiator version mismatch can cause troubles

BACKUP STRATEGIES

For TABS database

We will keep the original database untouched throughout the Upgrade process time window. And
we may keep the database for several days, in case, if decision comes to revert back to previous
version of database.

To keep this with original setting, we will use different voting disk and different OCR for the Oracle
10g CRS installation. So we will take following measure prior to install CRS.

Stop GSD services using gsdctl

copy the contents of 9i OCR into the 10g OCR device using the dd command

$ whoami
oracle

$ dd if=<9i_ocr_file> of=<renamed_9i_ocr_file> bs=1024

Where ocr file name found in /var/opt/oracle/srvConfig.loc

Edit the srvConfig.loc file to point to the <renamed_9i_ocr_file> file

For TABSMED database

For TABSMED we are going use DBUA for upgrading the database. So for this we need to have a
valid backup to be restored if any emergency.The preferred method is to take a full database cold
backup to disk. After Shutting down the database

Page 10 of 21
 rman"target/nocatalog"

startupmount;

RUN
{
ALLOCATECHANNELc1 TYPEDISK;
BACKUPascopyDATABASEFORMAT'some_backup_directory%U'TAGbefore_upgrade;
BACKUPascopyCURRENTCONTROLFILEformat'backup_location';
}

p.sAllocateasmuchasthenumberofcpuinmachine

For APPLICATION SERVERS

For application servers we will re-configure the servers. So we will keep a copy of the server to
reinstate when necessary. For each server, before reconfiguration

Take a system state backup

Take a copy backup of full file system after shutting down all oracle services

ROLLBACK PLANS

For TABS database

As the existing database is not touched during the upgrade, we need to re-point to the previous
configuration files and bring up the GSD components and run 9i database on it. To bring up the 9i
database we need to perform following steps

stop 10g database with srvctl

srvctl stop database -d TABS

stop CRS with crsctl

crsctl stop crs

Edit the srvConfig.loc file to point to the <9i_ocr_file>

Start GSD with gsdctl

gsdctl start

Start the database using srvctl

srvctl start database -d TABS

Page 11 of 21
 If in case, CRS installation need to be removed, following steps needs to considered

Run following scripts from $ORA_CRS_HOME/install

rootdelete.sh
rootdeinstall.sh

if any problem to run these scripts or if it fails then proceed to next step to manually remove
CRS configuration

Stop the Nodeapps on all nodes

srvctl stop nodeapps -n <node name>

Prevent CRS from starting when the node boots

rm /sbin/init.d/init.cssd
rm /sbin/init.d/init.crs
rm /sbin/init.d/init.crsd
rm /sbin/init.d/init.evmd
rm /sbin/rc2.d/K960init.crs
rm /sbin/rc2.d/K001init.crs
rm /sbin/rc3.d/K960init.crs
rm /sbin/rc3.d/S960init.crs
rm -Rf /var/opt/oracle/scls_scr
rm -Rf /var/opt/oracle/oprocd
rm /etc/inittab.crs
cp /etc/inittab.orig /etc/inittab

Reboot the nodes

Remove the ocr.loc

De-install the CRS home in the Oracle Universal Installer

Remove the CRS install location

Clean out the OCR and Voting Files with dd commands

For TABSMED database

In case of abandoning the upgrade process the backup taken before upgrade needs to be restored.
To bring up the database, following steps needs to be done

rman"target/nocatalog"

STARTUPNOMOUNT

RUN
{
REPLICATECONTROLFILEFROM'backup_location';
ALTERDATABASEMOUNT;
RESTOREDATABASEFROMTAGbefore_upgrade;
ALTERDATABASEOPENRESETLOGS;

Page 12 of 21
}

For APPLICATION SERVERS

To restore and reinstate the previous application servers we need to restore

The files and folders from backup

Restore the system state backup

UPGRADING DATABASES

For TABS database

PREUPGRADE TASKS

1. Purging all data according to the policy

2. Dropping all temporary tables
3. Generate scripts from production for following objects

TABLESPACES
ROLES
PROFILES
USERS
DATABASE LINK
DIRECTORIES
VIEWS
PROCEDURES
FUNCTIONS

Page 13 of 21
 PACKAGES
TYPES
TRIGGERS
JOBS
SEQUENCES
GRANTS
SYNONYMS

4. Customize the tablespace script to reflect the new locations of the datafiles
5. Taking Backup of oracle binaries and orainventory
6. Taking optimizer statistics for the TABS schema

UPGRADE TASKS

Current Version/Release: 9.2.0.6

Destination Version/Release: 10.2.0.4
Upgrade Method: Export/Import
ORACLE_HOME: /orabin/appl/oracle/product/10.2.0

1. Export TABS, HIST schema and also any other schema used by in-house built applications.
2. Add entries for VIP in /etc/hosts files
3. Edit network configuration to add alias for the VIP with port 801
4. Shutdown the 9i database
5. shutdown GSD Daemons
6. Apply Operating system patches
7. Adjusting kernel parameters
8. Create another pair of voting disk and OCR to preserve previous ones.
9. Modify srvConfig.loc to point new copied OCR
10. check /var/opt/oracle with ll
11. Install Oracle 10g CRS
12. Change ownership of all the raw devices to oracle.
13. Run root.sh in both nodes
14. check /var/opt/oracle with ll and compare with pre-CRS check
15. Check for CRS processes
16. Upgrade CRS and apply patchset 4
17. Install oracle 10g database binaries
18. Apply database patchset 4
19. Create a new database
20. Create the tablespaces and users
21. Import TABS schema
22. Adjusting TABS schema with updated script
23. Recreate all synonyms
24. Recreate all database links
25. Recreate all sequences

POSTUPGRADE TASKS

1. Take full database backup

2. Change all environment variables accordingly (tabsenv, TABS-Profile.sh). Following parameters
should be changed/added
ORACLE_HOME
PATH
ORA_NLS10
LD_LIBRARY_PATH

Page 14 of 21
 3. Set THRESHOLD values for tablespace alerts if necessary
4. Upgrade the catalog database

Known Issues and workarounds

Problem: VIPCA fails at the end of CRS installation

Reason: VIPCA treats 172 series ip as private ip
Solution: We need to add the nodeapps manually with srvctl command. For example,

srvctl add nodeapps -n machinename -o <ORACLE_HOME> -A 1.2.3.4/255.255.255.0 (interface)

UPGRADING DATABASES

For TABSMED database

PRE-UPGRADE TASKS

1. Purging all data according to the policy

2. Dropping all temporary tables
3. Generate scripts from production for following objects

TABLESPACES
DATABASE LINK
DIRECTORIES

Page 15 of 21
 JOBS
SEQUENCES
SYNONYMS

4. Taking Backup of oracle binaries and orainventory

5. Keep minimum possible partitions in historical data
6. Move tables to tablespaces with standard block sizes from non standard block sizes.
7. Adjust Buffer cache parameter

UPGRADE TASKS

Current Version/Release: 9.2.0.8

Destination Version/Release: 10.2.0.4
Upgrade Method: DBUA
ORACLE_HOME: /orabin/appl/oracle/product/10.2.0

1. Running the Pre-upgrade script utlu102i.sql

2. Adjust the database according to the output of pre-upgrade script.
3. shutdown database
4. Take a full cold database backup as copy using RMAN.
5. Apply operating system Patches
6. Adjusting kernel parameters
7. Install Oracle binaries
8. Install Patch level 4
9. Install additional components for Oracle 10g companion media
10. Take a full cold database backup to disk using RMAN
11. startup open the database
12. Run netca from 10g database home to configure listener and tnsnames.ora
13. Run DBUA and select the database
14. Create sysaux tablespace with 3G size when prompted
15. Check the DBUA upgrade log for any discrepancies.
16. Recreate all database links

POSTUPGRADE TASKS

1. Take full database backup

2. Change all environment variables accordingly (tabsenv, TABS-Profile.sh). Following parameters
should be changed/added

ORACLE_HOME
PATH
ORA_NLS10
LD_LIBRARY_PATH

3. Set THRESHOLD values for tablespace alerts if necessary

Page 16 of 21
APPLICATION SERVER CONFIGURATION

Operating System: - Windows 2003 server

Application Server: - 10 g Release 2 with Infrastructure (10.1.2)
Webutil: - Version 106 with Jacob 18

1. Install 10G release 2

Page 17 of 21
 Installation type=Oracle Forms and Business intelligence

For Infrustructure

Oracle home= Infrahome

ORACLE_HOME=d:\oracle10gR2INFRA
ORACLE_SID=TABSINF

For AS Middle tier

Oracle home= AShome

ORACLE_HOME= d:\oracle10gR2AS

2. Create $bin Location (For Front End executables)

Create folder c:\$BIN

3. Copy jar files

Copy jar files to $ORACLE_HOME/forms/java directory

Namely
JGo, JGoSVG ,ITSBeans ,FTPBean ,Sftp ,Hst65 ,Icons ,Browser, Browser2

4. Customize Application server environment files

Copy default.env to tabs.env in $ORACLE_HOME/forms/server

Add/edit following parameters

FORMS_PATH=c:\$bin
REPORTS_PATH=c:\$bin
NLS_LANG=AMERICAN_AMERICA.AR8ISO8859P6

5. Customize formweb.cfg File

Go to following location

d:\oracle10gR2AS\forms\server

Take a Backup of formsweb.cfg file before modifying. Add config section tabs in the formsweb.cfg file
tabs config in formsweb.cfg will look like,

[tabs]
envfile=tabs.env
HTMLbeforeForm=<script>var windowHandle;function openWindowURL(URLString){var sptest
= URLString.split("/");var mstring = sptest[sptest.length-1];var fstring =
mstring.split("_");var furl =
sptest[0]+"//"+sptest[2]+"/"+sptest[3]+"/"+fstring[0].substring(2,fstring[0].length)
+"/"+mstring;a = "100%";mw_content =
"<html><head><title>Bill_View<\/title><\/head><body oncontextmenu='return false;'
topmargin='0' leftmargin='0' rightmargin='0'><iframe width='"+a+"' height='"+a+"'
frameborder='0' src='"+furl+"'><\/iframe><\/body><\/html>";windowHandle =

Page 18 of 21
 window.open("javascript:opener.mw_content","Bill","location=false,scrollbars,resizab
le");windowHandle.defaultStatus="Bill";windowHandle.focus();}</script>
otherparams=CONNECTSTRING=tabs FRUN=sommenu.fmx
webUtilArchive=/forms/webutil/frmwebutil.jar,/forms/webutil/jacob.jar,/forms/webutil
/JScript.jar,/forms/webutil/CtcPjcBean.jar,/forms/webutil/medTcp.jar,/forms/webutil/
ocxTcp.jar,/forms/webutil/rolloverbutton.jar
baseHTMLJInitiator=webutiljini.htm
baseHTMLjpi=webutiljpi.htm
baseHTML=webutilbase.htm
imageBase=Codebase
archive_jini=frmall_jinit.jar,icons.jar,colorpicker.jar,xalan.jar,xercesImpl.jar,xml
-
apis.jar,xmlcomp.jar,xmlparserv2.jar,FormsGraph.jar,modcursor.jar,TCPIP.jar,browser.
jar,xerces.jar,browser2.jar,frmall.jar,JScriptWindowPJC.jar,jdev-rt.jar
form=SOMPASS.fmx
separateFrame=true
width=1100
height=760
splashScreen=false

Restart the Application server Middle tier

d:\oracle10gR2AS\opmn\bin\opmnctl stopall
d:\oracle10gR2AS\opmn\bin\opmnctl startall

6. ADD tns entry of the backend database in tnsnames.ora file

Add tns entry in following files

d:\oracle10gR2AS\network\admin\tnsnames.ora
d:\oracle10gR2INFRA\network\admin\tnsnames.ora

For example

TABSRND =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 172.16.10.14)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = TABSRND)
)
)

7. overwrite key resource file namely

fmrweb_utf8.res
fmrweb.res

Location (windows): d:\oracle10gR2AS/forms/

Take backup of Previous files and then place the files.

CONFIGURING WEBUTIL

webutil version: 106

Jacob version: 18

Page 19 of 21
 Create a Webutil folder inside $ORACLE_HOME/forms. Copy webutil folder in
$ORACLE_HOME/forms we will refer this as webutil home.

Create a directory lib inside webutil folder

Edit $ORACLE_HOME/forms/server/forms.conf

Comment out
#AliasMatch ^/forms/webutil/(..*) "/appl/oracle/ias10/forms/webutil/$1"

And add
AliasMatch ^/forms/webutil/(..*) "/appl/oracle/ias10/forms/webutil/lib/$1"

Modify entry of of WEBUTIL_CONF parameter in application server env file

WEBUTIL_CONFIG=$ORACLE_HOME/forms/webutil/server/webutil.cfg

Modify $ORACLE_HOME/forms/server/formsweb.cfg to add following

webUtilArchive=/forms/webutil/frmwebutil.jar,/forms/webutil/jacob.jar
baseHTMLjinitiator=<webutil_home>\server\webutiljini.htm
baseHTMLjpi=<webutil_home> \server\webutiljpi.htm
baseHTML=<webutil_home>\ server\webutilbase.htm

Sign the jar files

Set JDK_HOME=$ORACLE_HOME/jdk
Sign_webutil.bat Jacob.jar
Sign_webutil.bat frmwebutil.jar

Install WEBUTIL_DB package in webutil user

following paths in CLASSPATH of your application server env file

frmwebutil.jar,Jacob.jar,rt.jar

Put forms_base_ie.js and forms_ie.js in $ORACLE_HOME/forms/java folder

CUTOVER PLAN

Page 20 of 21
The Upgrade process consists of following components

1. Upgrading the RAC environment for TABS database.

2. Re-creating TABS database with Export/import
3. Upgrading the TABSMED database
4. Reconfiguring the Application server

The tasks are listed down with approximate timings. Some of the activities will be performed in parallel.
For the Upgrade activities and Pre go-live testing we will require 48 hours downtime (approx.)

DB/AS Task Duration (Approx.)

TABS Export TABS, HIST schema and also any 6 hrs
other schema used by in-house built
applications.

Shutdown the 9i RAC 20 min

Apply Operating system patches and 3 hrs
adjusting kernel parameters
Create another pair of voting disk and OCR 20 min
to preserve previous ones.
Install Oracle 10g CRS and patching CRS 1 hrs
Install oracle 10g database binaries and 3 hrs
apply patches
Create a new database 1 hrs
Create the tablespaces and users 1 hrs
Import TABS schema 15 hrs
Adjusting TABS schema with updated script 1 hrs
Checking for invalid objects and recreating 1 hrs
all necessary objects
Updating and adjusting the environment for 1 hrs
TABS applications
Deploying applications 20 min

TABSMED Running the Pre-upgrade script utlu102i.sql 1 hrs

and adjust the database
Shutdown the database and Take a full cold 3 hrs
database backup as copy using RMAN
Apply operating system Patches and 3 hrs
adjusting kernel parameters
Install Oracle binaries and apply the patches 3 hrs
Run DBUA followed by DBUA to upgrade the 3 hrs
database
Apply synchronization scripts 2 hrs

Application server For each application server to commission 6 hrs

Page 21 of 21