h6540 Oracle Backup Recovery Perform Avamar Rman WP

Oracle Backup, Recovery, and Performance
Tuning using EMC Avamar and Oracle RMAN

Best Practices Planning
Abstract
This white paper provides an in-depth review of the capabilities of the EMC Avamar Oracle plug-in. The
primary focus of this paper is two-fold:
1. Provide the reader with a complete understanding of Avamar Oracle plug-in capabilities including
Oracle backup and recovery features and advanced concepts such as performance tuning
2. Describe how to protect large Oracle databases with the Avamar Oracle plug-in via multiple streams.
September 2009
Copyright 2009 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is
subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION
MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE
INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable
software license.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com
All other trademarks used herein are the property of their respective owners.
Part Number h6540
Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN
Table of Contents
Executive summary ............................................................................................ 5
Introduction ......................................................................................................... 5
Audience ...................................................................................................................................... 5
Benefits of using EMC Avamar.......................................................................... 5

Avamar and large databases ............................................................................. 6
Key features of using RMAN with the Avamar Oracle plug-in ........................ 7
Oracle database objects explained ................................................................... 7
System Global Area (SGA) .......................................................................................................... 7
Oracle control files ....................................................................................................................... 7
Oracle datafile .............................................................................................................................. 7
Oracle tablespace ........................................................................................................................ 8
Oracle online redo logs ................................................................................................................ 8
Oracle archived redo logs ............................................................................................................ 8
Oracle parameter file (init(SID).ora or pfile)................................................................................. 8
Other Oracle objects .................................................................................................................... 8
Recovery catalog ................................................................................................ 9

Oracle hot or cold backups................................................................................ 9
Online (hot) backups.................................................................................................................... 9
Offline (cold) backups .................................................................................................................. 9
Target database preparation............................................................................ 10

Install the Avamar operating system file system agent. ............................................................ 10
Install the SBT library on the target database............................................................................ 10
Create users on the Avamar Grid, target database, and recovery catalog ............................... 11
Create a user called backup in the Avamar GUI ................................................................. 11
Create a user in the target DB................................................................................................ 11
Create users in the recovery catalog ..................................................................................... 12
Archive Log Mode............................................................................................. 13

Oracle password file explained .................................................................................................. 14
Create your Oracle password file .............................................................................................. 14
RMAN and Oracle backups .............................................................................. 15

RMAN backup script broken down ............................................................................................ 15
Avamar-specific parts in the RMAN script.............................................................................. 16
Universal parts of the RMAN script ........................................................................................ 16
RMAN backup script execution and sample output ............................................................... 16
Avamar and RMAN backup example......................................................................................... 17
My-Avtar-Flags.txt explained ........................................................................... 18

Avamar and RMAN recovery scenarios for datafile recovery.................................................... 18
Recovery types in Oracle ....................................................................................................... 19
Avamar and RMAN Oracle disaster recovery ........................................................................ 19
Oracle Open Reset Logs command....................................................................................... 20
Performance tuning Oracle backups .............................................................. 20

Identifying the client-side physical system bottleneck ............................................................... 20
Physical bottleneck................................................................................................................. 20
Avamar cache sizing for databases ................................................................ 22

Rules for sizing hash caches properly ....................................................................................... 22
Examples of properly sized hash cache files ......................................................................... 23
The case for multiple RMAN channels............................................................ 23

Example scripts and multiple Avamar cache files...................................................................... 24
Multiple channel RMAN Avamar backup script ...................................................................... 24
Multiple channel RMAN restore script.................................................................................... 25
Multiple channel Avamar My-Avtar-Flags file......................................................................... 25
Sample outputs ................................................................................................. 26

Conclusion ........................................................................................................ 28
Appendix A: Avamar Oracle plug-in recovery catalog creation ................... 29
Create the catalog database instance in Oracle 9i.................................................................... 29
Create a Oracle database in Oracle 9i ...................................................................................... 29
Create catalog users.................................................................................................................. 30
Create the catalog schema .................................................................................................... 30
Register the target database .................................................................................................. 30
Appendix B: Avamar Oracle plug-in backup and recovery scripts .............. 31

Appendix C: Avamar Administrator Oracle GUI backups ............................. 35
Snapup options .......................................................................................................................... 35
Restore options.......................................................................................................................... 36
Troubleshooting GUI operations................................................................................................ 37
Appendix D: Avamar and Oracle theory of operations.................................. 37

Avoracle theory of operations .................................................................................................... 37
Avoracle operations ................................................................................................................... 38
Browse.................................................................................................................................... 38
Backup.................................................................................................................................... 38
Restores ................................................................................................................................. 39
Appendix E: Troubleshooting, hints, and tips................................................ 40

Diagnostic commands: A sample session ................................................................................. 40
Different Var directories depending on operating system.......................................................... 41
Executive summary
This white paper details procedures to enhance the performance and scalability of the EMC Avamar
Oracle plug-in for larger Oracle databases. In most cases the Oracle database size was greater than 1
terabyte.
The main goal of the testing was to demonstrate that the Avamar Oracle plug-in can address the data
protection needs of these larger databases. The test sample database was approximately 5 terabytes in size.
Multiple streams were used to achieve the recovery time objectives that were required.
Introduction
This white paper describes backup and recovery options for Oracle via Avamar and Oracle Recovery
Manager (RMAN) using the SBT_TAPE interface supplied in Avamar.
The primary focus of this paper is two-fold:
1. Provide the reader with a complete understanding of the Avamar Oracle plug-in capabilities including
Oracle backup and recovery concepts and advanced concepts such as performance tuning
2. Describe how to protect large Oracle databases with the Avamar Oracle plug-in via multiple streams.
It should be noted that Avamar complies with the standards that are set forth by Oracle. A shared library is
installed into the Oracle environment. This library allows Oracle RMAN to invoke the SBT interface. In
addition, there are several field-tested samples of scripts and configurations.
Audience
This white paper is intended for the technicians and database administrators (DBAs) who want to
understand how to utilize Avamar for their Oracle data protection needs. This group includes customers,
backup administrators, DBAs, technical consultants, delivery engineers, technical support engineers, and
EMC internal professionals.
Benefits of using EMC Avamar

The Avamar Oracle plug-in utilizes Avamars powerful source, global data deduplication technology.
Unlike traditional backup solutions, EMC Avamar identifies redundant data segments at the source (client)
before they are transferred across the network. By moving only new, unique subfile data segments,
Avamar delivers fast daily full backups while reducing the required daily network bandwidth by up to
500X. This capability allows companies to utilize existing network bandwidth for backup and disaster
recovery of remote offices and data centers, despite slow or congested networks and infrastructure. Data
can be encrypted both in flight and at rest for added security, and centralized management makes protecting
hundreds of remote offices easy and efficient.
By storing just a single instance of each subfile data segment globally, Avamar also reduces total back-end
storage by up to 50X, enabling cost-effective disk-based recovery over extended periods of time. Although
Avamar backs up data to disk, it can also work with existing tape and traditional backup software such as
EMC NetWorker. In addition, EMC Avamars grid architecture provides online scalability, and patented
redundant array of independent nodes (RAIN) technology delivers high availability.
Avamar and large databases

Any data protection discussion should focus on the customers business requirements such as desired
backup and recovery time objectives. Environmental factors should also be considered, especially for
databases larger than 500 GB. Key factors and questions to consider are the following:
Database size and version?
Available CPU and memory?
What type of disk subsystem is available?
What type of Avamar environment is in place and how large is it?
Was the Oracle database installed on the storage system using Oracles best practices?
By allocating multiple channels, and also having the proper environmental factors in place, Avamar is a
perfect fit for larger Oracle environments.
Several rounds of testing have been done on 4 TB and 5 TB Oracle databases. Avamar was used to back up
the database within a 10-12 hour nightly backup window.
Following are three real-world customer examples.
Example No. 1
Customer environment: 5 TB database hosted on a Sun Solaris machine with 128 GB of RAM (64 GB
of free RAM) and 12 core processors. The back-end disk is EMC Symmetrix DMX. Customer has a
recovery time objective of 24 hours and the backup window is 12 hours.
Can the Avamar Oracle plug-in be leveraged to protect this configuration?
Answer: Avamar would be a good fit here. You can allocate multiple RMAN channels to get the
backups and restores done within their windows.
Example No. 2
Customer environment: Sun Solaris machine with 128 GB of RAM running 25 instances of Oracle.
The system has 12 KB of RAM free.
Answer: In this case, the machine is already memory constrained with its current load. You can try
Avamar for backup and recovery, but the RAM bottleneck will still be there.
Example No. 3
Customer environment: 30 TB data warehouse hosted on a large AIX server with plenty of RAM and
processors, and a very fast disk subsystem.
Answer: It would be very difficult to complete the backup within the standard 10-12 hour backup
window. Avamar is likely not the best fit.
Key features of using RMAN with the Avamar Oracle

plug-in
The Avamar Oracle plug-in is a fully integrated third-party module that is designed to leverage Oracles
RMAN. By using RMAN, all of the key features of Oracles backup utility are available to Avamar.
RMAN is Oracles preferred method to back up and restore an Oracle database. You essentially run RMAN
and call for a SBT device type. This means RMAN data is sent to a third-party backup provider like
Avamar.
Following is a short list of key RMAN features.
Online backups Full and Incremental
Offline backups Full and Incremental
Archived redo log backups
Dedicated Oracle database to track backups recovery catalog
Oracle block-level corruption detection
Restore capabilities
Full system restore
Datafile restore
Tablespace level restore
Point-in-time restore
Archived redo log restores
In recent versions of Oracle, RMAN provides some advanced functionality. Please refer to Oracle
documentation for a complete listing.
Oracle database objects explained

The following section provides an explanation of all physical objects in the database that need to be
protected.
System Global Area (SGA)

When you mount or open an Oracle database, it invokes some background process, then reads what is
called a control file and loads the SGA. In simple terms the SGA is the area where Oracle does its work.
The more work you can do in RAM the faster the work will get done. The SGA is where your queries are
processed, it is a temporary area for your data blocks, and it is also where a good deal of the RMAN work
is processed.
Oracle control files

This file is a small binary file. It keeps track of the state of the entire database environment, and the system
change numbers (SCNs) of all objects. The database is useless without the control file. There is typically a
group of at least three of these. Oracle recommends them to be on separate spindles. They are the registry
of the Oracle system.
An example is /u01/oracle10g/ctrl1.ctl.
Oracle datafile
This is a physical file that typically has tables of data inside it. It is where the data is essentially stored. This
object can be put online or offline. A single datafile can be backed up or restored.
An example is F:\oradata\payroll.dbf.
Oracle tablespace
A tablespace is a logical collection of datafiles. It is a logical object, not a physical object. It can be
confused with an Oracle data table. It is useful to recover an entire tablespace depending on where the
database corruption occurred and how far it spread. The entire tablespace can be taken offline and online.
Examples are /u01/oradata/payroll.dbf, /u01/oradata/currency.dbf, and /u01/oradata/taxes.dbf.
Oracle online redo logs

These objects are where data is stored temporarily before it is committed to a datafile. It allows you to roll
back the database to a point in time in case of a corrupted block or if a mistake is made.
Online redo logs are typically in groups; however, there is a limited amount of space in the online redo
logs. Lets say we have three online redo logs. You fill the first, then the second, then the third. You would
then start back and overwrite the first log. This is fine for a smaller database with a small amount of
change. You typically do not back up online redo logs unless you shut the database down. This is because
they are constantly changing
To keep a history of changes in the database, activate Archive Log Mode. Archived Log Mode allows you
to make an archive copy of the log before it is overwritten.
Examples of online redo logs are E:\oradata\logs\rd_1.log, E:\oradata\logs\rd_2.log, and
E:\oradata\logs\rd_3.log.
Oracle archived redo logs

Archived redo logs (also referred to as archive logs) keep a history of the database in case you want to roll
back to a point in time. These archive logs are key objects to protect in the backup and recovery design.
RMAN does a great job of protecting these and also utilizing them during recovery efforts.
Examples are E:\oradata\arch\log_1.arc, E:\oradata\arch\log_2.arc, and E:\oradata\arch\log_3.arc.
Oracle parameter file (init(SID).ora or pfile)

The parameter file tells the Oracle database about the environment when it loads the SGA, and reads the
control files. It sets important parameters such as Archive Log Mode, how much RAM to use for SGA, and
many others. It is similar to an .ini file in Windows.
This used to be a flat file but has changed to be more like a control file in later versions of Oracle. This file
is protected in Oracle 9i and later by using the control file Autobackup function.
Other Oracle objects

Oracle objects (control files, datafiles, and archived redo logs) make up the physical objects that are
protected with Avamar and Oracle RMAN. The following objects are objects inside the database. They
help keep the system running efficiently.
Tablespace: Mentioned previously, it is a logical grouping of datafiles.
Index: Allows you to be more efficient with queries.
Table: Where you store data. It is comprised of rows and columns. Various datatypes.
Constraints: A DBA term for rules with data.
Views: Logical views of specific data. For example, V$Database allows you to see statistics on the
database. Sometimes referred to the system schema.
Instance: This is the actual target database you are working with. There may be a production instance,
a test instance, and also a development instance. This is typically referenced by a System Identifier
(SID) name. A good example of this may be Payroll_Prod.
Recovery catalog
The recovery catalog database is an Oracle database that is created and used to keep track of the RMAN
objects and allows you to recover them if you lose your target database control file. The target database is
the database that you are backing up. This catalog also allows you to perform advanced reporting on
backups and lists the target database schema.
If you choose not to use the recovery catalog database, your RMAN information is then stored in the target
databases control file.
Later versions of Oracle 9i have a featured called control file Autobackup, which makes a snapshot copy of
the control file and parameter file, and stores them in a location that you can take with the normal Avamar
file system backup. You can then recover the control file directly from the Autobackup location. This is a
configurable location.
It is important to understand that Oracle recommends that all production RMAN environments use a
recovery catalog. However Test and Dev environments running Oracle 9i and later can use the option. Be
careful on what the DBAs perception is. Most will push to use the recovery catalog either way.
The recovery catalog database should be running on a separate machine from the target database, which
will allows you to get your environment up and running quicker. Imagine the difficulty in having to recover
the catalog and then the target database.
Oracle hot or cold backups

Online (hot) backups
A backup of your environment while it is running is referred to as a hot backup or online backup. The
database must be running in Archive Log Mode to run a hot backup. This means that you are making
copies of online redo logs and archiving them before you overwrite them. It is a setting within the database.
RMAN is not needed to perform this hot backup. Simply put your Oracle tablespace in hot backup mode
and then perform a backup of the Oracle objects using the regular file system backup technology.
An example is as follows:
Alter Table space Payroll begin backup within the Oracle system.
To use RMAN to perform hot backups, the database must be running in Archive Log Mode.
Offline (cold) backups

A backup of your environment while it is shut down is an offline backup or cold backup. You do not need
to be running in Archive Log Mode to do this. Simply issue the proper shutdown command within Oracle
and then back up physical objects using the Avamar File System plug-in. While the database is shutting
down, all of the SGA and buffers are flushed to their logs and datafiles.
RMAN can be used to perform a cold backup. This is a little tricky because you then embed a SQL
statement into the RMAN script. Shut down the database normally, then start up the Oracle system with the
Mount Option. This is like a single user mode in UNIX. Then use RMAN to back up the objects.
Target database preparation

The following are the steps involved in getting the target database ready for a backup with Avamar and
RMAN.
1.
Ensure that the Avamar File System client is installed and the client is registered to the Avamar Utility
Node. Please refer to the EMC Avamar Oracle Client Installation Guide and Reference Manual.
2.
Install the Avamar SBT_TAPE library into your target database. This is outlined well in the EMC
Avamar Oracle Client Installation Guide and Reference Manual.
3.
Create a user within the target database and grant that user privileges. Also create the same user in
Avamar. If you are using a recovery catalog database, create that user in the recovery catalog as well.
4.
Put your target database into Archive Log Mode. This is required for hot backups.
5.
Create an Oracle password file on the target database.
Install the Avamar operating system file system agent.

This is a key step to make sure the client and server are communicating. The EMC Avamar Oracle Client
Installation Guide and Reference Manual has more details. You should be able to test a small backup of the
file system before you begin to test the Oracle backups. It is recommended that you back up and restore the
operating system data (file or file system) before you attempt to back up and restore the Oracle data.
Install the SBT library on the target database

This is somewhat similar and well documented for each operating system. The installer installs the Avamar
Library into a directory like /usr/local/avamar/lib. Reference the EMC Avamar Oracle Client Installation
Guide and Reference Manual for specific installation instructions.
This depends on the OS. You may need to shut down your target database and bring it back up to bind the
SBT_TAPE library with the Oracle kernel.
An example of this file in UNIX is called libobk_avamar.so. Once this library is in place you will be able to
reference it in the Avamar RMAN script.
The following samples are performed on an Oracle 10g system with Avamar version 4.x.
Example of the installation on Red Hat.

rpm -hi AVAMARRMAN.rpm
You can then log in to the database with SQL plus.

oradba@oratest1# sqlplus /nolog
connect
/ as sysdba
shutdown immediate
startup
exit
This step may be necessary to bind the SBT library with the Oracle kernel.
10
Create users on the Avamar Grid, target database, and recovery

catalog
You will need to create users within Avamar, the target database, and the recovery catalog. You will also
need to grant the proper privileges to these users. Follow these examples.
Create a user called backup in the Avamar GUI

The following are steps to create a user by utilizing the Avamar Administrator.
1. Start Avamar Administrator. Choose Navigation > Administration or click the Administration
launcher button. The Administration window appears.
2. Click the Account Management tab. In the tree, select the client you want to add the new user to.
3. Choose Actions > Account Management > New User(s) or click the toolbar icon shown immediately
to the left. The New User dialog box appears. Use the Axion Authentication System. It is the default.
4. Create a user backup with a password backup. Type the password backup to confirm.
If you need help with these steps please consult the EMC Avamar System Administration Manual. There are
some graphical examples in this guidel
Create a user in the target DB

Create a user backup in the target database and grant this user privileges. The samples below show proper
syntax.
This procedure may vary a little depending on which version of Oracle you are running. However, these
examples should get you fairly close. Please check with your Oracle DBA if you find issues. This is an
example from UNIX creation. These commands are run from the target system.
SQL*Plus: Release 9.2.0.1.0 - Production on Wed Apr 23 21:33:25 2003
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
SQL> connect
/ as sysdba
Connected to:
Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.1.0 Production
SQL> alter user backup identified by backup; Oracle 9

User altered.
SQL> alter user backup account unlock; Oracle 9
User altered
You may need to identify the users default tablespace. Here is an example.
SQL> alter user backup identified by backup
DEFAULT TABLESPACE users;
User altered.
Here is a more elaborate example of the preceding tablespace declaration.

11
SQL> create user username identified by password default

tablespace user_tablespace temporary tablespace temp_tablespace;
Remember to unlock the account!
SQL> alter user backup account unlock; Oracle 9i and above.
User altered.
SQL> grant sysdba to backup;
Grant succeeded.
Universal
SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.1.0 Production
JServer Release 9.2.0.1.0 - Production
Create users in the recovery catalog

You can create a user/password called backup in the recovery catalog and grant this user the proper
privileges. It is essentially the same procedure used to create the user backup with a password backup in the
target database.
You will need to log in to the machine that is running the recovery catalog database.
This procedure may vary a little depending on which version of Oracle you are running. However, these
examples should get you fairly close. Please check with your Oracle DBA if you find issues. This is an
example from UNIX creation. These commands are run from the target system.
oradba@rcvcat# sqlplus /nolog
SQL*Plus: Release 9.2.0.1.0 - Production on Wed Apr 23 21:33:25
2003
SQL>connect
/ as sysdba
Connected to:
Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production
SQL> alter user backup identified by backup; Oracle 9
User altered.
SQL> alter user backup account unlock; Oracle 9
User altered.
You may have to identify the users default tablespace. Here is an example.
SQL> alter user backup identified by backup
DEFAULT TABLESPACE users;
User altered.
Here is a more elaborate example of the preceding tablespace declaration.

SQL> create user username identified by password default
tablespace user_tablespace temporary tablespace temp_tablespace;
Remember to unlock the account!
12
SQL> alter user backup account unlock; Oracle 9i and above.

User altered.
SQL> grant sysdba to backup;
Grant succeeded.
Universal
SQL> grant connect,resource,recovery_catalog_owner to backup;

Grant succeeded.
SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.1.0 Production
Archive Log Mode

If you will be doing hot (online) backups, which is likely, you will need to put your target database in
Archive Log Mode. Here are some examples of doing so. This procedure may also vary with different
versions of Oracle.
Archive logging is enabled by updating the log startup parameter.
If you are running Oracle 9i with a PFILE, add the following entry to the init.ora file for the database to be
backed up. Example: /space/home/oracle/admin/eval/pfile/init(sid).ora
log_archive_start = true
If you are running Oracle 9i with an SPFILE do the following. You will not need to do this with Oracle 10g
and later.
SQL*Plus: Release 9.2.0.1.0 - Production on Thu Apr 24 17:38:56 2003
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected
to: Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the
Partitioning, OLAP and Oracle Data Mining options
SQL> connect / as sysdba
SQL> alter system set log_archive_start=true scope=spfile;
System altered.
SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options JServer Release
9.2.0.1.0 Production.
Now we can actually put the database into Archive Log Mode.
oracle@oratest1# setenv ORACLE_SID targetdb
SQL*Plus: Release 9.2.0.1.0 - Production on Thu Apr 24 17:38:56 2003
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected
to: Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the
Partitioning, OLAP and Oracle Data Mining options
SQL> connect /
as sysdba
SQL> shutdown immediate

Database closed.
13
Database dismounted.
ORACLE instance shut down
SQL> startup mount
ORACLE instance started.
(This is like single user mode in UNIX.)
SQL> alter database archivelog;

Database altered.
SQL> alter database open;
Database altered.
SQL> archive log list

Database log mode Archive Mode
Automatic archival Enabled
Archive destination /space/home/oracle/product/9.2.0/dbs/arch
Oldest online log sequence 1 Next log sequence to archive 3
SQL> exit
Oracle password file explained

You can create a password file on the target database system, as well as the recovery catalog system. This
will help avoid authentication problems when trying to make connections to get backups and recoveries
started.
The following is a short explanation of the password file and what it is used for.
If the DBA wants to start up an Oracle instance there must be a way for Oracle to authenticate this user.
That is, if they are allowed to do so.
Obviously, the users password cannot be stored in the database, because Oracle cannot access the
database before the instance is started up. Therefore, the authentication of the user must happen outside of
the database.
There are two distinct mechanisms to authenticate the user: using the password file or through the operating
system. The init parameter remote_login_passwordfile specifies if a password file is used to authenticate
the DBA or not. If set either to shared or exclusive, then a password file will be used.
The following is an appropriate example of the creation of this password file. You will use the orapwd
utility.
Create your Oracle password file

These are the commands required to create the password file on the Oracle database system.
oracle@oratest1# setenv ORACLE_SID targetdb (or catalog db )
oracle@oratest1# cd $ORACLE_HOME/dbs
/oracle/product/10.1.0/Db_1/dbs
oracle@oratest1# orapwd file=/oracle/product/10.1.0/Db_ 1/dbs/orapw
password=PASSWORD
Where PASSWORD is the sys account password on the database that is pointed to by ORACLE_SID.
Verify that the Oracle password file was created by entering:
14
ls -la orapw
The following information appears in your command shell:

-rwSr----- 1 oracle oracle 1536 Oct 1 17:02 orapw
Now we can have Oracle read the passwd file when Oracle starts up.
This next step is required for the password file to be read during database
initilization.
oracle@oratest1# vi /oracle/product/10.1.0/Db_1/dbs/initdbname.ora
(or the equivalent initSID.ora file for your database)
un-comment the following entryin the init.ora file
REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE
Restart the database in normal mode.

oradba@rcvcat# sqlplus /nolog
SQL*Plus: Release 9.2.0.1.0 - Production on Wed Apr 23 21:33:25 2003
SQL> connect
/ as sysdba
SQL> shutdown immediate;

SQL> startup;
SQL> Exit
RMAN and Oracle backups

There are a few different ways to invoke RMAN backups with Avamar.
Invoke RMAN from the Avamar Administrator. Please review Appendix C: Avamar Administrator
Oracle GUI backups on page 35.
Invoke RMAN manually from the command interpreter (good for testing)
Invoke RMAN via the OS scheduler (cron) using RMAN cmdfiles. The Oracle 9i syntax is:
o oradba@targ1# rman cmdfile=fullbackup.rman
o oradba@targ1# rman @`fullbackup.rman` (the most popular option)
These commands can also be invoked by the OS scheduler (Cron).
Invoke RMAN via a stored script inside the recovery catalog.

o oradba@targ1#rman target backup/backup@payroll catalog backup/backup@rcvcat
o RMAN>run rmanscriptname
RMAN backup script broken down

Here is an example of a basic RMAN backup script. The Avamar-specific lines are in blue.
run
15
{
allocate channel T1 type 'SBT_TAPE'
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,
ENV=(PATH=/bin:/usr/bin)";
send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';
backup database;
release channel T1;
Avamar-specific parts in the RMAN script

This line helps RMAN find the libobk sbt library.
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,
ENV=(PATH=/bin:/usr/bin)";
This should be on a single line. The "ENV=(PATH=/bin:/usr/bin)" sets the environment variable so that the
Avamar script can find "avtar" and "uname".
This line helps with the proper Avamar Avtar flags.
Universal parts of the RMAN script

run
{
backup database;
release channel T1;
}
Plenty of other backup and recovery options can be added to this script and most DBAs will have their own
scripts already written. The only action needed is to copy the PARMS= line and the send line to their script.
The my-avtar-flags.txt file will also need to be created.
RMAN backup script execution and sample output

(These samples are performed on an Oracle 9i system with EMC NetWorker for sample purposes.)
oracle@legato1% rman cmdfile backup_database.rman
Recovery Manager: Release 9.2.0.1.0 - Production
RMAN> connect target rman/rman@eval
2> connect catalog rman/rman@rman
3> run {
4> allocate channel T1 type 'SBT_TAPE';
5> backup database plus archivelog delete input;
6> }
7>
connected to target database: EVAL (DBID=1928152189)
16
connected to recovery catalog database

allocated channel: T1
channel T1: sid=12 devtype=SBT_TAPE
channel T1: NMO v4.0.0.0
Starting backup at 23-APR-03
current log archived
channel T1: starting archive log backupset
channel T1: specifying archive log(s) in backup set
input archive log thread=1 sequence=6 recid=1 stamp=492128357
channel T1: starting piece 1 at 23-APR-03
channel T1: finished piece 1 at 23-APR-03
piece handle=04elai38_1_1 comment=API Version 2.0,MMS Version 4.0.0.0
channel T1: backup set complete, elapsed time: 00:00:25
channel T1: deleting archive log(s)
archive log filename=/space/home/oracle/product/9.2.0/dbs/arch1_6.dbf
recid=1 stamp=492128357
Finished backup at 23-APR-03
Starting backup at 23-APR-03
channel T1: starting full datafile backupset
channel T1: specifying datafile(s) in backupset
including current SPFILE in backupset
including current controlfile in backupset
input datafile fno=00001 name=/space/home/oracle/oradata/eval/system01.dbf
input datafile fno=00002 name=/space/home/oracle/oradata/eval/undotbs01.dbf
input datafile fno=00005 name=/space/home/oracle/oradata/eval/example01.dbf
input datafile fno=00011 name=/space/home/oracle/oradata/eval/test1a.dbf
input datafile fno=00012 name=/space/home/oracle/oradata/eval/test1b.dbf
channel T1: starting piece 1 at 23-APR-03
channel T1: finished piece 1 at 23-APR-03
piece handle=05elai44_1_1 comment=API Version 2.0,MMS Version 4.0.0.0
Document Number: 1-1033-01 NetWorker Module for Oracle 8i and 9i on Solaris
Evaluation Guide 15
channel T1: deleting archive log(s)
archive log filename=/space/home/oracle/product/9.2.0/dbs/arch1_7.dbf
recid=2 stamp=492128476
released channel: T1
Recovery Manager complete.
Avamar and RMAN backup example

The following is a sample RMAN script that will back up the entire database to Avamar, including the
archived redo logs.
Backup database and archived logs
This could be saved to a file called fullbkavmar.rman.
connect target backup/backup@evaldb
17
connect catalog backup/backup@rcvcat

run
{
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bi
n)";
backup database plus archivelog delete input;
release channel T1;
}
Then invoke the script like this.
oracle@test1# rman cmdfile fullbkavamar.rman
My-Avtar-Flags.txt explained
The following is a sample copy of the My-Avtar-Flags file with flag descriptions and values.
--debug
--pidname=Oracle
--pidnum=1002
--logfile=/usr/local/avamar/var/avtar.log
--vardir=/usr/local/avamar/var
--id=backup
--ap=backup
--path=/clients/vaerp-db4.va.company.org
Where --id is the username within the Avamar utility node.

If you specify "--id=backup"you are implicitly specifying "--id=backup@avamar/". The "avamar"
denotes that the "backup" user is in the default Avamar authentication system, rather than, say, in an
external authentication system. The "/" signifies that the "backup" user is a root-level user within the
Avamar application.
While it is acceptable to perform the initial tests with a root level user (in this case, "backup"), for
security reasons, it is best to define an account user that is specifically authorized to perform only
backups for that particular client account. In this case, that is the following:
"--id=backup@avamar/clients/clientname".
--ap is the password for the user.
--pidnum is a number for the OS of the target database.

Linux 1002
Solaris 2002
Windows 3002
HP-UX 4002
AIX 5002
--path is the Avamar domain and client of the target database.

/clients/clientname
Avamar and RMAN recovery scenarios for datafile recovery

Recovery scenarios are more detailed in Oracle due to the rules to recover. A couple of terms are important
to point out here.
18
RMAN Restore Command This command initiates the actual physical rebuild of the Oracle object
that you need. It is rebuilding the data from the Avamar Storage Node. This is the physical restore.
RMAN Recovery Command This command allows you to bring the database to a consistent point
in time. Depending on the options, you may need to apply the archived redo logs. This is the logical
recovery.
Recovery types in Oracle

It is important to discuss the different restore/recovery scenarios in Oracle. This may help you understand
the rules to recovery when approached by each scenario. You will want the DBA to help you understand
what type of recovery scenario is required for the Oracle system.
Using existing control and parameter files
If you have control files and Pfiles in place you are in good shape for most recovery scenarios. You will be
able to bring your database to a point in time or recover a datafile or tablespace. Following is an example
of how this would look in an Oracle RMAN script. (Most recoveries are done in this fashion.)
connect target backup/backup@targetdb
run
{
n)";
restore datafile '/d1/dev3data/usr01.dbf' ;
recover datafile '/d1/dev3data/usr01.dbf';
release channel T1;
}
NOTE: You might need to change the "--id" and "--ap" parameters in the "my-avtar-flags.txt" file to specify a user
that is authorized to perform restore operations.
Avamar and RMAN Oracle disaster recovery

With a complete system outage the control file must be restored/recovered before anything else can be
done. Start up the database in nomount mode then recover the control file. Then mount the database and
then restore/recover the database. The following is an example of how this would look in a RMAN script.
The DBA should be nearby for this test.
Note the resetlogs option here.
connect target backup/backup@targetdb
set DBID = 92755261
startup force nomount;
run
{
n)";
restore controlfile;
alter database mount;
recover database;
alter database open resetlogs;
release channel T1;
}
19
You will find various permutations of this exercise depending on what needs to be recovered and rolled
back. You can set the SCN to find a point in time to recover the database from.
It is also important to note that Oracle 9i and later have the auto control file backup option. If this is being
used the syntax to restore the control file is slightly different.
Oracle Open Reset Logs command

The previous example showed an example of the alter database open resetlogs. The example above reflects
an incomplete recovery scenario. This is basically a recovery of an Oracle object that may be out of sync
with the rest of the database. This could be due to the fact that you have lost an archive log or a user
caused database corruption.
If we take this down another level, the system change numbers (SCNs) of various objects are out of sync
with the rest of the database, and the resetlogs command should be used to synchronize the database.
The resetlogs command starts a new database logical life or a new set of SCNs. It records the beginning of
a new DB life and the old SCNs are maintained. The counter is not reset back to zero.
The old SCNs are left behind, which are sometimes referred to as incarnations of the Oracle system. Once
the database is opened with the resetlogs command, the entire database should be backed up immediately.
It is a difficult task to restore from a previous incarnation of the database.
Performance tuning Oracle backups

Identifying the client-side physical system bottleneck
It is apparent that when architecting the proper Oracle backup and recovery solution, there will be a some
bottlenecks within the Oracle system. This lends to a discussion around what exactly are the environmental
limitations that you will have to work with.
There are several variables that are set inside the database itself, including the System Global Area size.
This area is where the main RMAN I/O originates.
This tuning is typically the job of the DBA, and is not in the scope of this paper. A great reference book
that covers this tuning in detail is Oracle Database 10G RMAN Backup & Recovery by Matthew Hart and
Robert Freeman.
Physical bottleneck
The system in question here would be the Avamar backup client system that hosts the Oracle database. It
could be a physical or virtual machine with various hardware components.
Every system built will have a slowest component that translates into a bottleneck. The goal is to identify
the bottleneck to improve the efficiency of the backup and recovery jobs. Often, once you have addressed
one bottleneck, another bottleneck will quickly surface.
The process is outlined in the following sections. Table 1 shows some examples of physical system
bottlenecks. Your backup and recoveries will go only as fast as the system bottleneck. Every system will
have a physical bottleneck.
20
Table 1. Potential physical system bottlenecks

Operation type
Backup
Backup
Backup
Backup
Restores
Restores
Physical component
Memory
CPU
I/O Subsystem
Network (First Backup)
Network
I/O Subsystem
Avamar will process Oracle backups as fast as the memory and CPU will allow. Be aware that Avamars
initial first backup will appear to utilize more network and server resources as the effect of the
deduplication will not be fully realized. Subsequent backups will process faster and utilize less resources
as the local cache files and data store are populated.
During backups, the I/O subsystem read performance may be the bottleneck and will need to provide
exceptional read and seek performance.
EMCs experience at customer sites is that database backup performance ranges from approximately 60 to
100 GB/hour per backup stream. Performance may even be in excess of 200 GB/hour per stream. EMC
has, in numerous cases, observed a linear increase in backup performance as multiple parallel backup
streams are launched to back up the data.
Allocating multiple RMAN channels to boost the overall backup performance is definitely worth exploring
under the following conditions:
If the I/O subsystem is well architected and can support multiple concurrent backup streams that are
each capable of streaming backup data in excess of 100 GB per hour.
NOTE: "Well architected" in this case means that backing up the data through multiple streams should not create
contention in the I/O subsystem and therefore, slow down, rather than improve, the overall backup performance.
If the host has multiple processors available during the backup window. By default, the Avamar client
will distribute the SHA-1 hashing and compression out to all the processors on the client. However,
the main avtar thread is mapped to a single processor, and on some clients with slower individual
processors, this could be the bottleneck. In this case, allocating multiple RMAN channels creates
multiple avtar processes, each of which will have a main thread that is mapped to a different processor,
thus boosting overall performance.
As described in the next section "Avamar cache sizing for databases, the maximum amount of
memory that will be consumed by the Avamar process to optimally perform the Avamar backups of
the database data will be approximately 0.001 (1/1,000) of the total size of the database.
For example, to back up a 5 TB Oracle database, the maximum amount of memory consumed will be
approximately 5 GB total for all concurrent streams.
Although the examples later are based on using four RMAN channels to back up the Oracle database data,
the optimum number of required RMAN channels needs to be determined based on the following factors:
The optimum number of separate backup streams that can be supported by the I/O subsystem before
excessive contention slows down performance.
The CPU utilization will tend to continue to improve as more channels are added, until the number of
channels equals the number of processors.
21
As seen in the earlier formula, the amount of memory that needs to be allocated for the backup process
is independent of the number of channels that are used to perform the backup.
For reasons that will become apparent in the next section, enough RMAN channels will need to be
allocated so that the average amount of data per channel does not exceed 1.5 TB.
Based on current lab testing, the Avamar/Oracle recovery should go as fast as the network will allow.
This pertains to the client-side network interface, which in the testing to date has been a single GigE
interface. A second client-side network interface should help reduce this bottleneck.
Avamar cache sizing for databases

For more background information about the client cache files, please refer to the EMC Avamar Operational
Best Practices manual.
The hash cache file will allow you to tune your backups to run faster. They do not have any effect on
restores (p_cache.dat).
For database backups, having properly sized hash cache files (p_cache.dat files) is extremely important.
The hash cache stores the hashes of the chunks and composites that have been sent to the Avamar server.
The hash cache is used to quickly identify which chunks or composites have previously been backed up to
the Avamar server.
It is very important that the hash caches be sized appropriately. If the hash cache is not large enough, the
overall performance of the backups will slow down. This is because if the avtar process finds that a hash of
a chunk that is not contained in the hash cache, it queries the Avamar server for the presence of the hash.
These "lookups" of the hashes on the Avamar Grid require random seeks, which detract from the
performance of the Avamar Grid.
Conversely, if the hash caches are oversized, then the hash caches, which are loaded into the client's
memory at the beginning of the backup, will consume too much memory. This, in turn, could cause
swapping on the client, which slows down the backup performance, or could even cause the backups to fail
because of memory allocation failures.
NOTE: Properly sizing the hash caches is particularly important when backing up Oracle databases using multiple
RMAN channels. If the hash caches are undersized, then the combined effect of multiple avtar processes querying
the Avamar Grid for the presence of hashes of chunks that were previously backed up could cause the Avamar
Grid to become the bottleneck. This slows down overall performance. On the other hand, if we over allocate
memory for the hash caches, then the client could run out of memory resources to support all concurrent avtar
streams.
Rules for sizing hash caches properly

There are a few rules that you need to know to size the hash caches appropriately:
1) By default, the hash cache could consume up to 1/16th of the physical RAM on the Avamar client.
2) If the hash cache files already exist and are larger than the required size, the cache files
(p_cache.dat) will need to be deleted and re-created to be resized to a smaller size.
3) The hash cache doubles in size each time it needs to grow. The current hash cache sizes are 24 MB,
48 MB, 96 MB, 192 MB, 384 MB, 768 MB, and 1,536 MB. At this time, the maximum hash cache
size that is supported is 1,536 MB.
4) The hash cache includes only one 20-byte SHA-1 hash per chunk, which is the hash of the contents
of the chunk, plus another 4 bytes of overhead per chunk. Therefore, each entry in the hash cache
consumes 24 bytes per entry.
5) During Oracle backups, the average chunk size (before compression) is 24 KB per chunk.
6) To control the size of the hash cache used for a dataset, you can specify the "--hashcachemax=<>"
option. The value to which you set this attribute is the "not to exceed" size of the hash cache file in
22
MBs. Therefore, for example, setting "--hashcachemax=500" will allow the hash cache file to grow to
384 MB (because 384 MB is the largest hash cache file less than 500 MB).
From rules (3), (4), (5), and (6), we can derive the following rule for setting the "hashcachemax" option:
--hashcachemax = 2 x MB <Size of database in GB>
So if the database is 700 GB the formula should look like this.
Hash Cache Max should be less than 2 x 700 MB (Converted GB to MB) = 1400 MB
Examples of properly sized hash cache files

Here are some examples of how to properly specify the "hashcachemax" option and the resulting size of the
hash cache:
Table 2. Sample hash cache sizes based on stream size and hashcachemax
Stream size
Hashcachemax
Hash cache size
10 GB DB
Stream
--hashcachemax=20
24 MB
100 GB DB
Stream
--hashcachemax= 200
192 MB
500 GB DB
Stream
--hashcachemax= 1000
768 MB
700 GB DB
Stream
--hashcachemax=1400
768 MB
1000 GB DB
Stream
--hashcachemax=2000
1536 MB
1536 GB DB
Stream
--hashcachemax=3072
1536 MB
So you can see as we get to the larger dataset size the maximum hash cache will be likely 1,536, which is
the largest it can possibly get today. A dataset over 2 TB that uses a single hash cache will likely slow
down because it does not have a large enough hash cache. This is where the multiple stream discussion
becomes relevant.
You can specify the "--hashcachemax=<>" parameter in the my-avtar-flags file.
You will have to rename the original p_cache.dat file and your next backup will build a new one.
When you allocate multiple RMAN channels to back up the Oracle database, each channel will process
roughly the same amount of data. This means that you can specify the same "--hashcachemax=<2 x
amount of data per RMAN channel (in GB)>" in the my-avtar-flags.txt file.
When backing up large databases (greater than 2 TB databases), it might be important to minimize the
amount of memory consumed on the host. In this case, the amount of data backed up per RMAN channel
should be ~750 GB or ~1.5 TB. This ensures that we can effectively utilize a 768 MB or 1,536 MB hash
cache file with very little wasted space in the hash cache file.
It is also important to understand that if your cache files are close to these values and not exactly 100
percent the same, you should be safe. You should be aware of major deviations from the rules.
The case for multiple RMAN channels

When the system bottleneck has been identified, you can test a single channel RMAN script for the backup
and recovery of the database. It appears that a single RMAN stream will run around 60-100 GB per hour.
This may be adequate to satisfy the backup and recovery time objectives for your database.
The issue here is that the larger databases do require the backup and restores to run at faster speeds to
satisfy the backup and recovery time objectives.
For example a 2 TB database will require at least 21 hours to restore if using a single stream. In the same
context a 4 TB database would require at least 42 hours.
23
This leads to the case for allocating multiple channels, which is equivalent to generating multiple streams
or multiple Avtar processes. This is definitely needed when it comes to a restore of a larger database.
Example scripts and multiple Avamar cache files

When architecting an Oracle backup and recovery that will use multiple RMAN channels in Avamar, use
multiple sets of Avamar cache files. This is similar to the way multiple cache files are invoked for multiple
Avtar processes for very dense file system backups. As a matter of fact, we will allocate multiple Avtar
processes in the end. This is simplified by using the Avamar "- - cacheprefix" flag. This is an actual Avtar
flag. Read more about the "--cacheprefix" flag in EMC Avamar Operational Best Practices.
The good news here is by using this option there is not a lot of heavy lifting or changes that need to be
made to allow for multiple cache files.
The actual cache files sit in the /usr/local/avamar/var directory. Of course, the cache files are f_cache and
p_cache. To be more specific the cache files actually sit in the /var/avamar directory. They are referenced
by a symbolic link in the /usr/local/avamar directory.
Multiple channel RMAN Avamar backup script

run {
configure controlfile autobackup on;
set controlfile autobackup format for device type sbt to "CONTROLFILE.%F";
allocate channel c1 type sbt
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so
ENV=(PATH=/bin:/usr/local/avamar/bin)";
send channel='c1' '"--flagfile=/scripts/my-avtar-flags.txt" "--cacheprefix=c1"
"--logfile=/usr/local/avamar/var/c1_avoracle.log"';
backup database plus archivelog;
release channel c1;
release channel c2;
release channel c3;
release channel c4;
The blue text outlines the unique parts of this script. This is essentially how to address the multiple
channels and map them to the multiple cache files. The cacheprefix=c(n) creates a unique set of cache
files for that channel. There are four in this example.
The logfile= creates a unique set of log files for each channel.
All of these files are stored in the standard /usr/local/avamar/var directory.
24
Multiple channel RMAN restore script

run {
send channel='c1' '"--flagfile=/scripts/my-avtar-flags.txt" "-logfile=/usr/local/avamar/var/c1_avoracle.log"';
restore database;
recover database;
release channel c1;
release channel c2;
release channel c3;
release channel c4;
}
This script is used for the restore and looks very much like the backup script. Always make sure the DBA is
involved with the restore and recovery of an Oracle database. There are many restore scenarios and the
DBA will need to be aware of the state of the database before you begin.
The main concept to gather from this example is that the recovery will come back in multiple streams. This
is just how the backup was performed.
Multiple channel Avamar My-Avtar-Flags file

--hashcachemax=<2 x amount of data per RMAN channel (in GB)>
--expires=30
--server=avamar220.lss.emc.com
--pidname=Oracle
--pidnum=1002
--id=backup
--ap=backup
--path=/Oracle/avamar161.lss.emc.com
This looks similar to earlier examples. The hashcachemax flag was added to accommodate the large
database files. The rest is standard.
By using the cache prefix option, only one my-avtar-flags file is needed. Each channel reads this file.
25
Sample outputs
Figure 1. Prebackup
Figure 2. Start backup
Figure 3. Backup output multiple channels
26
Figure 4. Check for multiple avtars

The first channel completes fast because it handles the archived redo logs and the control file backups.
Figure 5 shows how to see the progress in the Avamar GUI.
Figure 5. Avamar GUI
27
Conclusion
As can be seen by examples in this white paper, Avamar can be used as a viable solution for larger database
environments. By using multiple RMAN channels you can easily achieve respectable backup and recovery
time objectives.
28
Appendix A: Avamar Oracle plug-in recovery catalog

creation
Create the catalog database instance in Oracle 9i
RMAN tracks backed up initialization files, control files, datafiles, and logs using the RMAN repository.
The RMAN repository facilitates automated identification of files required for a given backup or restore
operation.
The RMAN repository can be written to the target database control file by running RMAN in no catalog
mode or to a separate database instance known as the catalog database. A catalog database is recommended
for the RMAN repository if you are backing up multiple database instances or if a single database instance
has more than 20 datafiles.
If you choose to run RMAN in no catalog mode, it is critical that the control file is backed up frequently
and regularly since the backup repository is in the control file. If the control file is lost or damaged, it must
be recovered from a non-RMAN backup before a recovery of the database is possible. The use of RMAN in
no catalog mode is not shown in the examples. This, of course, changes in Oracle 9i with the control file
Autobackup option.
If you choose to run RMAN with a backup repository in a separate database instance, it is recommended
that the catalog database be on a separate server from any target database that is registered with that catalog
database.
The examples provided use a target database called eval and a catalog database called rman.
Create a Oracle database in Oracle 9i

1.
Log in to the Oracle server as oracle.
SunOS 5.8
login: oracle
Password:
Last login: Wed Jan 21 08:06:31 from legato1
Sun Microsystems Inc. SunOS 5.8 Generic February 2000
Test1%
2.
Start the Oracle Database Configuration Assistant.
oracle@test1% setenv ORACLE_SID rman

oracle@test1% /space/home/oracle/product/9.2.0/bin/dbca &
3.
Select Next on the Welcome screen.
4.
Select Create a database and click Next.
5.
Select General Purpose on the Database Templates screen and click Next.
6.
Enter the Database Identification information and click Finish. Example:
Global Database Name: rman.legato.com

SID: rman
7.
Confirm the database creation by clicking OK on the Summary screen.
29
8.
Enter password information on the Oracle Database Configuration Assistant screen and click Exit.
Example:
SYS password: backup

Confirm SYS password: backup
SYSTEM password: backup
Confirm SYSTEM password: backup
Create catalog users

Create the catalog schema
Create a catalog database user
A catalog database user for use by the RMAN facility must be created in the catalog database instance. The
user in this example is called backup. This user will require connect, resource, and recovery_catalog_owner
privileges. It is also recommended to grant this user sysdba privileges. See the Create users in the recovery
catalog section for examples.
Create the recovery catalog
Connect as the new user to the recovery catalog database and populate the recovery catalog tablespace.
Oracle 9.x
oracle@test1% rman target backup/backup@eval catalog backup/backup@rman
RMAN> create catalog
recovery catalog created
RMAN> exit
Recovery Manager complete.
Register the target database

A target database must be registered with the catalog database before RMAN can perform any backup
operations for the target. Execute the following commands to register the target database:
oracle@legato1% rman target backup/backup@eval catalog backup/backup@rman
RMAN> register database;
database registered in recovery catalog
starting full resync of recovery catalog
full resync complete
RMAN> exit Recovery Manager complete.
oracle@test1%
RMAN> resync catalog;
If you have not executed the register database command prior to executing the
backup, restore, or resync catalog commands you will get the following error:
RMAN-20001: target database not found in recovery catalog
30
Appendix B: Avamar Oracle plug-in backup and

recovery scripts
This section shows examples of scripts used for various backup and recovery scenarios.
Back up all and delete the logs
run
{
n)";
release channel T1;
}
Crosscheck archive logs

run
{
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/b
in)";
crosscheck archivelog all ;
release channel T1;
}
Back up the datafile

{
in)";
backup datafile '/d1/dev3data/usr01.dbf' ;
release channel T1;
Recover the datafile

run
{
in)";
restore datafile '/d1/dev3data/usr01.dbf' ;
31
recover datafile '/d1/dev3data/usr01.dbf';

release channel T1;
}
Back up the tablespace

run {
PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:
/usr/bin)";
backup tablespace DEMO;
release channel T1;
}
Recover the tablespace

run {
/usr/bin)";
restore tablespace DEMO;
recover tablespace DEMO;
sql "alter tablespace DEMO online";
release channel T1;
}
Restore archive log all

run
{
in)";
restore archivelog all ;
release channel T1;
}
Disaster recovery using Autobacked up control files

set DBID = 92755261
32
run {
/usr/bin)";
#restore spfile;
restore spfile from autobackup;
release channel T1;
}
shutdown immediate;
startup nomount;
Disaster recovery restoring a control file

run {
/usr/bin)";
#restore controlfile;
restore controlfile from autobackup;
restore database;
recover database;
release channel T1;
}
Simpler DR example of incomplete recovery

connect target /
set DBID = 92755261
run {
/usr/bin)";
#restore controlfile;
restore controlfile from autobackup;
recover database;
alter database open resetlogs;
}
Perform cold backup using RMAN

run {
shutdown immediate;
startup mount;
33

in)";
backup
(DATABASE format 'df_t%t_s%s_s%p');
release channel t1;
sql Alter Database Open;
}
Oracle on Windows example

run {
PARMS="SBT_LIBRARY=C:\PROGRA~1\avs\bin\orasbt.dll,ENV=(PATH=C:\PROGRA~1\avs\bin
)" format '%d_%U';
set controlfile autobackup format for device type sbt to
"CONTROLFILE.DUBLIN.%F";
send '"--flagfile=C:\TEMP\FlagFile.txt" --debug';
}
Notice the short pathname for SBT_LIBRARY=C:\PROGRA~1\avs\bin\orasbt.dl

Flagfile located at C:\TEMP\FlagFile.txt:
--debug
--pidname=Oracle
--pidnum=3002
--logfile=C:\TEMP\avtar.log
--vardir="C:\Program Files\avs\var"
--id=root
--ap=8RttoTriz
--path=/clients/vm-qasys-02
34
Appendix C: Avamar Administrator Oracle GUI backups

Snapup options
The options in defined Figure 6 can be specified during on-demand Oracle database backup operations, or
within the datasets. They are explained below.
Figure 6. Options for on-demand Oracle database backup
Oracle Instance Name specifies the Oracle server name. This field can normally be left blank because
the Oracle instance name is automatically stored when the target client is browsed during on-demand
snapup or when a dataset is applied to a member of a group during a scheduled snapup.
User name specifies the username used to authenticate with the Oracle database. If this field is left
blank, RMAN tries to log in with the same username and password that the Avamar client agent is
running under, and attempts to assume SYSDBA privileges. Typically, this field should contain the
special account name (backupuser) manually created as part of database preparation. Username and
Password comprise a connection string to Oracle. The connection string must specify a user who has
backup privileges for the database.
Password specifies the password for the username account.
The Delete Archive Logs after backup checkbox, if selected, automatically deletes Oracle archive
logs immediately following a successful database backup.
Enhanced Commonality Factoring enables or disables enhanced data deduplication. During snapups,
enhanced data deduplication typically reduces the amount of client data that must be sent to the server
but requires additional client CPU resources. Choices are:
Disabled
Default (use the global enhanced data deduplication setting already set on the server)
Enabled
The Enable debugging messages checkbox, if selected, adds all available debugging output to both
avtar and avoracle. This is useful when submitting a problem to EMC Technical Support so that an
issue can be more easily diagnosed.
The Use Recovery Catalog checkbox is followed by three recovery catalog fields. The Recovery
Catalog Server Name, Recovery Catalog Username and Recovery Catalog Password settings are used
to specify a Recovery Catalog Server connection string for RMAN. By using a Recovery Catalog
Server, many specialized features of RMAN may be used. A thorough discussion of these features is
beyond the scope of this paper.
35
Recovery Catalog Server Name specifies the recovery catalog server name.
Recovery Catalog Username the specifies the recovery catalog username.
Recovery Catalog password specifies the recovery catalog password.
Restore options
The options in Figure 7 can be specified during the restore operations. They are explained below.
Figure 7. Options for restore operations
Oracle Instance Name specifies the Oracle server name. This field can normally be left blank because
the Oracle instance name is automatically stored when the target client is browsed during restore
operations
User name specifies the username used to authenticate with the Oracle database. If the field is left
blank, RMAN tries to log in with the same username and password the Avamar client agent is running
under, and attempts to assume SYSDBA privileges. Typically, this field should contain the special
account name (backupuser) manually created as part of database preparation. Username and Password
comprise a connection string to Oracle. The connection string must specify a user that has backup
privileges for the database.
Password specifies the password for the username account.
Recovery Until SCN is for the recovery portion, so you can play back archive log files until the
transaction with this SCN is reached. If this field is left blank, avoracle will scan through the known
backups in the backup catalog (either in the system control file or the recovery catalog) and find the
highest SCN associated with a backup file.
The Enable debugging messages checkbox, if selected, adds all available debugging output to both
avtar and avoracle. This is useful when submitting a problem to EMC Technical Support so that an
issue can be more easily diagnosed.
The Only restore control file checkbox, if selected, restores only the control file, and known backups
in the backup catalog (either in the system control file or the recovery catalog) are listed. This is useful
when the user wants to find an SCN for a database or archive log without actually performing the
restore.
36
Restore Command Options is for additional command line options that will be passed to the RMAN
restore command.
Recover Command Options is for additional command line options that will be passed to the RMAN
recover command.
For descriptions of Use Recovery Catalog, Recovery Catalog Server Name, Recovery Catalog
Username, and Recovery Catalog Password, see the Snapup options section.
Troubleshooting GUI operations

If you are encountering issues with browsing your database(s) within the GUI, this will help you identify
what the issue is.
Avoracle looks at the oratab file for database names and then cross-references those names to the tnsnames
file. Local databases that resolve this way are then presented in the browser window for selection for
backups.
Avoracle defaults to the $ORACLE_HOME/network/admin directory. So, make sure the tnsnames file is in
the $ORACLE_HOME/network/admin directory.
If these instructions do not resolve the browse issue, please do the following:
1.
Create the file /opt/AVMRclnt/var/avoracle.cmd
2.
Add the following two lines to the file:
--debug
--logfile=/opt/AVMRclnt/var/avoracle_browse.log
3.
Try to browse again. The browse attempt will be logged in avoracle_browse.log.
4.
Look in the avoracle_browse.log file and you will be able to identify the error you have and take
the necessary steps to correct it.
Appendix D: Avamar and Oracle theory of operations

Avoracle theory of operations
Avoracle is normally invoked whenever any operations on the Oracle database are carried out through the
Avamar Administrator. The three operations supported by Avoracle are:
Browse
Backups
Restores
While the functionality of Backup and Restore should be fairly obvious, Browse is an intermediate and not
so visible operation.
When an Avamar Oracle operation is selected in Avamar Administrator, it sends a message to the
concerned client node. This message is called a workorder, and it contains all the information a client needs
to perform an operation.
Communication between the Avamar Administrator and the client node is always initiated through
Avagent. Avagent runs as a daemon (service) and listens for messages from the Administrator.
37
Avagent inspects the workorder received, and uses the information to decide which of the Avamar clients
needs to be invoked. If the operation is meant for the Oracle plug-in, Avagent starts Avoracle and passes it
the workorder.
Avoracle operations
Browse
The first step in starting a backup of an Oracle database is the Browse operation.
When the Oracle plug-in is selected in the Backup panel of Avamar Administrator, it sends a Browse
request to Avoracle (through Avagent). This triggers a database discovery process in Avoracle.
Avoracle reads the /etc/oratab to get a list of databases on the system and their home directories. Using the
Oracle home directories it then looks for the tnsnames.ora file.
The special fields of interest in tnsnames.ora are HOST and SERVICE. Using this, Avoracle generates a
list of databases that are local to the system. This list is then returned to Avamar Administrator in a
message. Avamar Administrator then uses this list to populate the Backup panel.
Backup
When one or more databases are selected for backup, a new message is sent to Avoracle. This message
contains the list of databases to be backed up, together with any options that the user specifies. The
message also contains credentials that Avoracle can use to communicate with the Avamar Server.
In the current implementation Avoracle can only do backups and restores sequentially. So, even if multiple
databases have been specified, they are done one by one. The following description is for one database. For
multiple databases the sequence is just repeated.
Avoracle uses RMAN and Oracles SBT feature for taking backups. For this Avoracle specifies a loadable
library that can communicate with the Avamar Server. This library implements certain entry points that
have been written to Oracles specification. The library path and some other parameters are communicated
to RMAN through a script that Avoracle creates.
In Oracle 10g and later, Oracle has a restriction that prohibits RMAN from running as root. Avoracle
scripts are run as the default user oracle unless an alternate user id has been specified.
The RMAN script, credentials, and logfiles are created and stored in the Avamar var directory. Then
avoracle creates another process and starts RMAN, telling it what script to use. Avoracle then goes into a
loop waiting for a connection from avtar (more on this later) or for RMAN to complete its operations and
exit.
Once RMAN starts, it communicates with Oracle and tells it which library needs to be loaded. Oracle loads
this library (which is actually an Avoracle component) and uses it for backup. After this point the backup
process is controlled by RMAN. RMAN, by default, backs up archive logs, the database file, and the
control file. Based on certain factors RMAN splits up the entire backup into one of more backup pieces.
For each backup piece (which is actually opaque to Avoracle), RMAN begins by communicating the piece
name to libobk. It then performs the write using multiple calls to libobk, ending the operation with a close
command.
When libobk receives the backup command it creates an avtar process. The name of the backup piece is
communicated to avtar together with some other information.
38
Avtar is also told how to connect with the already running Avoracle. This last step is required so that avtar
can get the workorder from Avoracle and be able to send it status updates.
Avtar establishes connection with the Avamar Server and then waits for data from libobk. Every time the
write entry point of libobk is called by Oracle, libobk takes the data and send it to avtar.
Avtar will process the data (chunking and hashing) and store it on the server. The backup piece is stored on
the server with the same name that RMAN had generated. This makes it easier to find the right file during
restores. Once all the data for a backup piece has been received avtar terminates and libobk returns control
to RMAN.
After the entire database has been backed, RMAN unloads libobk and exits, returning control to Avoracle.
Restores
The Restore process is similar to the Backup process except for the direction of data flow. The Restore
browse panel is populated by the Avamar Administrator and it lets the user select one or more backups to
be restored.
Avoracle receives this list and does preprocessing on it to extract the control file names. For every control
file that has been selected for restore the following happens.
Avoracle first creates an RMAN script for restoring just the control file. It starts RMAN with the script
(everything else is the same as the backup operation). RMAN loads libobk and restores the control file. If
the user has not specified the SCN for restore, Avoracle examines the RMAN logfile and extracts the latest
SCN corresponding to the control file selected.
Avoracle then creates a new script for recovery of the database files. At this point RMAN has mounted the
database using the control files. Once again RMAN is started up with the proper script, and libobk is
loaded. Based on the SCN specified and the backup history in the control file, RMAN decides which
backup pieces need to be restored. These piece names are communicated to libobk, which in turn talks to
avtar. Avtar gets these files from the Avamar Server and the data is sent back to RMAN through the
corresponding entry point in libobk. Once all the required files have been restored, RMAN does a recover
operation, opens the database, and exits.
Following is a graphical representation of the entire process.
Figure 8. Restore process

39
Appendix E: Troubleshooting, hints, and tips

The following are some important factors for diagnosing the Media Management Library:
For 64-bit Oracle vs. a 32-bit library, verify the "bitness" is the same.
Must pass the correct PATH, such that RMAN can initiate "avtar" and "uname".
Must set the right environment variable such that the Avamar library can be located and loaded.
Different platforms (Solaris, HP-UX, Linux, AIX, and Windows) sometimes use different environment
variable names for the library.
Different shells (cmd, bash, sh, csh) use different syntax for setting and exporting environment
variables.
Avamar software, config, and log location (--vardir, Library, logs) vary by platform.
The PATH and ENVIRONMENT are important because of the complex process structure involved:
1.
Avagent receives a workorder, and invokes Avoracle.
2.
Avoracle invokes (via script) "rman".
3.
RMAN invokes "oracle".
4.
Oracle invokes "libok_avamar" (Media Management Library).
5.
The library invokes avtar.
The first step can be short-cut, by invoking "rman" directly from a command line. However, the PATH and
ENVIRONMENT are still critical.
Diagnostic commands: A sample session

Locate avtar and uname. Their location will need to be added to the PATH.
which avtar
/usr/bin/avtar
which uname
/usr/bin/uname
Ensure the flag file is set up correctly.
cat
/opt/AVMRclnt/bin/my_avtar_flags.txt
--debug
--pidname=Oracle
--pidnum=2002
40
--id=backup
--ap=backup
--path=/clients/ux210.fairisaac.com
Different Var directories depending on operating system

Note that on a Solaris platform, --vardir and --logfile must point to /opt/AVMRclnt/var, and "c:\program
files\avs\var" or the equivalent for Windows. The /usr/local/avamar/var is used in the case of Linux
systems.
Ensure that the Media Management Library exists and has the expected size, bitness, and version. (The
following commands are for Solaris64.)
ls -la /opt/AVMRclnt/lib/
total 5684
drwxr-xr-x
2 root
root
512 Jan 23 10:47 ./
drwxr-xr-x
5 root
root
512 Jan 23 09:06 ../
-rwxr-xr-x
1 root
root
-rwxr-xr-x
1 root
root
55556 Sep 13 13:47 libobk_avamar.so*
-rwxr-xr-x
1 root
root
66280 Sep 13 13:47 libobk_avamar64.so*
-rwxr-xr-x
1 root
root
2232464 Sep 13 13:45 libstdc++.so.2.10.0*
528320 Sep 13 13:47 libgcc_s.so.1*
file /opt/AVMRclnt/lib/libobk_avamar64.so
41
/opt/AVMRclnt/lib/libobk_avamar64.so: ELF 64-bit MSB dynamic lib SPARCV9

Version 1, dynamically linked, not stripped
file /usr/oracle/product/9.2.0.8.64bit/bin/oracle
/usr/oracle/product/9.2.0.8.64bit/bin/oracle: ELF 64-bit MSB executable
SPARCV9 Version 1, dynamically linked, not stripped
ls -l $ORACLE_HOME/lib/libok.so
/usr/oracle/product/9.2.0.8.64bit/lib/libok.so: No such file or directory
Try loading the library using the oracle "SBTTEST" (again for Solaris64, use bash). The library path and
command to set the appropriate variable have the following variations:
Table 3. Operating system and environment variable name
OS
Env Var Name
Linux, Solaris LD_LIBRARY_PATH

HPUX
SHLIB_PATH
AIX
LIBPATH
The following are the commands to set the variable changes, depending on the shell:
bash, sh
LD_LIBRARY_PATH="$LD_LIBRARY_PATH":/opt/AVMRclnt/lib
export LD_LIBRARY_PATH
csh
setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":/opt/AVMRclnt/lib
echo Test file contents > testfile
set env LD_LIBRARY_PATH=/opt/AVMRclnt/lib
echo $LD_LIBRARY_PATH
/usr/oracle/product/9.2.0.8.64bit/lib:/usr/lib:/usr/dt/lib:/usr/openv/netbackup
/bin
set env LD_LIBRARY_PATH=/opt/AVMRclnt/lib
42
/usr/oracle/product/9.2.0.8.64bit/lib:/usr/lib:/usr/dt/lib:/usr/openv/netbackup
/bin
sbttest testfile -libname /opt/AVMRclnt/lib/libobk_avamar64.so
/opt/AVMRclnt/lib/libobk_avamar64.so could not be loaded.
Check that it is installed properly, and that the LD_LIBRARY_PATH environment variable (or its
equivalent on your platform) includes the directory where this file can be found. Additional error
information is listed below:
ld.so.1: sbttest: fatal: libgcc_s.so.1: open failed: No such file or

directory
Retry, with the right environment variable:
export LD_LIBRARY_PATH=/opt/AVMRclnt/lib
/opt/AVMRclnt/lib
sbttest testfile -libname /opt/AVMRclnt/lib/libobk_avamar64.so
The sbt function pointers are loaded from /opt/AVMRclnt/lib/libobk_avamar64.so library.
Avamar
: sbtinit V1
Init: Proxy disabled
-- sbtinit succeeded
Avamar
: sbtinit V1
43
-- sbtinit (2nd time) succeeded
sbtinit: Media manager supports SBT API version 2.0
sbtinit: vendor description string=EMC|Avamar (avtar backup)
sbtinit: allocated sbt context area of 4672 bytes
sbtinit: Media manager is version 4.0.0.37
Avamar
: sbtinit2
-- sbtinit2 succeeded
-- regular_backup_restore starts ................................
Avamar
: sbtbackup
: file=testfile
: block_size=16384
Child process started
Avamar
: Waiting for avtar to start ... sleep 10
avtar Warning <0000>: Cannot create log /opt/AVMRclnt/var/avtar-20090128_07471000001.log
44
avtar Info <5551>: Command Line: /opt/AVMRclnt/bin/avtar.bin --vardir=/opt/AVMRclnt/var

--bindir=/opt/AVMRclnt/bin
--sysdir=/opt/AVMRclnt/etc --label=testfile -c
testfile --from-stdin --check-stdin-
path=false
avtar FATAL <5086>: No server address specified.
Reinstall software or add --server=
address.
Avamar
: avtar exited
MMAPI error from sbtbackup: 7501, sbtbackup: avtar exited, trying to create testfile
-- sbtbackup failed
rm testfile
Set LD_LIBRARY_PATH (Or equivalent on HPUX/AIX)
export LD_LIBRARY_PATH=/opt/AVMRclnt/lib
Then re-running the RMAN command.
rman
Recovery Manager: Release 9.2.0.8.0 - 64bit Production
Copyright (c) 1995, 2002, Oracle Corporation.
All rights reserved.
45
RMAN> connect target /
connected to target database: IVVINT0 (DBID=3840860340)
RMAN> connect catalog rman920/rman920@rmnp0
RMAN> run
PARMS="SBT_LIBRARY=/opt/AVMRclnt/lib/libobk_avamar64.so,ENV=(PATH=/bin:/usr/bin:/opt/AVMR
clnt/bin)";
send '"--flagfile=/opt/AVMRclnt/bin/my_avtar_flags.txt"';
release channel T1;
Avamar
: sbtinit V1
trace_file=/usr/oracle/admin/ivvint0/udump/sbtio.log
Avamar
: sbtinit2
allocated channel: T1
46
channel T1: sid=17 devtype=SBT_TAPE
channel T1: EMC|Avamar (avtar backup)
Avamar
: sbtcommand = "--flagfile=/opt/AVMRclnt/bin/my_avtar_flags.txt"
sent command to channel: T1
Starting backup at 2009-01-28:07:36:31
current log archived
channel T1: starting archive log backupset
channel T1: specifying archive log(s) in backup set
input archive log thread=1 sequence=1931 recid=1922 stamp=676951918
--Snip--
channel T1: starting piece 1 at 2009-01-28:07:36:35
Avamar
: sbtbackup
: file=83k5u2c2_1_1
: block_size=262144
Child process started
Avamar
: Waiting for avtar to start ... sleep 10
--Snip--
47
avtar Info <5551>: Command Line: /opt/AVMRclnt/bin/avtar.bin --vardir=/opt/AVMRclnt/var -bindir=/opt/AVMRclnt/bin
--sysdir=/opt/AVMRclnt/etc --flagfile=/opt/AVMRclnt/bin/my_avtar_flags.txt -debug --pidname=Oracle --pidnum=2002
--logfile=/usr/local/avamar/var/avtar.log --vardir=/usr/local/avamar/var -id=backup --password=******
--account=/clients/ux210.fairisaac.com --label=83k5u2c2_1_1 -c 83k5u2c2_1_1 -from-stdin --check-stdin-path=false
2009/01/28-13:36:35.85013 [avtar]
sock::libinit(enc=, ciphers=, verify=0)
socktype=sock_tcp
-- Snip -Recovery Manager complete
48

h6540 Oracle Backup Recovery Perform Avamar Rman WP

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

h6540 Oracle Backup Recovery Perform Avamar Rman WP

Uploaded by

Copyright:

Available Formats

Oracle Backup, Recovery, and Performance

Tuning using EMC Avamar and Oracle RMAN

Copyright 2009 EMC Corporation. All rights reserved.

Benefits of using EMC Avamar.......................................................................... 5

Recovery catalog ................................................................................................ 9

Target database preparation............................................................................ 10

Archive Log Mode............................................................................................. 13

RMAN and Oracle backups .............................................................................. 15

My-Avtar-Flags.txt explained ........................................................................... 18

Performance tuning Oracle backups .............................................................. 20

Avamar cache sizing for databases ................................................................ 22

The case for multiple RMAN channels............................................................ 23

Sample outputs ................................................................................................. 26

Appendix B: Avamar Oracle plug-in backup and recovery scripts .............. 31

Appendix D: Avamar and Oracle theory of operations.................................. 37

Appendix E: Troubleshooting, hints, and tips................................................ 40

Benefits of using EMC Avamar

Avamar and large databases

Database size and version?

Available CPU and memory?

What type of disk subsystem is available?

What type of Avamar environment is in place and how large is it?

Key features of using RMAN with the Avamar Oracle

Online backups Full and Incremental

Offline backups Full and Incremental

Archived redo log backups

Dedicated Oracle database to track backups recovery catalog

Oracle block-level corruption detection

Oracle database objects explained

System Global Area (SGA)

Oracle control files

Oracle online redo logs

Oracle archived redo logs

Oracle parameter file (init(SID).ora or pfile)

Other Oracle objects

Tablespace: Mentioned previously, it is a logical grouping of datafiles.

Index: Allows you to be more efficient with queries.

Constraints: A DBA term for rules with data.

Oracle hot or cold backups

Offline (cold) backups

Target database preparation

Create an Oracle password file on the target database.

Install the Avamar operating system file system agent.

Install the SBT library on the target database

Example of the installation on Red Hat.

You can then log in to the database with SQL plus.

Create users on the Avamar Grid, target database, and recovery

Create a user called backup in the Avamar GUI

Create a user in the target DB

SQL> alter user backup identified by backup; Oracle 9

Here is a more elaborate example of the preceding tablespace declaration.

SQL> create user username identified by password default

Create users in the recovery catalog

Here is a more elaborate example of the preceding tablespace declaration.

SQL> alter user backup account unlock; Oracle 9i and above.

SQL> grant connect,resource,recovery_catalog_owner to backup;

Archive Log Mode

SQL> shutdown immediate

(This is like single user mode in UNIX.)

SQL> alter database archivelog;

SQL> archive log list

Oracle password file explained

Create your Oracle password file