DETAILED ADDM REPORT FOR TASK 'TASK_324' WITH ID 324

----------------------------------------------------
Analysis Period: from 04-DEC-2010 20:53 to 06-DEC-2010 20:51
Database ID/Instance: 2025606740/1
Database/Instance Names: TEST/test
Host Name: es.kuttipaiyan.com
Database Version: 10.2.0.1.0
Snapshot Range: from 56 to 63
Database Time: 488 seconds
Average Database Load: 0 active sessions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

FINDING 1: 46% impact (225 seconds)

-----------------------------------
RMAN I/O activity had a significant impact on the throughput of the I/O
subsystem.
RECOMMENDATION 1: DB Configuration, 46% benefit (225 seconds)
ACTION: Consider slowing down RMAN activity, or scheduling RMAN jobs
when user activity is lower.
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: The throughput of the I/O subsystem was significantly lower
than expected. (12% impact [57 seconds])
SYMPTOM: Wait class "User I/O" was consuming significant database
time. (46% impact [225 seconds])
FINDING 2: 33% impact (159 seconds)
-----------------------------------
Undo I/O was a significant portion (71%) of the total database I/O.
NO RECOMMENDATIONS AVAILABLE
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: The throughput of the I/O subsystem was significantly lower
than expected. (12% impact [57 seconds])
SYMPTOM: Wait class "User I/O" was consuming significant database
time. (46% impact [225 seconds])
FINDING 3: 28% impact (137 seconds)
-----------------------------------
Individual database segments responsible for significant user I/O wait were
found.
RECOMMENDATION 1: Segment Tuning, 18% benefit (88 seconds)
ACTION: Investigate application logic involving I/O on database object
with id 10706.
RELEVANT OBJECT: database object with id 10706
RATIONALE: The SQL statement with SQL_ID "3xpsk5cub3nvc" spent
significant time waiting for User I/O on the hot object.
RELEVANT OBJECT: SQL statement with SQL_ID 3xpsk5cub3nvc

RECOMMENDATION 2: Segment Tuning, 10% benefit (49 seconds)

ACTION: Run "Segment Advisor" on TABLE "SCOTT.TEST2" with object id
10376.
RELEVANT OBJECT: database object with id 10376
 ACTION: Investigate application logic involving I/O on TABLE
"SCOTT.TEST2" with object id 10376.
RELEVANT OBJECT: database object with id 10376
RATIONALE: The I/O usage statistics for the object are: 2 full object
scans, 29924 physical reads, 0 physical writes and 0 direct reads.
RATIONALE: The SQL statement with SQL_ID "1kh38t0mf98ts" spent
significant time waiting for User I/O on the hot object.
RELEVANT OBJECT: SQL statement with SQL_ID 1kh38t0mf98ts

SYMPTOMS THAT LED TO THE FINDING:

SYMPTOM: Wait class "User I/O" was consuming significant database time.
(46% impact [225 seconds])
FINDING 4: 20% impact (96 seconds)
----------------------------------
Buffer cache writes due to small log files were consuming significant database
time.
NO RECOMMENDATIONS AVAILABLE
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: The throughput of the I/O subsystem was significantly lower
than expected. (12% impact [57 seconds])
SYMPTOM: Wait class "User I/O" was consuming significant database
time. (46% impact [225 seconds])
FINDING 5: 14% impact (66 seconds)
----------------------------------
Waits for redo log buffer space were consuming significant database time.
RECOMMENDATION 1: Host Configuration, 14% benefit (66 seconds)
ACTION: Investigate the possibility of improving the performance of I/O
to the online redo log files.
RATIONALE: The average size of writes to the online redo log files was
567 K and the average time per write was 87 milliseconds.
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: Wait class "Configuration" was consuming significant database
time. (21% impact [104 seconds])
FINDING 6: 12% impact (57 seconds)
----------------------------------
The throughput of the I/O subsystem was significantly lower than expected.
RECOMMENDATION 1: Host Configuration, 12% benefit (57 seconds)
ACTION: Consider increasing the throughput of the I/O subsystem.
Oracle's recommended solution is to stripe all data file using the
SAME methodology. You might also need to increase the number of disks
for better performance. Alternatively, consider using Oracle's
Automatic Storage Management solution.
RATIONALE: During the analysis period, the average data files' I/O
throughput was 1.6 K per second for reads and 4.7 K per second for
writes. The average response time for single block reads was 15
milliseconds.
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: Wait class "User I/O" was consuming significant database time.
(46% impact [225 seconds])
FINDING 7: 8.7% impact (42 seconds)
-----------------------------------
Time spent on the CPU by the instance was responsible for a substantial part
of database time.
RECOMMENDATION 1: DB Configuration, 8.7% benefit (42 seconds)
ACTION: Consider not using the compression option for RMAN jobs.
Alternatively, consider slowing down RMAN activity, or scheduling
RMAN jobs when user activity is lower.
ADDITIONAL INFORMATION:
The instance spent significant time on CPU. However, there were no
predominant SQL statements responsible for the CPU load.
FINDING 8: 7% impact (34 seconds)
---------------------------------
Log file switch operations were consuming significant database time while
waiting for checkpoint completion.
RECOMMENDATION 1: DB Configuration, 7% benefit (34 seconds)
ACTION: Verify whether incremental shipping was used for standby
databases.
ADDITIONAL INFORMATION:
This problem can be caused by use of hot backup mode on tablespaces.
DML to tablespaces in hot backup mode causes generation of additional
redo.
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: Wait class "Configuration" was consuming significant database
time. (21% impact [104 seconds])
FINDING 9: 5.9% impact (29 seconds)
-----------------------------------
Wait event "enq: CF - contention" in wait class "Other" was consuming
significant database time.
RECOMMENDATION 1: Application Analysis, 5.9% benefit (29 seconds)
ACTION: Investigate the cause for high "enq: CF - contention" waits.
Refer to Oracle's "Database Reference" for the description of this
wait event.
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: Wait class "Other" was consuming significant database time.
(12% impact [59 seconds])
FINDING 10: 4.8% impact (23 seconds)
------------------------------------
Wait event "os thread startup" in wait class "Concurrency" was consuming
significant database time.
RECOMMENDATION 1: Application Analysis, 4.8% benefit (23 seconds)
ACTION: Investigate the cause for high "os thread startup" waits. Refer
to Oracle's "Database Reference" for the description of this wait
event.
SYMPTOMS THAT LED TO THE FINDING:
SYMPTOM: Wait class "Concurrency" was consuming significant database
time. (5.9% impact [29 seconds])
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ADDITIONAL INFORMATION
----------------------
Wait class "Application" was not consuming significant database time.
Wait class "Commit" was not consuming significant database time.
Wait class "Network" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
The database's maintenance windows were active during 99% of the analysis
period.
The analysis of I/O performance is based on the default assumption that the
average read time for one database block is 10000 micro-seconds.
An explanation of the terminology used in this report is available when you
run the report with the 'ALL' level of detail.