Professional Documents
Culture Documents
Bruce McCartney
bruce.mccartney@dbinfosystems.com
DBIS
Outline
• Speaker Introduction
• Motivation for session modification
• Examples I have used it for
• Overview of Oracle features relating to session tracing
• How to exploit these features - overview
• Example/Demo
2
DBIS
Speaker Introduction
• Oracle Corp Consultant 1989-1991
– Performance tuning
• Co-founder of DBCORP (1991-2002)
– DBA products, Remote DBA Services
– Performance tuning services
• COGNICASE acquired DBCORP in 2002, CGI acquired
COGNICASE in 2003
• Database Information Systems Inc. (April 2004)
• Past President of Calgary Oracle Users Group
• Current CIPS DBA SIG Lead
• Several IOUW/OOW presentations
3
DBIS
Motivation
• The Problem is:
– Need the ability to pinpoint tracing of a session by some set of
criteria – automatically, often in PRODUCTION
– Need to change the Oracle behavior for SOME SPECIFIC sessions
in production
– Do it with ease and flexibility, consistent with new and upcoming
Oracle functionality
4
DBIS
Motivation
• Compounding The Problem is:
– Purchased applications often cannot see, let alone change the
source code
– Application (both purchased and in-house) are not instrumented
– Three-tier architecture and connection pooling makes in difficult to
determine end-to-end access paths
– Production change control often dictates no code changes (but data
changes easier to justify)
5
DBIS
For Example –Tracing Focus
• Problem:
– Need to trace production job launched by ‘group’ user_name
running production code (may run tonight at 2am local time; only if
other stuff happens)
• Solution
– Set up automation for that USER and PROGRAM/TERMINAL to
issue
ALTER SESSION SET SQL_TRACE=TRUE
6
DBIS
For Example – Diagnostics Events
• Problem:
– Production look-up using HUGE amount of CPU. Vendor claims
not possible; explain looks ok INDEX RANGE SCAN on Primary
Key Index; but LIO >> expected
– Unable to duplicate in Vendor or Test environment
• Solution
– Set up automation for that PROGRAM to issue “ALTER SESSION
and SET events ‘10046 trace name context forever, level 12’”
– Bind variable data reveals data type mismatch on 3rd column of 7
column PK; range scan scope was 2000 blocks instead of 20
– Trace file given to vendor as proof
7
DBIS
For Example – Custom Behavior
• Problem:
– Production Load batch job needed huge SORT/HASH area size
to optimize (Version 8.1.7), 350 Users online with same Oracle
USERNAME; code untouchable
• Solution
– Set up automation for that OS user for BATCH to issue
8
DBIS
For Example - Custom Behavior
• Problem:
– Production SQL statement SQL PLAN needed to change
(index->full table scan) for one ‘class’ of user of a program
due to data skew
– Other classes of user and programs need to use the index
• Solution
– Used Stored Outlines for that user
• Create outline for SQL with/without HINT
• Switch Outlines
– Set up automation for that program to
• ALTER SESSION SET USE_STORED_OUTLINES = TRUE;
9
DBIS
For Example – Application Instrumentation
• Problem:
– For service level management, need to tie Application
Components to specific sessions
• Solutions
– Login trigger to query application tables to set MODULE,
ACTION and CLIENT_INFO using
DBMS_APPLICATION_INFO
– Poll v$session_longops to tie long running operations with
Application components
– Create month-end by component tracking capability
10
DBIS
Oracle features supporting tracing
• DBMS_MONITOR
– Enable/disable ‘statistics’ for client_id or service/module/action
– Enable/disable trace for session, client_id or service/module/action
• DBMS_SESSION
– Set identifier (client_id)
– Set Context / SQL_TRACE
• DBMS_APPLICATION_INFO
– SET<MODULE|ACTION|CLIENT_IDENTIFIER>
– OCI_ATTR_<MODULE|ACTION|CLIENT_IDENTIFIER>
• Alter session
• Events
• Trace Files and Alert Log
• Trace Analysis (trcsess)
• Database procedures and triggers
11
DBIS
DBMS_MONITOR
• ‘10g’ way to enable tracing
– oradebug
– dbms_system.set_ev(&sid, &serial, 10046, 12, '');
– dbms_session.sql_trace
– DBMS_SUPPORT.START_TRACE_IN_SESSION( sid , 0, waits=>TRUE,
binds=>TRUE )
• Trace Enabling
– Enabled for all instances and persists across restarts
– SERV_MOD_ACT is a hierarchy
– Session requires session_id and serial#
– <CLIENT_ID|SERV_MOD_ACT|SESSION>_TRACE_<ENABLE|DISABLE>
• Statistics Enabling
– New V$CLIENT_STATS, V$SERV_MOD_ACT_STATS views
– <CLIENT_ID|SERV_MOD_ACT|SESSION>_STAT_<ENABLE|DISABLE>
12
DBIS
DBMS_MONITOR
• Issues with 10g features of DBMS_MONITORING
– SERVICE definition and management
• Default SYS$USERS
• Select * from v$services
• DBMS_SERVICE used to manage
– CLIENT_ID management
• Default NULL, must be set by APPLICATION (OEM sets it)
• Set by DBMS_SESSION – any 64-byte string
– Can’t distinguish by these ‘session’ attributes
• PROGRAM, TERMINAL, SERVER,MACHINE, OS_USER, CLIENT_INFO
– Are your applications instrumented to use this?
13
DBIS
DBMS_SESSION
• Version 8+
– dbms_session.set_context(namespace, attribute,
value /*10g*/, username, client_id)
• Version 9+
– dbms_session.set_identifier(client_id varchar2)
• Used to communicate via context in VPD
• Required for statistic aggregation in 10g
DBMS_MONITOR in v$client_stats
• Are your applications instrumented to use this?
14
DBIS
DBMS_APPLICATION_INFO
• Version 8+
– READ_CLIENT_INFO
– READ_MODULE
– SET_ACTION
– SET_CLIENT_INFO
– SET_MODULE
– SET_SESSION_LONGOPS
• Module - high level application component
• Action – lower level procedure/routine
• Essential Application Instrumentation
– Tied into v$session; v$sqlarea
– Oracle trace files (i.e. event 10046)
• Are you applications instrumented for this?
15
DBIS
DBMS_APPLICATION_INFO
• New in Version 10: OCI Attributes
– OCI_ATTR_MODULE
– OCI_ATTR_ACTION
– OCI_ATTR_CLIENT_IDENTIFIER
• Enable OCI programs to include inside database calls (i.e. no
additional calls/overhead required)
• Are your applications instrumented for this?
16
DBIS
Alter Session
• Used to change characteristics of the current session
ALTER SESSION
{ ADVISE { COMMIT | ROLLBACK | NOTHING } |
CLOSE DATABASE LINK dblink | { ENABLE | DISABLE }
COMMIT IN PROCEDURE |
{ ENABLE | DISABLE | FORCE } PARALLEL { DML | DDL |
QUERY } [ PARALLEL integer ] |
{ ENABLE RESUMABLE [ TIMEOUT integer ] [ NAME
string ] | DISABLE RESUMABLE } |
alter_session_set_clause } ;
17
DBIS
Alter Session
• Set Clause
SET parameter_name = parameter_value
[ parameter_name = parameter_value ]... [ COMMENT =
'text' ]
18
DBIS
Alter Session Initialization Parameters
– NLS parameters (15 in
10g)
• OBJECT_CACHE_MAX_SIZE_PERCENT
– Optimizer options •
•
OBJECT_CACHE_OPTIMAL_SIZE
QUERY_REWRITE_ENABLED
• OPTIMIZER_DYNAMIC_SAMPLING • QUERY_REWRITE_INTEGRITY
• OPTIMIZER_FEATURES_ENABLE • STAR_TRANSFORMATION_ENABLED
• OPTIMIZER_INDEX_CACHING • OLAP_PAGE_POOL_SIZE
• OPTIMIZER_INDEX_COST_ADJ • OSM_POWER_LIMIT
• OPTIMIZER_MODE
• REMOTE_DEPENDENCIES_MOD
– Session behavior •
•
RESUMABLE_TIMEOUT
SESSION_CACHED_CURSORS
• PARALLEL_INSTANCE_GROUP
• SKIP_UNUSABLE_INDEXES
• PARALLEL_MIN_PERCENT • SORT_AREA_RETAINED_SIZE
• PLSQL_CODE_TYPE • SORT_AREA_SIZE
• PLSQL_COMPILER_FLAGS • SQL_TRACE
• PLSQL_DEBUG • SQLTUNE_CATEGORY
• PLSQL_OPTIMIZE_LEVEL • STATISTICS_LEVEL
• PLSQL_V2_COMPATIBILITY • TIMED_OS_STATISTICS
• PLSQL_WARNINGS • TIMED_STATISTICS
• TRACEFILE_IDENTIFIER
• WORKAREA_SIZE_POLICY
19
DBIS
Alter Session Options
• Session Options
– CONSTRAINTS
– ISOLATION LEVEL
– STORED_OULTLINES
– TIME_ZONE
– SKIP_UNUABLE_INDEXES
– CURRENT_SCHEMA
• Lots of opportunity to change session behavior and in some
cases ALTER SESSION is the only way
20
DBIS
Events
• Setting an Event causes Oracle Kernel to do something
specific to the event ID
• Usually causes Oracle to log/dump things to a trace file
• Can be set for instance (alter system/init.ora/spfile) or
session (alter session)
• Complete list can be browsed in the file
$ORACLE_HOME/rdbms/mesg/oraus.msg on Unix
– Range 10000-10999 events used
21
DBIS
Common Diagnostic Events
• Note: Various ‘levels’ can effect details captured
Event Description
10046,10241 SQL Trace, Remote
SQL
10053 Optimizer Trace
10032,10033 Sort Activity
10248, 10249, 10257 MTS tracing
22
DBIS
Oracle “Wait” Event 10046
• Oracle V7 instrumented to capture Kernel waits
• v$sysstat and v$session_event are summarized
views
• Event 10046 level >3 causes trace file for each wait
• “Idle” waits – not impacting response
• “Non-idle” waits – component of response time
24
DBIS
Alert Log
• Alert_log
– System wide information kept in background_dump_dest directory
usually called alertORACLE_SID.LOG
– Application can write via PL/SQL call:
dbms_system.ksdwrt(2,:text_message)
25
DBIS
Trace files
• Trace_files
– Process specific information kept in user_dump_directory
(foreground processes) called something like
ORACLE_SID_ora_PROCESS#.trc)
– You can change the name of these files using (caution if
connection pooling or have set 10046)
ALTER SESSION SET TRACE_IDENTIFIER=‘BRUCE’
– You can also write to this file via:
DBMS_SYSTEM.KSDWRT(1,:text_message)
26
DBIS
Trace File Analysis
• Background processes also have trace files in
background_dump_dest
• Connection pooling effects trace file as many session
share a single server process – trace files are marked
with sid, serial#. You have to scrape a single session out
– HINT: trcsess in Oracle 10g does it for you – even for Version 8i
& 9i trace files
• Can be large (max_dumpfile_size)
– Worthwhile looking through and understanding
27
DBIS
Trace Analysis Tools
• TKPROF
– Summarizes SQL and waits
– Lose some information (clock time, hash_value, module/action)
• Report summarizes data (missing wait times)
28
DBIS
TKPROF Example
********************************************************************************
EXPLAIN PLAN
29
DBIS
TKPROF Example
Rows Row Source Operation
------- ---------------------------------------------------
0 UPDATE (cr=1837228 r=122899 w=0 time=4629619263 us)
0 TABLE ACCESS BY LOCAL INDEX ROWID TSETTLEMENT (cr=1837228 r=122899 w=0 time=4629618504 us)
5226800 NESTED LOOPS (cr=1747539 r=33252 w=0 time=1858427726 us)
475440 VIEW (cr=36769 r=53 w=0 time=7252734 us)
475440 SORT UNIQUE (cr=36769 r=53 w=0 time=6852549 us)
475440 MERGE JOIN (cr=36769 r=53 w=0 time=4981792 us)
15848 SORT JOIN (cr=34289 r=0 w=0 time=565784 us)
15848 TABLE ACCESS BY INDEX ROWID TRETAILER_RUN_SITE_MAP (cr=34289 r=0 w=0 time=456506 us)
31776 NESTED LOOPS (cr=32571 r=0 w=0 time=395094 us)
15848 TABLE ACCESS BY INDEX ROWID TREQUEST_JOB_BATCH_ENTITY (cr=639 r=0 w=0 time=65275 us)
15848 INDEX RANGE SCAN RJBE_IDX01 (cr=381 r=0 w=0 time=23565 us)(object id 56182)
15848 INDEX RANGE SCAN RR_SITE_MAP_IDX01 (cr=31932 r=0 w=0 time=255699 us)(object id 57910)
475440 FILTER (cr=2480 r=53 w=0 time=4222833 us)
1933456 SORT JOIN (cr=2480 r=53 w=0 time=3448926 us)
146080 TABLE ACCESS FULL TCALENDAR (cr=2480 r=53 w=0 time=1452529 us)
4751280 PARTITION RANGE ITERATOR PARTITION: KEY KEY (cr=1710770 r=33199 w=0 time=1841836133 us)
4751280 INDEX RANGE SCAN SETTLEMENT_NEW_LPUDX01 PARTITION: KEY KEY (cr=1710770 r=33199 w=0 time=1835972096
us)(object id 58770)
30
DBIS
Trace Analysis Tools
• TRCA (PL/SQL See Metalink Note 22470.1)
– Detailed view of trace file by CURSOR
– Overhead executing
– Report very long, but very detailed information avaiable
31
DBIS
TRCA Example
*********************************************************************************************T
RCANLZR.sql 115.9 NOTE:224270.1 2004-03-09 15:22:32 Trace Analyzer
*********************************************************************************************
TRACE_DIRECTORY..........: /ai01/oracle/admin/SETP/udump (ALIAS:UDUMP)
TRACE_FILENAME...........: setp_j008_28232.trc (TRACE_ID:2)
INCLUDE_SYS_COMMANDS.....: YES
32
DBIS
TRCA Example
SUMMARY OF CALLS BY USER (INTERNAL LAST) AND NON-RECURSIVE/RECURSIVE
====================================================================
OVERALL TOTALS PER CALL FOR ALL RECURSIVE STATEMENTS FOR USER 26 (UNET_SYSTEM)
33
DBIS
TRCA Example
Details for wait event Times Count Max. Total Blocks
'buffer busy waits (reason code 220)' Waited Zero Time Wait Waited Accessed
----------------------------------------------------------------- --------- --------- --------- --------- ---------
sys._syssmu19$................................................... 1015 0 3.15 3004.40 223300
unet_system.req_batch_ent_pk..................................... 31 0 0.03 0.08 6820
sys._syssmu20$................................................... 19 0 0.01 0.01 4180
34
DBIS
Trace Analysis Tools
• Hotsos Profiler
– Done right!
35
DBIS
Session Performance Info
– V$ views that can be queried in real time
View Description
V$SESSION This view lists session information for each current session.
V$PROCESS This view contains information about the currently active processes
38
DBIS
For the GUI-DBA
• Oracle Enterprise Manager “session detail” (Oracle 9)
• EM Grid Control Oracle 10
• TOAD – session browser
• QUEST/EMBARCADERRO/BMC
39
DBIS
Database procedures/ triggers
• Procedures
– Stored PL/SQL logic
• Triggers
– Data change triggers
– Database triggers on events:
• SESERVERERROR
• LOGON
• LOGOFF
• STARTUP
• SHUTDOWN
• SUSPEND
40
DBIS
Other opportunities
• v$session_longops
– Graphical representation in OEM
– Oracle uses for scans > 10K blks; Sorts and SQL Executions etc
– Use DBMS_APPLICATION_INFO to instrument your ‘long’ operations –
batch jobs
– Query and sample
select sid, serial#, message, to_char(start_time, 'Mon dd
HH24:MI:SS') as Start_Time, (sofar/totalwork)*100 as "%
Complete" from v$session_longops where LAST_UPDATE_TIME >
sysdate - 1/24 and message like 'Table Scan%' and sofar <
totalwork
– Polling table and saving data can provide wealth of information
41
DBIS
Exploiting the features
• Basic Concept of Technique
– Logon trigger reads table of alter session clauses based upon
session specific attributes matched in the table
– Logoff trigger dumps session summary to trace file
• Extensions that are possible
– Logon trigger integrates to application to extend instrumentation
via DBMS_APPLICATION_INFO
– Trace file ‘mining’ extracts SQL failing services level and notifies
workflow
– All kinds of goodies
42
DBIS
Making it work
• Some installation objects
– Can create a special user or use SYSTEM
– Privileges
• MGMT_USER, SELECT_CATALOG_ROLE, RESOURCE
• ADMINISTER_DATABASE_TRIGGER, ALTER SESSION
• EXECUTE SYS.DBMS_SYSTEM, SELECT on some V_$ tables
– Objects
• SESSION_MODIFICATION TABLE
• APP_TRACE PACKAGE
• LOGINTRIG TRIGGER
43
DBIS
Logic flow
User
Login
LOGON
TRACEON
TRIGGER
LOGOFF DUMPSTATS
44
DBIS
Installation
• Script download from www.dbinfosystems.com/news.htm
45
DBIS
LOGINTRIG
CREATE OR REPLACE TRIGGER
LOGINTRIG AFTER LOGON ON DATABASE
Begin
app_trace.session_mod;
End;
• Notes
– Invalidation causes login prevention
– Alter trigger LOGINTRIG disable;
– Need ADMINISTER DATABASE TRIGGER privilege or SYSDBA
46
DBIS
SESSION_MODIFICATION
CREATE TABLE "SESSION_MODIFICATION"
("USERNAME" VARCHAR2(30 BYTE),
"SERVER" VARCHAR2(9 BYTE),
"OSUSER" VARCHAR2(30 BYTE),
"PROCESS" VARCHAR2(9 BYTE),
"MACHINE" VARCHAR2(64 BYTE),
"TERMINAL" VARCHAR2(30 BYTE),
"PROGRAM" VARCHAR2(48 BYTE),
"MODULE" VARCHAR2(48 BYTE),
"CLIENT_INFO" VARCHAR2(64 BYTE),
"CLIENT_IDENTIFIER" VARCHAR2(64 BYTE),
"SERVICE_NAME" VARCHAR2(64 BYTE)
"SET_CLAUSE" VARCHAR2(1024 BYTE),
)
• Notes
– Need USERNAME populated
– SET_CLAUSE column applied to matching rows
47
DBIS
PACKAGE APP_TRACE
PROCEDURE MYDUMPSTATS /* called by LOGOFF TRIGGER - dumps to session trace */
PROCEDURE SESSION_MOD /* called by LOGON TRIGGER - alter session commands*/
PROCEDURE SET_TRACE /* turn on/off 10046 event */
PROCEDURE TRACE_OUT /* debug trace file output */
Argument Name Type In/Out Default?
------------------------------ ----------------------- ------ --------
ITEXT VARCHAR2 IN
IFLUSH BOOLEAN IN DEFAULT
• Notes
– Use for application instrumentation
• Manages Trace output
• DBMS_APPLICATION_INFO
48
DBIS
Driving Query
SELECT m.USERNAME m_USERNAME, m.SERVER m_SERVER,
m.OSUSER m_OSUSER, m.PROCESS m_PROCESS,
m.MACHINE m_MACHINE, m.TERMINAL m_TERMINAL,
m.PROGRAM m_PROGRAM, m.MODULE m_MODULE,
m.CLIENT_INFO m_CLIENT_INFO, m.SET_CLAUSE m_SET_CLAUSE ,
m.SERVICE_NAME m_SERVICE_NAME , m.CLIENT_IDENTIFIER m_CLIENT_IDENTIFIER ,
s.USERNAME s_USERNAME, s.SERVER s_SERVER,
s.OSUSER s_OSUSER, s.PROCESS s_PROCESS,
s.MACHINE s_MACHINE, s.TERMINAL s_TERMINAL,
s.PROGRAM s_PROGRAM, s.MODULE s_MODULE,
s.CLIENT_INFO s_CLIENT_INFO, s.SERVICE_NAME s_SERVICE_NAME ,
s.CLIENT_IDENTIFIER s_CLIENT_IDENTIFIERfrom session_modification m,
select username, server,osuser,process, machine,terminal,program,module,client_info,
client_identifier,service_name
from v$session
where sid=USERENV('SID')) s
where m.username = user and
m.username=s.username and
(m.osuser = s.osuser OR m.osuser is NULL) and
(m.CLIENT_INFO = s.CLIENT_INFO OR m.CLIENT_INFO is NULL) and
(m.MODULE = s.MODULE OR m.MODULE is NULL) and
(m.PROGRAM = s.PROGRAM OR m.PROGRAM is NULL) and
(m.TERMINAL = s.TERMINAL OR m.TERMINAL is NULL) and
(m.MACHINE = s.MACHINE OR m.MACHINE is NULL) and
(m.PROCESS = s.PROCESS OR m.PROCESS is NULL) and
(m.SERVICE_NAME = s.SERVICE_NAME OR m.SERVICE_NAME is NULL) and
(m.CLIENT_IDENTIFIER = s.CLIENT_IDENTIFIER OR m.CLIENT_IDENTIFIER is NULL) and
(m.SERVER = s.SERVER OR m.SERVER is NULL);
49
DBIS
Demonstration
• Questions?
• bruce.mccartney@dbinfosystems.com
• References/Credits on PL/SQL:
– asktom.oracle.com (Tom Kyte : instrumentation and using dbms_%)
– www.minmaxplsql.com (Steven Feuerstein capture/trace PL/SQL)
– http://www.jlcomp.demon.co.uk/faq/utlfiledebug.html (Jonathan Lewis)
50
DBIS