ProxySG Performance Monitoring and Troubleshooting Webcast - Final

ProxySG Performance Monitoring and
Troubleshooting
April 2016
Rob Ritchardson: Product Support Specialist
Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 1

Why is performance monitoring important?
ProxySG has a finite amount of resources which it uses to process

traffic.
Internal and external issues can create situations where available
resources become scarce and traffic processing is impacted.
It is critical that ProxySG administrators understand what those
resources are, what data is available, how to monitor that data and
how to react to issues impacting resources.
Three key areas of performance monitoring on ProxySG:
CPU
Memory
Bandwidth
Agenda
CPU monitoring via the management console and CLI

Statistics available and their use, CPU Monitor
Identifying and investigating current or past CPU issues
Memory monitoring via the management console and CLI
Statistics available and their use, Threshold monitor
Identifying and investigating current and past memory issues
Bandwidth monitoring via the management console
Understanding bandwidth impact on proxy
Troubleshooting CPU and memory issues

CPU monitoring via the management
console and CLI
CPU statistics in the management console UI
CPU percentages are available in many places within the

management console UI:
Statistics->Summary->Device
Statistics->Health Monitoring->General
Statistics->System->Resources->CPU

Statistics->System->Resources->CPU includes historical graphs

of CPU usage over different time periods.

All three CPU reports/graphs are generated from the same set
of data on ProxySG. This data is available in the persistent
data manager (PDM) statistics in a sysinfo and default
snapshots.
All three CPU reports/graphs show a single CPU percentage
On ProxySG 6.6.1.x or older (6.5, 6.4, etc.) the busiest CPU on multiple
CPU ProxySG platforms is shown in these reports/graphs
On ProxySG 6.6.2.x and newer an average of all the platforms CPUs is
shown in these reports/graphs
The CPU percentage shown is the average CPU over 60 seconds
Very short spikes in CPU usage might not show in these reports/graphs

CPU statistics in the CLI
CLI also has many ways to display

CPU:
show status
show cpu
#show cpu
Current maximum CPU usage (%):
3.5
Single CPU reports follow same

calculations as management
console UI reports.
CPU statistics in the CLI
#show cpu all
Current CPU usage (%):
show cpu has two optional flags to CPU 0: 3.3

give additional information: CPU 1: 0.6
[all] which shows all CPUs individually #show cpu extended
[extended] which shows CPU usage Current maximum CPU usage (%):
over 1, 5, 10 and 60 second averages
1sec 5sec 10sec 60sec
Command examples are from a 2 CPU CPU: 3.3 3.3 3.3 3.5
SG900
#show cpu extended all
The extended information contains Current CPU usage (%):
shorter timeframes for CPU average
which allows for short CPU spike CPU 0:
1sec
3.2
5sec 10sec 60sec
3.3 3.3 3.5
visibility. CPU 1: 0.5 0.6 0.6 0.6

CPU statistics in the advanced URLs
Advanced URLs include two CPU

statistics pages:
CPU Usage Statistics
CPU Monitor
CPU Usage Statistics Information:
Advanced URL:
/Diagnostics/CPU/Statistics
CPU 0 at the top and increments
going down
1, 5, 30 and 60 second averages

CPU Monitor general information
CPU Monitor is a tool that allows an administrator a way to

identify suspect ProxySG processes or components when
investigating high CPU issues. This information is key for quicker
resolution of high CPU issues.
CPU Monitor is the ProxySGs equivalent to Linuxs top
command or Windowss task manager application
CPU Monitor is available in an advanced URL and CLI
CPU Monitor configuration retained after a reboot
Running CPU Monitor incurs 1-2% CPU overhead.

CPU Monitor general information
Example: Configured interval duration:

Current interval complete in:
5 seconds
4 seconds
Configurable CPU 0
HTTP and FTP
99%
89%
interval duration Object Store 10%
Access Logging 1%
CPUs listed with its Miscellaneous 1%
CPU usage CPU 1
TCPIP
22%
20%
(rounded up) DNS service 1%
Components shown
Most component names are meaningful
Two components commonly seen that need clarification:
Object store Kernel, Cache Engine, Storage
Miscellaneous Processing that does not fit into a main component

CPU Monitor in the advanced URL
CPU Monitor advanced URL:

/Diagnostics/CPU_Monitor/Statistics
Advanced URL options:
Start CPU Monitor
Stop CPU Monitor
View CPU Monitor data
(automatic browser refresh)
CPU Monitor advanced
URL included in sysinfo
and default snapshots

CPU Monitor in the CLI
Configuration done in configure terminal mode.

CPU Monitors interval configuration in CLI only.
#conf t <enter>
#(config)diagnostics <enter>
#(config diagnostics)cpu-monitor ?
disable Disable the CPU Monitor

enable Enable the CPU Monitor
interval Configure the CPU Monitor interval

CPU Monitor in the CLI
Viewing CPU Monitor output from the CLI must be done from
enable mode
Command to view CPU monitor is show cpu-monitor
Data is not updated #show cpu-monitor
until interval expires CPU Monitor:
Configured interval duration: 59 seconds
Time in interval Current interval complete in: 18 seconds
remaining also
displayed CPU 0
Console Agent
6%
3%
Miscellaneous 2%

CPU Health check alerting
How to view CPU usage is clear but constant viewing in

anticipation of a CPU issue is not reasonable. Health
Monitoring includes CPU alerting along with many other
resource alerting.
Statistics->Health Monitoring->General shows:
CPU utilization
Current state
Health shows states in prioritized order: Critical, Warning, OK

Health Monitor states are controlled by configurable

thresholds. Changes in state trigger alerts to all of the
configured alerting mechanisms.
Configurations in Maintenance->Health Monitoring->General
Configurable CPU percentage thresholds for critical and warning state
Configurable intervals for critical and warning state
Selectable log, email and trap alerting facilities
Email and trap alerts require additional configurations on the
ProxySG to function properly.

Sample image of CPU alert configurations:
Typically thresholds and intervals are not changed.

Identifying and investigating an active CPU issue
When investigating an active CPU issue the main goal is to first

identify the component(s) using the most CPU.
Check health status in the management console UI. Is it Warning or
Critical state?
Observe CPU usage trends in Statistics->System->Resources->CPU.
Constant high CPU usage versus CPU spikes must be kept in mind
when working with CPU monitor.
Enable CPU monitor and record samples based on the CPU
usage trends seen in Statistics->System->Resources->CPU.
If the management console UI is unresponsive use the CLI (SSH
or serial console) and use show cpu and show cpu-monitor
Identifying and investigating an active CPU issue
Once a suspect component is identified analysis can begin

Suspect: Policy
Configured interval duration: 5 seconds
Investigation: Current interval complete in: 4 seconds
Policy change? CPU 0 99%
Policy evaluation - HTTP 50%
Regex policy rules? HTTP and FTP 35%
Active sessions analysis Object Store 10%
Access Logging 1%
Access logs Miscellaneous 1%
More examples like this in CPU 1 22%
TCPIP 20%
the troubleshooting section DNS service 1%

Identifying and investigating a past CPU issue
Investigating past CPU issues are more difficult since CPU

Monitor is typically not enabled. Graph or statistical data
analysis is needed to match CPU trends that are found.
Check each graph duration in Statistics->System->Resources-
>CPU to see if the CPU issue can be seen. Use that duration in
other graph data (Traffic Mix, Client Workers, etc.) to see if
there are matching trends to help identify a root cause.
SNMP monitoring can greatly assist in this type of investigation
SNMP monitoring knowledge asset: https://youtu.be/PvH30MLfEQY
SNMP resource monitoring document on BTO

If nothing is found in the graph data, statistical analysis is an

option
Persistent Data Management (PDM) statistical data is available in the
ProxySGs sysinfo, default snapshots and heartbeats.
All graphs within the management console UI are built from PDM data.
Snapshots and heartbeats provide historical statistics that can
provide visibility into past CPU issues, sometimes at a very
granular level.
Snapshots can be viewed and download from the
/Diagnostics/Snapshot advanced URL
Heartbeats are sent on a daily bases to all email recipients configured

PDM statistical data example:

system:cpu-usage~hourly@Fri, 01 Apr 2016 00:08:00 UTC[07](60, 60): 9 9 9 9 9
9 9 9 9 9 9 9 10 9 9 9 9 9 9 9 12 9 10 9 10 9 10 9 9 9 9 9 9 9 9 9 9 9 9 9 9
10 9 9 9 9 9 9 9 12 10 9 18 12 9 10 10 10 9 9
The above shows CPU usage over an hour of time

Name of statistic follows PDM syntax: system:cpu-usage~hourly
Date and time mentioned
Current/newest sample is on the right side
(x, y) or (60, 60) above; x = number of samples and y = time in seconds
for each sample. 60 samples * 60 seconds = 3600 seconds or 1 hour
Each sample the value at that time, not an average.

All CPU related data available:

system:cpu-usage~hourly@Fri, 01 Apr 2016 00:08:00 UTC[07](60, 60): 9 9 9 9 9 9 9 9 9 9 9 9 10 9 9 9
9 9 9 9 12 9 10 9 10 9 10 9 9 9 9 9 9 9 9 9 9 9 9 9 9 10 9 9 9 9 9 9 9 12 10 9 18 12 9 10 10 10 9 9
system:cpu-usage~daily15minute@Fri, 01 Apr 2016 00:00:00 UTC[95](96, 900): 2 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 5 5 5 5 5 5 5 5 5 11 9 9 9 9 9 9 9 9 9
system:cpu-usage~daily@Fri, 01 Apr 2016 00:00:00 UTC[23](24, 3600): 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 4 5 7 9 9
system:cpu-usage~weekly@Fri, 01 Apr 2016 00:00:00 UTC[19](28, 21600): 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 5
system:cpu-usage~monthly@Fri, 01 Apr 2016 00:00:00 UTC[17](31, 86400): 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2
system:cpu-usage~yearly@Sun, 27 Mar 2016 00:00:00 UTC[15](52, 604800): 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
Default snapshots contain all of the above. Heartbeats only

contain daily15minute.
Once a CPU issue is found in the PDM data other types of data
can be analysed to find correlations. An example:
system:cpu-usage~hourly@Fri, 01 Apr 2016 00:08:00 UTC[07](60, 60): 9 9 9 9 9
9 9 9 9 9 9 9 10 9 9 9 9 9 9 9 12 9 10 9 10 9 10 9 9 9 9 9 9 9 9 9 9 9 9 9 9
10 9 9 9 9 9 9 9 12 10 9 50 90 100 46 10 10 9 9
users:current~hourly@Fri, 01 Apr 2016 00:08:00 UTC[07](60, 60): 1 0 1 1 1 0 0
0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 1 0 1 1 0 0 0 0 1 0 0 0 1 0
0 0 0 0 1 0 3057 5932 6348 2680 1 0 0 0
In this example we can see that a spike in user counts

correlates with the spike in CPU usage.

There are many PDM statistics that are helpful in investigating

CPU issues. Most of the names are meaningful so that you can
determine what they track.
PDM data is space separated for ease of graphing in the
spreadsheet tool of your choice.
If the CPU issue has a predictable pattern use Health
Monitorings Warning state to alert you early enough that the
issue is going to happen so it can be investigated live.
Enable CPU monitor and leave it enabled if future occurrences
of the CPU issue are expected.
Memory monitoring via the management
console and CLI
Memory statistics in the management console UI
Like CPU, Memory percentages are available in many places

within the management console UI:
Statistics->Summary->Device:
Only historical view
Statistics->Health Monitoring->General:
Statistics->System->Resources->Memory Use:

Memory statistics in the management console UI
Statistics->System->Resources->Memory Use includes a number

of data points.
For memory issues look at
the following:
Committed and available
memory at the top
Committed and free
application memory at the
bottom
Issues are usually with
application memory

Memory statistics in the CLI
Two CLI commands show

available memory:
show status
show resources contains disk
and memory information

Memory statistics in the advanced URLs
Advanced URLs include two
Memory statistics pages:
System Memory Statistics
Threshold Monitor
System Memory Statistics
Information:
Advanced URL: /System/memory
Similar information in the
management console UI
Committed memory / Application
data
Threshold monitor general information
Threshold Monitor is a set of statistics that track bytes of
memory allocated per ProxySG component in 3 different time
intervals.
Threshold Monitor allows an administrator a way to identify
suspect ProxySG components when investigating high
memory issues. This information is key for quicker resolution of
high memory issues.
Threshold Monitor is available in an advanced URL and the
advanced URL data can be shown in the CLI.
The memory allocation statistics are cleared on a reboot.

Threshold monitor general information
Threshold Monitor statistics are included in the default
snapshots and sysinfo.
Threshold Monitor statistics in the advanced URL displays
component names where the same statistics in the default
snapshots and sysinfo does not. Example:
Advanced URL (from hourly, Linear memory stats) Snapshot/Sysinfo
1, Miscellaneous TM004.1.0
1, Authentication TM004.1.1
Comparison of the advanced URL of a ProxySG to the
snapshot/sysinfo data is recommended.

Threshold monitor advanced URL
Threshold Monitor advanced URL: /TM/Statistics

Statistics contain:
Each ProxySG component and its memory usage in bytes
Current/newest entries on the right
- indicate no data recorded (reboot or short uptime)
Three grouping of sample intervals
60 minutes total; 30 samples at 2 minutes each
1 day total; 24 samples at 1 hour each
1 month total; 30 samples at 1 day each
Two groupings of the above; Linear and Physical memory
CSV values for easy graphing in preferred spreadsheet application

Threshold monitor advanced URL
TCPIPs memory usage over the last hour, static (this is OK):
1, TCPIP: 3358720, 3358720, 3358720, 3358720, 3358720, 3358720, 3358720, 3358
720, 3358720, 3358720, 3358720, 3358720, 3358720, 3358720, 3358720, 3358720,
3358720, 3358720, 3358720, 3358720, 3358720, 3358720, 3358720, 3358720, 33587
20, 3358720, 3358720, 3358720, 3358720, 3358720,
SSLs memory usage over the last hour, changes drastically:

1, SSL and Cryptography: 2195826688, 2200390451, 2131156718, 2001598054,
1963358617, 1908186180, 1840209783, 1707371861, 1525337702, 1403926664,
1203994077, 989983402, 978771285, 1045835776, 1109430408, 1493174681,
1624545416, 1783512541, 1830913092, 1929578222, 1929619046, 1950044706,
2146023833, 2117209838,
ProxySG had 4GBs of RAM in the above examples

Memory Health check alerting
Similar to CPU alerting, Health Monitoring also allows for

memory usage alerts.
Statistics->Health Monitoring->General shows:
Memory utilization
Current state
Health shows states in prioritized order: Critical, Warning, OK

Memory Health check alerting
Health Monitor states are controlled by configurable

thresholds. Changes in state trigger alerts to all of the
configured alerting mechanisms.
Memory warning threshold is 90% and critical threshold is 95%
Interval times are the same; 120 seconds
TCP regulation (Memory protection function that delays new requests,
more on this on the next slide) occurs at 80% memory usage,
changing the warning threshold to 75% will alert you about this event
Use the warning threshold to proactively alert administrators of
upcoming memory issues that need to be resolved.

Identifying memory issues
Understanding TCP Acceptance Regulation
At 80% memory pressure the ProxySG will go into TCP Regulation.
When this occurs the Proxy will STOP accepting new TCP
Connections until memory drops below the threshold ( lower limit ).
Recorded in the event log and Threshold Monitor statistics.

Identifying memory issues
Understand memory usage patterns as a climb in memory
usage might not always be an indication of an issue
Memory will rise and fall
during operational times
A leak is an ever
increasing value over
time.
A small, slow leak can be

hidden within normal
usage

Identifying and investigating an active memory issue
When investigating an active memory issue the main goal is to

first identify the component(s) using the most memory.
Check health status in the management console UI. Is it Warning or
Critical state?
Observe memory usage trends in Statistics->Summary->Device
Check memory statistics in Statistics->System->Resources->Memory Use
Access Threshold Monitors advanced URL, save output and
analyze the data to find the suspect component.
If the management console UI is unresponsive use the CLI (SSH
or serial console) to enter enable mode and save output of
show advanced-url /TM/Statistics command.
Identifying and investigating an active memory issue
Once a suspect component is identified analysis can begin

1, Configuration: 2689495040, 2756556526, 3015291153, 3166623744, 3238112597,
3353121177, 3356315648, 3426601642, 3452166144, 3385105203, 3356364800,
3407458850, 3452166144, 3452166144, 3452165597, 3452149760, 3452149760,
3452149760, 3448963481, 3547983872, 3598999278, 3643637760, 3621263906,
3643473920,
Suspect: Configuration Unusual component to be

Investigation: leaking memory
Configuration change? This is a bug
Director involvement? More examples like this in the
Configuration failures in troubleshooting section
event logs
Identifying and investigating a past Memory issue
Investigating a past memory issue is difficult as usually this is

done after a reboot. A reboot clears all of the memory related
statistics.
Aspects of investigating CPU issues applies to memory issues
Analysis of the graphs in the management console UI can be
helpful.
Statistics->Protocol details, most proxy types include client count
graphs. Spike in clients can cause memory issues.
Statistics->Traffic mix, look for large spikes in load
SNMP monitoring can greatly assist in this type of investigation

Identifying and investigating a past Memory issue
If nothing is found in the graph data, statistical analysis is an

option using PDM data which we previously discussed for CPU
issue investigation.
Memory statistics are included with CPU statistics in the same
snapshot and heartbeat historical data.
PDM statistical data example:
system:memory-usage~hourly@Tue, 21 Oct 2014 15:34:00 UTC[33](60, 60): 78 79
78 78 78 78 79 79 78 79 79 79 79 78 78 78 78 78 78 79 79 78 79 79 79 79 79 79
79 79 79 79 79 79 79 79 79 79 79 79 79 79 78 79 78 79 79 79 79 79 79 79 79 76
76 76 76 76 76 76

Identifying and investigating a past memory issue
All memory related data available:

system:memory-usage~hourly@Tue, 21 Oct 2014 15:34:00 UTC[33](60, 60): 78 79 78 78 78 78 79 79 78 79
79 79 79 78 78 78 78 78 78 79 79 78 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 78
79 78 79 79 79 79 79 79 79 79 76 76 76 76 76 76 76
system:memory-usage~daily15minute@Tue, 21 Oct 2014 15:30:00 UTC[61](96, 900): 70 70 70 70 70 71 71
72 73 73 73 74 74 75 74 74 74 74 74 74 74 74 72 72 71 71 71 70 70 70 70 70 69 69 68 69 68 68 67 66
66 67 66 65 64 66 66 66 66 66 66 66 66 66 66 66 66 66 66 65 65 66 66 66 66 66 65 64 63 62 62 62 62
62 63 62 63 64 65 66 67 69 72 74 75 76 76 77 77 77 77 78 78 78 79 78
system:memory-usage~daily@Tue, 21 Oct 2014 15:00:00 UTC[14](24, 3600): 70 70 72 74 74 74 71 70 69
68 67 65 66 66 66 65 66 63 62 63 67 74 77 78
system:memory-usage~weekly@Tue, 21 Oct 2014 12:00:00 UTC[09](28, 21600): 0 0 0 0 0 0 0 0 0 0 0 0 0
21 60 55 56 56 55 53 54 55 54 55 68 72 66 64
system:memory-usage~monthly@Tue, 21 Oct 2014 00:00:00 UTC[16](31, 86400): 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 57 54 62
system:memory-usage~yearly@Sun, 19 Oct 2014 00:00:00 UTC[44](52, 604800): 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8

Identifying and investigating a past memory issue
Correlate memory usage trends to trends found in the

Threshold Monitor statistics. Example:
system:memory-usage~daily@Tue, 21 Oct 2014 15:00:00 UTC[14](24, 3600): 70 70
72 74 74 74 71 70 69 68 67 65 66 66 66 65 66 63 62 63 67 74 77 78
2, SSL and Cryptography: 2915475456, 2923571063, 2963431150, 2999666824,
2990583125, 2951294020, 2793216955, 2659983633, 2433017173, 2283298542,
2088039628, 1958240529, 1905309832, 1818618265, 1810595976, 1804682854,
1752943001, 1280302557, 1158922376, 1299194675, 1788627217, 2681711684,
2884126446, 2936206677,
In this example we can see how the drops and increases in

SSLs memory usage match the memory usage percentage.
Investigating SSL load, worker counts, connections, etc. is next step.

Bandwidth monitoring via the
management console
Bandwidth impact on ProxySG
Client and server side bandwidth processing directly affects

the ProxySGs resources; CPU and memory.
Each ProxySG platform is sized to be able to process a certain
amount of bandwidth while maintaining an appropriate level
of resource usage.
Increases in bandwidth where the amount is over what was sized for
the platform can cause high CPU and/or memory usage.
Changes in the types of traffic being processed while maintaining the
same bandwidth can also cause high CPU and/or memory usage.
Understanding bandwidth processing is needed for normal
operations and problem investigations.
Bandwidth statistics in the management console UI
Statistics->Traffic Details->Traffic Mix
Total processed bandwidth for

client side, server side and
bypassed traffic.
Multiple durations available
By service name or proxy type
reporting
Total bytes counted and
savings calculated

Statistics->Traffic Details->Traffic History
Processed bandwidth for client

side, server side and bypassed
traffic per service name or
proxy type
Multiple durations available
Separate graphs showing
client and server bandwidth
together or individually
Bytes counted and savings
calculated
Both Traffic Mix and Traffic History use the same service names
and proxy types.
Service name tracks bandwidth matching IPs or ports. Within
that traffic different proxies can process the traffic.
Explicit HTTP can contain HTTPS traffic within it
Proxy reports give better visibility into types of traffic processed

Suspect component identification using CPU monitor or

Threshold Monitor speeds up investigation time dramatically.
Without these graph data from the management console UI or
statistical data from snapshots or heartbeats must be analyzed for
correlations.
For memory issues understand memory usage for normal
operations versus a leak
If a reboot is planned to resolve either issue then a full core
should be dumped
KB http://bluecoat.force.com/knowledgebase/articles/Solution/How-
do-I-enable-a-full-core-dump-on-the-ProxySG
High CPU in TCPIP component
CPU Monitor shows high CPU in TCPIP

Check connection table CPU 0 99%
Advanced URL: /TCP/connections TCPIP 90%
SSL and Cryptography 4%
Time-wait entries (2MSL)
HTTP and FTP 3%
Attacks Policy evaluation HTTP 1%
Check interface statistics Object Store 1%
Statistics->Network-Interface Access Logging 1%
Miscellaneous 1%
Errors?
Bypass data in transparent deployments
Extremely high packets per second

TCP connections advanced URL
TCP connection table

Advanced URL: /TCP/connections
Lists all incoming and outgoing
connections
Can be very large
Shows problematic clients that open
to many connections
Connection states listed
Large time_wait lists can consume
CPU
Half-opened connections could be
a sign of an attack

High CPU in HTTP or Policy components
CPU Monitor shows high CPU in HTTP or Policy

Check active sessions CPU 0 99%
Policy evaluation HTTP 64%
Many connections from a single SSL and Cryptography 16%
client? HTTP and FTP 10%
Suspicious destinations? TCPIP 10%
User counts and connections Object Store 1%
Access Logging 1%
Connection table Miscellaneous 1%
TCP user counts in advanced
URL: /TCP/users
View access log for suspicious activity live on the ProxySG

Statistics->Access Logging, Tail main access log

TCP users advanced URL
User information tracked in the TCPIP stack

Advanced URL: /TCP/users
Active users list shows list of IP addresses and number of connections
they have opened
High connection counts on a single IP can be a sign of an issue

High CPU in SSL component
CPU Monitor shows high CPU in SSL and Cryptography

SSL interception consumes CPU 0 99%
SSL and Cryptography 62%
CPU HTTP and FTP 20%
Configurations lower CPU: Policy evaluation HTTP 13%
Disable DHE support TCPIP 7%
Object Store 1%
Increase certificate timeout
Access Logging 1%
Add splash text to policy Miscellaneous 1%
KB Article:
http://bluecoat.force.com/knowledgebase/articles/Solution/000024136
SSL interception on exception default mode in SGOS 6.2+
Add splash text to SSL interception on exception policy rule

CPU issues caused by general load
Proxy operations involves many components

HTTP, SSL, TCPIP, Policy, DNS, Object Store
High CPU issues divided amongst these components typically
points to sizing issues (high bandwidth, high user counts)
CPU 0 99%
Busiest: HTTP, SSL, Policy SSL and Cryptography 33%
HTTP and FTP 25%
Policy evaluation HTTP 20%
Middle: Object store, TCPIP TCPIP 15%
Object Store 7%
Access Logging 1%
Lowest: DNS, Access Logging Miscellaneous 1%

CPU issues caused by general load
Check the following when CPU pattern looks like general load:
Did something in the environment change that triggered the CPU?
More traffic moving from HTTP to HTTPS?
Cloud services being adopted?
Bandwidth being processed from Traffic Mix graphs. Is the proxy sizing
correct?
User counts and connections, are the values expected?
Statistics->System->Resources->Concurrent Users
TCP Users advanced URL: /TCP/users
Active sessions, clients accessing data that should be controlled?

High memory in HTTP/TCP/SSL/ADN components
Analyze load in Traffic Mix. Is the bandwidth too high?

Examine connection counts, users, and worker counts
Connection table: /TCP/connection
Users connections: /TCP/users
Management console UI Statistics->Protocol details
Each protocol has a worker or client count, are the values the same as the
baseline
If load look good, verify dependency health
Statistics->Health checks, response times for DNS, Auth, and ICAP
Statistics->ICAP, queued connections? (sign of an ICAP server issue)

Blue Coat Customer Forums
Community where you can learn from and share your valuable
knowledge and experience with other Blue Coat customers
Research, post and reply to topics relevant to you at your own
convenience
Blue Coat Moderator Team ready to offer guidance, answer
questions, and help get you on the right track
Access at forums.bluecoat.com and register for an account
today!

Thank you for Joining Today!
Please provide feedback on this webcast and suggestions for

future webcasts to:
john.dyer@bluecoat.com
Webcast replay and slide deck found here within 48 hours:

https://bto.bluecoat.com/training/customer-support-technical-
webcasts
(Requires BTO log-in)

Quick Survey
We are truly committed to continuous improvement for these

Technical Webcasts. At the end of the event you will be re-
directed to a very short survey about satisfaction with this
Program. Please help us out by taking two minutes to complete
it. Thank you!
Questions for Rob?


ProxySG Performance Monitoring and Troubleshooting Webcast - Final

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ProxySG Performance Monitoring and Troubleshooting Webcast - Final

Uploaded by

Copyright:

Available Formats

ProxySG Performance Monitoring and

Rob Ritchardson: Product Support Specialist

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 1

ProxySG has a finite amount of resources which it uses to process

CPU monitoring via the management console and CLI

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 3

CPU percentages are available in many places within the

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 5

Statistics->System->Resources->CPU includes historical graphs

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 6

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 7

CLI also has many ways to display

Single CPU reports follow same

show cpu has two optional flags to CPU 0: 3.3

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 9

Advanced URLs include two CPU

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 10

CPU Monitor is a tool that allows an administrator a way to

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 11

Example: Configured interval duration:

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 12

CPU Monitor advanced URL:

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 13

Configuration done in configure terminal mode.

disable Disable the CPU Monitor

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 14

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 15

How to view CPU usage is clear but constant viewing in

Health shows states in prioritized order: Critical, Warning, OK

Health Monitor states are controlled by configurable

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 17

Sample image of CPU alert configurations:

Typically thresholds and intervals are not changed.

When investigating an active CPU issue the main goal is to first

Once a suspect component is identified analysis can begin

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 22

Investigating past CPU issues are more difficult since CPU

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 23

If nothing is found in the graph data, statistical analysis is an

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 24

PDM statistical data example:

The above shows CPU usage over an hour of time

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 25

All CPU related data available:

Default snapshots contain all of the above. Heartbeats only

In this example we can see that a spike in user counts

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 27

There are many PDM statistics that are helpful in investigating

Like CPU, Memory percentages are available in many places

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 30

Statistics->System->Resources->Memory Use includes a number

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 31

Two CLI commands show

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 32

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 34

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 35

Threshold Monitor advanced URL: /TM/Statistics

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 36

SSLs memory usage over the last hour, changes drastically:

ProxySG had 4GBs of RAM in the above examples

Similar to CPU alerting, Health Monitoring also allows for

Health shows states in prioritized order: Critical, Warning, OK

Copyright 2016 Blue Coat Systems Inc. All Rights Reserved. 38