HP ArcSight Connector

Health Check
Tracy Barella
Chief Services Strategist
Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

HP ArcSight Connector health check

Agenda
What is a health check?
Health check steps by ArcSight
component
Connectors
Connector Appliances

Q&A

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Health Check overview

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

What is a health check?

Purpose
The purpose of performing a health check is to identify and remove performance
bottlenecks to enable top performance of the HP ArcSight implementation. Minor
issues can result in major performance degradations over time impacting system
availability and user satisfaction. Performing regular health checks will identify
issues allowing them to be remediated quickly and ensure continued top performance
of the HP ArcSight implementation.
In a nutshell
A Health Check consists of common administrative tasks to verify the ArcSight
solution is configured and performing optimally.
4

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Health Check steps

by ArcSight component

Note: Its impossible to cover every scenario in this presentation,

so only the common checks will be discussed.

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Health check steps by ArcSight component

Connectors
Tip: Check each ArcSight Component by the order of the Event Flow

1
Connectors
Up/Down Check
(Connector or Container)
Version Check
Connector Event Rate
Check (by EPS)
Cache Check

Connector
appliances
Version Check

Event Throughput Dashboard Check

CPU and Memory Check Current Event Sources Dashboard Check

Hardware and Operating System Check

Network Settings
Check

CPU and Memory Utilization Check

Configuration Backup
Check

ESM Manager JVM (memory) Utilization

Check

Logs Check
Configuration Check

Data Monitor Utilization Check

Active List/Session List Utilization Check
Rules Engine Check

Its just simple plumbing!!!

ESM Database
and storage

ESM Manager

DBCheck and Oracle RDA

Database Performance
Statistics Dashboard Check
Partition Check (Oracle)
Trend Jobs Check
Hardware and Operating
System Check
CPU and Memory Utilization
Check

Oracle version and patch level

Event Persistence (insertion) Performance
check
Check
Oracle alert log check
Error Check
Oracle memory parameters
Scheduled Task Check
check
server.properties Check
ESM Database Storage Check

Agent and Console Threads Check

6

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Loggers
CPU, Memory, and EPS In/Out
Check
Search Performance Check
Custom Report Performance
Check
Receivers and Forwarders Check
Storage Group Check
Index Configuration Check
Configured Alerts Check
Scheduled Task Check
Event Archive and Configuration
Backup Check
Logger System Health and Audit
Event Forwarding Check
Network Configuration Check
Online Event Storage Check
(Only Software-based or SAN
Logger)

Connectors
Connector (or Container) Up/Down Check
Connector Version Check
Are there any Connectors running a version older than ~1 year?
A minimum version of 4.8.1 is required to leverage the ESM v5.2 schema.

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Connectors (cont.)
Connector Cache Check

All Connectors should have 0 events in the cache

If most Connectors are continuously caching = Possible ESM level Event Insertion problem

If one or two Connectors are continuously caching = Possible Connector level problem or network issue

If a Connector caches for a moment and then clears the cache (batched events) = This is normal

Connector Event Rate Check (by EPS)

Are there any Connectors receiving a high event rate?

See below for definition of high EPS on common
Connector types:

Syslog Connector or CheckPoint Connector : >= 1,500 EPS

Windows Unified Connector: > 500 to 1,000 EPS

DB-based Connector or SourceFire eStreamer Connector: >=

200 EPS

Is the high EPS Connector stable? If not, we should

recommend another Connector to spread the load?

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Connectors (cont.)
Connector Logs Check

../current/logs/agent.out.wrapper.log

Java Heap Memory Utilization

Memory utilization
Frequency of Full GCs
Memory in Red Zone alerts

Unexpected Connector restarts

Connectivity errors
End Devices
ArcSight Destinations

../current/logs/agent.log

Parsing errors

DOSProtector

Chronic WARN and ERROR messages

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Connectors
Connector Logs Check (cont.)
Use Connector LogFu to graph the event
flow and memory utilization

10

../current/bin/arcsight agent logfu a

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Connectors (cont.)
Connector Configuration
Check
Destination Settings

Are there more than 2 Destinations on each

Connector?
Too many Destinations can negatively
impact performance of a Connector.

Common problems found:

Networks and CustomerURI are not applied
on every Connector
Fields-based Aggregation is not properly
applied (by Connector Type)
No tuning (Filter Out) applied on high EPS
Connectors
Settings are not the same on every
Destination (ESM, Logger, etc.)

11

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Connectors (cont.)
Connector Configuration Check (cont.)
Only check the following on problematic Connectors discovered in previous checks

../current/user/agent/agent.properties
Optimal settings are different for each Connector type
High EPS Connectors (>1200 EPS) such as Syslog, WUC, CheckPoint, and Blue Coat can be tweaked quite a bit here

../current/user/agent/agent.wrapper.conf
Only increase the Java Heap size if memory issues were found in agent.out.wrapper.log
Default Java Heap is 256MB
Maximum configurable Java Heap is 1024MB (1 GB)

Reminder: If you have 50+ Connectors in your environment, try to stay focused on problematic Connectors!

12

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Health check steps by ArcSight component

Connector Appliances
Tip: Check each ArcSight Component by the order of the Event Flow

1
Connectors
Up/Down Check
(Connector or Container)
Version Check
Connector Event Rate
Check (by EPS)
Cache Check
Logs Check
Configuration Check

Connector
appliances
Version Check

ESM Database
and storage

ESM Manager
Event Throughput Dashboard Check

CPU and Memory Check Current Event Sources Dashboard Check

Hardware and Operating System Check

Network Settings
Check

CPU and Memory Utilization Check

Configuration Backup
Check

ESM Manager JVM (memory) Utilization

Check
Data Monitor Utilization Check
Active List/Session List Utilization Check
Rules Engine Check

DBCheck and Oracle RDA

Database Performance
Statistics Dashboard Check
Partition Check (Oracle)
Trend Jobs Check
Hardware and Operating
System Check
CPU and Memory Utilization
Check

Oracle version and patch level

Event Persistence (insertion) Performance
check
Check
Oracle alert log check
Error Check
Oracle memory parameters
Scheduled Task Check
check
server.properties Check
ESM Database Storage Check

Agent and Console Threads Check

13

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Loggers
CPU, Memory, and EPS In/Out
Check
Search Performance Check
Custom Report Performance
Check
Receivers and Forwarders Check
Storage Group Check
Index Configuration Check
Configured Alerts Check
Scheduled Task Check
Event Archive and Configuration
Backup Check
Logger System Health and Audit
Event Forwarding Check
Network Configuration Check
Online Event Storage Check
(Only Software-based or SAN
Logger)

Connector appliances
Connector appliance version check
Is the version outdated?

Connector appliance network settings

check

Are there any known issues with the current version?

Common problems to check:

Connector appliance CPU and memory

check

Incorrect duplex settings on the network interface

DNS or NTP not configured properly

CPU utilization is continuously above 70-80% in Logger Dashboard

Connector appliance configuration backup

check

EPS In is continuously above 5,000 EPS (a single C5400 is designed

for 5,000 max EPS)

Check the Connector Appliances Monitor Dashboards for unusual

peaks or drops

The daily Configuration Backup job should be scheduled on

all Connector Appliances.

Check the System Process Status section of the Connector Appliance

If possible, SSH to the Connector Appliance and run commands such

as top, df, ifconfig, etc. to perform a deeper dive at the OS level

Review the following for excessive utilization:

14

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Additional resources

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

My favorite resources for keeping ArcSight healthy!

1. Any HP Protect presentation on ArcSight best practices or troubleshooting:
https://protect724.arcsight.com
2. KB Articles on the HP Support Site
3. Solutions listed in previous Support Tickets
4. HP ArcSight University
5. HP ArcSight product documentation

16

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Thank you

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Security for the new reality

Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.