You are on page 1of 96

IBM Security Server Protection for Windows

Administrator Guide for Server Protection for Windows Version 2.x

Copyright statement Copyright IBM Corporation 2005, 2011. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Contents
Overview . . . . . . . . . . . . . . v
What's new and changed in Server Protection for Windows . . . . . . . . . . . . . . . v How to Use Server Protection for Windows Documentation . . . . . . . . . . . . . vii Technical support contacts . . . . . . . . . vii

Chapter 4. Protecting against intrusions . . . . . . . . . . . . . 33


About security events . . . . . . . . Intrusion prevention or intrusion detection . Security events . . . . . . . . . . Descriptions for security event columns . Trusted addresses . . . . . . . . . Tuning the agent . . . . . . . . . Event filtering . . . . . . . . . . Advanced security event configuration . . Back tracing . . . . . . . . . . . Packet logging . . . . . . . . . . Evidence Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 35 35 36 36 37 39 40 40 41 42

Chapter 1. Introducing Server Protection for Windows . . . . . . . . . . . . . 1


Layered security . . . . About policy inheritance . . . . . . . . . . . . . . . . . . 1 . 3

Chapter 2. Deploying Server Protection for Windows agents . . . . . . . . . 5


Section A: Deployment scenarios . . . . . . . 5 Deployment scenario checklists . . . . . . . 6 Section B: Before you deploy an agent . . . . . . 7 Verifying agent licenses . . . . . . . . . . 8 Updating the SiteProtector database . . . . . 9 Updating the Agent Manager . . . . . . . . 9 Setting up SiteProtector Groups . . . . . . 10 Section C: Upgrading a version 1.0 agent to version 2.x . . . . . . . . . . . . . . . . . 11 Migrating policy settings from version 1.0 to version 2.x. . . . . . . . . . . . . . 12 Migrating policy settings from version 2.0 or 2.1 13 Process overview for upgrading use the heartbeat mechanism . . . . . . . . . . . . . 14 Task 1: Upgrading the agent version . . . . . 14 Task 2: Upgrading Policies . . . . . . . . 15 Process overview for upgrading using the agent build mechanism . . . . . . . . . . . 17 Upgrading Policies . . . . . . . . . . . 18 Section D: Configuring policies . . . . . . . . 19 Process overview for configuring policies . . . 19 Determining your policy inheritance needs . . . 19 Overriding parent policies . . . . . . . . 20 Configuring policies . . . . . . . . . . 20 Section E: Creating and Deploying an Agent Build 22 Generating an Agent Build . . . . . . . . 23 Installing an Agent . . . . . . . . . . . 24

Chapter 5. Protecting against buffer overflow exploits . . . . . . . . . . 43


Buffer Overflow Exploit Prevention . . . . . . 43 Understanding buffer overflow exploit prevention (BOEP) . . . . . . . . . . . . . . . . 44

Chapter 6. Enforcing antivirus compliance . . . . . . . . . . . . . 47


Understanding antivirus compliance . . . . Configuring antivirus compliance . . . . . Advanced antivirus compliance configuration . . . . . 47 . 48 . 49

Chapter 7. Enforcing application control . . . . . . . . . . . . . . . 51


Key concepts for application control . . . How to use application control . . . . . How to handle unknown applications . . How to handle known applications . . . Preparing and importing known applications Exporting event details from SiteProtector Editing the file . . . . . . . . . Importing known applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 52 53 54 56 57 57 58

Chapter 8. Auditing system integrity and policy compliance . . . . . . . . 59


Section A: Auditing operating system logs . . . Auditing Windows event logs . . . . . . Coalescing events . . . . . . . . . . Identifying a suspicious series of events . . . Creating exceptions to audit rules . . . . . Section B: Auditing registry entries . . . . . Monitoring the registry . . . . . . . . Creating exceptions to registry monitoring . . Section C: Auditing files and directories . . . . Monitoring directories and files. . . . . . File attributes the agent can monitor . . . . Importing a list of files to monitor . . . . . Scheduling a baseline comparison . . . . . Updating the baseline . . . . . . . . . Excluding directories and files from monitoring . . . . . . . . . . . . . . 59 60 60 61 62 62 63 63 64 64 65 66 67 67 68

Chapter 3. Firewall configuration . . . 25


Key concepts for firewall . . . . . Protection levels . . . . . . . . Firewall blocking . . . . . . . Firewall rules . . . . . . . . . Firewall precedence . . . . . . Examples of firewall precedence . Managing conflicts between firewall Opening a port through the firewall . Advanced firewall configuration . . . . . . . . . . . . . . rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 26 27 28 28 29 30 30 31

Copyright IBM Corp. 2005, 2011

iii

Chapter 9. Configuring bypass filters


Bypass filtering . . . . . . . . . . . .

69
. 69

Chapter 10. Configuring administrative settings . . . . . . . . . . . . . . 71


Configuring management settings . . . . . . Enabling agent protection . . . . . . . . Protecting during system start-up and shutdown . Setting up configuration sharing . . . . . . Configuring event caching . . . . . . . . . . . . . 71 73 74 75 76

Overriding parent policies does not give child policies with the parent policy . . . . . . . . Under heavy traffic conditions traffic seems to be bypassing analysis . . . . . . . . . . . . Not seeing any file integrity monitoring alerts . . . Traffic seems to be bypassing analysis . . . . . Refresh agent feature in the SiteProtector System not functioning . . . . . . . . . . . . . . Agent showing as offline . . . . . . . . . . System tray icon disappeared after upgrade . . .

79 80 80 81 82 82 83

Chapter 11. Troubleshooting . . . . . 77


Reporting a false positive. . . . . . . Not seeing any BOEP alerts . . . . . . Is Version 2.0 supported on ISA Server? . . How can I restore custom policy settings for group? . . . . . . . . . . . . . . . . . . . a child . . . . . . 77 78 78 79

Notices . . . . . . . . . . . . . . 85
Trademarks . . . . . . . . . . . . . . 86

Index . . . . . . . . . . . . . . . 87

iv

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Overview
This guide is designed to help you use the IBM Security Server Protection for Windows, Version 2.x agent to protect servers and ensure that they are in compliance with your corporate policy.

Scope
This guide describes the features of Server Protection for Windows agents and explains how to configure policies from SiteProtector and deploy agents.

Audience
This guide is intended for security managers who manage Server Protection for Windows agents from SiteProtector.

What's new and changed in Server Protection for Windows


This topic describes the new and changed features that are in the Server Protection for Windows release.

Updates for version 2.2


This release adds support for Windows Server 2008 R2 and includes enhancements to the following features: v Rebranded user interface The product formerly known as IBM Proventia Server Intrusion Prevention System for Windows is renamed to IBM Security Server Protection for Windows. v Improvements to File Integrity Monitoring The file integrity monitoring component in Server Protection for Windows now contains Recommended lists and User-defined lists on the Include and Exclude tabs. Each Include and Exclude tab now has an enabled checkbox and each of the Recommended and User-defined lists has a master enable checkbox. v Improvements to System Integrity Monitoring The system integrity monitoring component in Server Protection for Windows now includes new default system integrity monitoring events for all events introduced in Windows Server 2008 R2. v New default installation folder in Update Settings The default installation folder has been changed from C:\Program Files\ISS\Proventia Server to %ProgramFiles%\IBM\Server Protection.

Updates for version 2.1


This release adds support for Windows Server 2008 and includes the following updated features and enhancements: v Fail open You can instruct the agent to fail open or fail closed during overload conditions. This option can be controlled from the Administration policy on the Management tab. v Network monitoring You can instruct the agent to allow all network traffic to pass through without processing against the agent firewall, IPS, and IDS. This option can be controlled from the Administration policy on the Management tab. v Agent firewall
Copyright IBM Corp. 2005, 2011

You can control whether to use the Server Protection for Windows agent firewall or to use the Microsoft Windows Firewall. This option can be controlled from the Firewall policy. v Agent protection Agent protection is disabled by default. For more information about preventing unauthorized changes to agent files, see Enabling agent protection on page 73. v Event categorization for system integrity monitoring The system integrity monitoring component includes categorized events and the ability to filter events by operating system.

What's new in version 2.0


This release includes the following new features and enhancements: v Hierarchical policies Hierarchical policies provide an efficient way to use policies based on parent and child relationships. Child groups or child agents can inherit policies from parent groups, or you can override inheritance by defining a policy at the child level. v File Integrity Monitoring The file integrity monitoring component in Server Protection for Windows automates the detection of unplanned and unauthorized changes to files and directories so that you can identify and troubleshoot changes that may pose a threat to the system. v Registry Integrity Monitoring The registry integrity monitoring component in Server Protection for Windows monitors activity in the registry so that you can quickly identify potential threats to your system. v System Integrity Monitoring The system integrity monitoring component in Server Protection for Windows audits the system event logs for suspicious activity that may indicate a threat to the integrity of the host system or a violation of your security policy. v Packet bypass filtering The Server Protection for Windows agent can allow packets from certain IP addresses to bypass the protection offered by the firewall and the security event rules. You can use this feature to improve system performance by eliminating agent processing of trusted traffic such as traffic related to system backups. v Event filtering The Server Protection for Windows agent can ensure that traffic associated with useful or helpful addresses is not blocked by rules configured on the Security Events tab. You can use this feature to reduce false-positives and reduce the number of events sent to the Console. Consider filtering only those addresses that do not pose a threat to your system. Keep in mind that an intruder can spoof, or fake, the IP addresses of an internal system. v Antivirus compliance The Server Protection for Windows agent now prevents systems that are not in compliance with your antivirus policy from accessing certain resources on the network. The agent blocks outbound connections to any IP address specified on the IP Ranges tab in the Group Settings policy; other outbound connections are allowed so that the system can retrieve updates to the antivirus software and restore compliance. The agent also sends an alert to the console to notify you that the system is out of compliance. In Server Protection for Windows, Version 1.0, the agent blocked inbound connections to non-compliant systems.

vi

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

How to Use Server Protection for Windows Documentation


Use this documentation to help you set policies for Server Protection for Windows agents.

Related publications
The following documents are available for downloading from the IBM Security product Information Center at : http://publib.boulder.ibm.com/infocenter/sprotect/v2r8m0/index.jsp: v Server Protection for Windows System Requirements v Server Protection for Windows Administrator Guide v Server Protection for Windows User Guide v Server Protection for Windows Custom Parameters Help v IBM Security SiteProtector System Configuration Guide v IBM Security SiteProtector System Installation Guide v IBM Security SiteProtector System Policies and Responses Guide v IBM Security SiteProtector System User Guide for Security Analysts

License agreement
For licensing information on IBM Security Solutions products, download the IBM Licensing Agreement from: http://www.ibm.com/services/us/iss/html/contracts_landing.html

Technical support contacts


IBM Security Solutions provides technical support to customers that are entitled to receive support.

The IBM Support Portal


Before you contact IBM Security Solutions about a problem, see the IBM Support Portal at http://www.ibm.com/software/support

The IBM Software Support Guide


If you need to contact technical support, use the methods described in the IBM Software Support Guide at http://www14.software.ibm.com/webapp/set2/sas/f/handbook/home.html The guide provides the following information: v Registration and eligibility requirements for receiving support v Customer support telephone numbers for the country in which you are located v Information you must gather before contacting customer support

Overview

vii

viii

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 1. Introducing Server Protection for Windows


This chapter introduces the Server Protection for Windows agent and describes how the agent uses a layered security approach to protect your server. This chapter includes a process overview for using SiteProtector System to prepare, create, deploy, and manage agents.

Topics
Layered security About policy inheritance on page 3

Layered security
Server Protection for Windows provides multiple lines of defense by protecting the network vector, the application vector, and by monitoring files to detect unauthorized changes to the system. This layered protection approach provides greater overall protection for your host.

Protection from network threats


Network-based attacks use malicious network traffic to remotely compromise target systems. Unlike other threats, network-based attacks can penetrate a system, run, and propagate without human intervention. Network attack types include automated and direct attacks executed through worms, denial of service (DoS) attacks, and the installation of remote access backdoors. Server Protection for Windows provides the following components to prevent network-based attacks: v Firewall (FW) The firewall is the first line of defense against a network-based attack. The firewall can block inbound and outbound packets from particular IP addresses, port numbers, or protocols. By limiting inbound and outbound traffic to a set of predefined services and participants, organizations can minimize their exposure to threats. v Intrusion Prevention After authorized packets pass through the firewall the agent analyzes them for malicious content. The agent drops offending packets, and allows the original traffic to continue through the network stack unhindered. If the agent detects malicious content that poses an immediate threat to the system, it can block the packet and reset the connection. The agent can also configure the firewall to block all traffic to or from the intruder. This layer of defense is particularly effective because it works at the network-level. Inbound packets are immediately inspected before they can traverse to vulnerable sections of the operating system. As a result, this protection provides the equivalent of in-line inspection. v Buffer Overflow Exploit Prevention (BOEP) The buffer overflow exploit prevention (BOEP) component is the last layer of preventative defense. The BOEP component monitors for attacks that overrun writable memory regions or that attempt to access unauthorized execution spaces. Note: Buffer Overflow Exploit Prevention is not currently supported on any 64-bit system or on Windows Server 2008 32-bit systems.

Copyright IBM Corp. 2005, 2011

Protection from application threats


Application-based attacks use program files (or attachments) to attack and compromise target systems. Unlike network-based attacks, program files typically require some form of user involvement to initiate an attack. Email, files intentionally and unintentionally downloaded from the Internet, injected Web sites, popup windows, removable media, and peer-to-peer applications are common sources of application-based attacks. Server Protection for Windows provides the following components to protect against application-based attacks: v Antivirus (AV) Compliance Antivirus software is the computer's first layer of defense against an application-based attack. Signature-based antivirus products are extremely effective in detecting and preventing known viruses, worms, and some Trojans. The Server Protection for Windows agent can monitor the host system for the presence of antivirus software and required updates. You can block network access until the antivirus software has been updated. v The Application Control (AC) Application control is the last layer of preventative defense from an application-based attack. When an unknown application tries to start, the agent can terminate the application or allow it to run. When an unauthorized application tries to connect to a network, the agent can block its network access, terminate it, or allow it to connect. Note: Application control is not currently supported on 64-bit systems or Windows Server 2008.

Auditing and compliance protection


Audit-based protection detects and alerts on unauthorized changes that can indicate that a host has been compromised. Unauthorized changes represent one of the largest threats to the security of a system. Auditing system integrity is a powerful layer of defence for your infrastructure. Server Protection for Windows provides the following components to protect against unauthorized changes to files: v File Integrity Monitoring The file integrity monitoring component automates the detection of unplanned and unauthorized changes to files so that you can identify and troubleshoot changes that may pose a threat to the system. v Registry Integrity Monitoring The Windows registry contains highly sensitive data, such as user profiles and the ports used by the system, so it is essential that you secure the registry. You can provide a certain level of security by setting the appropriate access rights to subkeys and registry entries; however, you cannot completely lock down the registry because users and programs need access to certain areas to function correctly. Because you must allow some access to the registry, it is essential to monitor activity in the registry so that you can quickly identify potential threats to your system. v System Integrity Monitoring System event logs can hold important clues to problems you may be having with your computer or the applications that run on your computer. Auditing the event logs can help you identify, diagnose, and minimize potential threats to system integrity.

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

About policy inheritance


Policy inheritance provides an efficient way to use policies based on parent and child relationships. Child groups or child agents can inherit policies from parent groups, or you can override inheritance from a parent by defining a policy at the child level. An agent or group uses the policy defined at its own level or, if there is no policy defined at this level, the policy defined at the nearest parent node where a policy is defined. Important: Policy inheritance is a powerful way to manage agent policies. The effects of inheritance can be far reaching. Review the policy inheritance information in the SiteProtector System Policies and Responses Configuration Guide before you configure your policies.

Example
You configure the policies for a group of assets called Mail Servers. There are ten subgroups in the Mail Servers group. If all of the subgroups use the same policy settings, the policy settings can pass from the Mail Servers group to all ten subgroups. If certain subgroups in the Mail Servers group require customized policy settings, you can define a customized policy at the subgroup level for that policy without affecting the policy settings for the other subgroups. This approach is much quicker than configuring the policy settings for all ten subgroups.

Policies that are displayed in the left pane


If a policy is defined for a particular group or agent, this policy is listed as a member of the group or agent when you expand the group or agent in the navigation pane. When you select a policy in the navigation pane, the contents for that policy displays in the right pane.

Policies that are displayed in the right pane


When you select a group or agent in the tree, a list of policies that are in use for the selected object appears in the right pane.

Overriding a policy for a group or agent


If you need to customize a policy for an agent in a group or for a group in a Site, you can define a different policy (override the current policy) at whichever level you need. Reference: See the SiteProtector System Policies and Responses Configuration Guide.

When are policy updates applied to the agent?


When you make changes to a policy, you save the changes. It might, however, take some time before you see a change in agent behavior for the following reasons: v The agent only periodically checks for policy updates (based on the heartbeat interval) Note: You can force an agent to apply an updated policy immediately by using the SiteProtector System Refresh Agent feature. v Large groups of agents take longer to update

Chapter 1. Introducing Server Protection for Windows

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 2. Deploying Server Protection for Windows agents


Deploying Server Protection for Windows agents is a multi-stage process that includes updating SiteProtector components, configuring policies, and deploying configured policies to the host system.

Pilot program
Consider running a pilot program before you deploy Server Protection for Windows agents in your production environment. A pilot program is a small scale deployment of agents (usually on non-production systems) that allows you to test different policy settings in relative safety. During the pilot, you can collect information and use it to modify your policies to minimize usability problems when you deploy agents in a production environment.

Deployment scenario checklists


Refer to Deployment scenario checklists on page 6 to determine how best to use the information in this chapter.

Topics
Section A: Deployment scenarios Section B: Before you deploy an agent on page 7 Section C: Upgrading a version 1.0 agent to version 2.x on page 11 Section D: Configuring policies on page 19 Section E: Creating and Deploying an Agent Build on page 22

Section A: Deployment scenarios


This section includes deployment scenario checklists to guide you through the deployment process.

Topics
Deployment scenario checklists on page 6

Copyright IBM Corp. 2005, 2011

Deployment scenario checklists


Use the appropriate deployment scenario checklist in this topic to guide you through the deployment process.

Installing an agent for the first time


Use the following checklist to help you deploy an agent for the first time:
U h h h Complete the tasks outlined in... Section B: Before you deploy an agent on page 7 Section D: Configuring policies on page 19 Section E: Creating and Deploying an Agent Build on page 22

Updating a 1.0 agent to 2.x using the heartbeat mechanism


Use the following checklist to help you update an agent:
U h h h h Complete the tasks outlined in... Section B: Before you deploy an agent on page 7 Migrating policy settings from version 1.0 to version 2.x on page 12 Migrating policy settings from version 2.0 or 2.1 on page 13 Process overview for upgrading use the heartbeat mechanism on page 14

Updating a 1.0 agent to 2.x using the agent build mechanism


Use the following checklist to help you update an agent:
U h h h h h Complete the tasks outlined in... Section B: Before you deploy an agent on page 7 Migrating policy settings from version 1.0 to version 2.x on page 12 Migrating policy settings from version 2.0 or 2.1 on page 13 Process overview for upgrading using the agent build mechanism on page 17 Section E: Creating and Deploying an Agent Build on page 22

Updating an earlier 2.x agent to a later 2.x agent using the heartbeat mechanism
U h h h Complete the tasks outlined in... Updating the SiteProtector database on page 9 Updating the Agent Manager on page 9 Configuring policies on page 20

Updating an earlier 2.x agent to a later 2.x agent using the agent build mechanism
U h h h Complete the tasks outlined in... Updating the SiteProtector database on page 9 Updating the Agent Manager on page 9 Configuring policies on page 20
Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

U h

Complete the tasks outlined in... Section E: Creating and Deploying an Agent Build on page 22

Updating from a 2.0 agent or a 2.1 agent using the heartbeat mechanism
U h h h h h Complete the tasks outlined in... Updating the SiteProtector database on page 9 Updating the Agent Manager on page 9 Migrating policy settings from version 2.0 or 2.1 on page 13 Process overview for upgrading use the heartbeat mechanism on page 14 Configuring policies on page 20

Updating from a 2.0 agent or a 2.1 agent using the agent build mechanism
U h h h Complete the tasks outlined in... Updating the SiteProtector database on page 9 Updating the Agent Manager on page 9 Migrating policy settings from version 2.0 or 2.1 on page 13 Process overview for upgrading using the agent build mechanism on page 17 h h Configuring policies on page 20 Section E: Creating and Deploying an Agent Build on page 22

Section B: Before you deploy an agent


This section outlines the tasks you must complete before you can install or upgrade an agent on your computer.

Topics
Verifying agent licenses on page 8 Updating the SiteProtector database on page 9 Updating the Agent Manager on page 9 Setting up SiteProtector Groups on page 10

Chapter 2. Deploying Server Protection for Windows agents

Verifying agent licenses


About this task
You must have valid licenses before you can install Server Protection for Windows agents.

Verifying Agent Licenses


Procedure
1. In SiteProtector, click Tools > Licenses > Agent/Module. The Agent/Module License Information for Site window appears. 2. Select the Licenses tab. 3. In the Agent Type column, locate the appropriate row, and then verify the following columns in that row: v License Limitdisplays the number of available licenses v Statedisplays the status of the license; if the status is Key Good, the license is valid. Note: If there is any other status, refer to the SiteProtector Help.

Obtaining a license file


Before you begin
You should have received an email containing the license key for your agents.

Adding agent licenses


Procedure
1. In the navigation pane, select the Site Node. 2. Select Tools > Licenses > Agent/Module. The Agent/Module License Information for Site window appears. 3. Select the Licenses tab, and then click Add. 4. Navigate to the folder that contains the license you want to add, select the license file, and then click OK. Note: License key files have either a .key or .isslicense file extension. The End User License Agreement appears. 5. Click Accept. The license appears in the License tab.

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Updating the SiteProtector database


IBM Security Solutions delivers the policies for Server Protection for Windows agents through a database update package. You must update the SiteProtector system database before you can deploy Server Protection for Windows, Version 2.x agents.

About this task


Verifying and updating the database involves the following tasks:
U h h Task Verify the SiteProtector system has downloaded the database service pack. Apply the database service pack. Reference Task 1: Verifying the status of the database Task 2: Applying the database service pack

Task 1: Verifying the status of the database


Procedure
1. In the SiteProtector system, select the applicable group, and then select the Agent view. 2. In the Update Status column, verify the status of the database as follows: v If the status is Out of Date, there is an update available. Go to Task 2: Applying the database service pack to continue. v If the status is Current, the update has already been applied. Go to Updating the Agent Manager.

Task 2: Applying the database service pack


Procedure
1. In the SiteProtector system, select the applicable group, and then select the Agent view. 2. Right-click SiteProtector Database, and then click Updates > Apply XPU.

Updating the Agent Manager


IBM Security Solutions delivers the software for Server Protection for Windows agents through an Agent Manager update package. You must update the Agent Manager before you can deploy Server Protection for Windows, Version 2.x agents.

About this task


Verifying and updating the Agent Manager XPU involves the following tasks:
U h h Task Verify the SiteProtector system has downloaded the Agent Manager XPU. Apply the Agent Manager XPU. Reference Task 1: Verifying the status of the Agent Manager Task 2: Applying the Agent Manager XPU

Chapter 2. Deploying Server Protection for Windows agents

Task 1: Verifying the status of the Agent Manager


Procedure
1. In the SiteProtector system, select the applicable group, and then select the Agent view. 2. Verify the status of the Agent Manager in the Update Status column as follows: v If the status is Out of Date, there is an update available. Go to Task 2: Applying the Agent Manager XPU to continue. v If the status is Current, the update has already been applied. Go to Migrating policy settings from version 1.0 to version 2.x on page 12.

Task 2: Applying the Agent Manager XPU


Procedure
1. In the SiteProtector system, select the applicable group, and then select the Agent view. 2. Right-click Agent Manager, and then select Updates > Apply XPU.

Setting up SiteProtector Groups


About this task
A group is a collection of network assets and the SiteProtector components or agents that reside on those assets. A well designed group structure can help you manage your security more efficiently. Note: If you have already created your group structure within SiteProtector, refer to your deployment scenario checklist on page 18 for your next step. Groups provide the following: v A means for organizing, accessing, and managing important information about the network assets monitored by SiteProtector and other IBM ISS products. v A means for organizing and managing the other IBM ISS products that work with SiteProtector. v A means for performing SiteProtector tasks on groups of assets and agents, such as applying policies to agents in a group or viewing the security events for a specific group of assets. v A navigational tool in the Console that you can use to move between different groups of network assets and agents as you perform your security management tasks. Note: If you have already created your group structure within SiteProtector, go to Process overview for configuring policies on page 19.

10

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Adding a new group


Procedure
1. In the navigation pane, do one of the following: v to add the first group in a siteright-click the Site group, and then select New > Group. v to add a group to an existing groupright-click a group, and then select New > Group. The New Group folder appears below the selected group. 2. Type the group name in the highlighted box, and then press ENTER. Reference: For additional information about setting up groups in SiteProtector, see the SiteProtector Configuration Guide. The group appears in the navigation pane.

What to do next
Refer to your deployment scenario checklist in Deployment scenario checklists on page 6.

Section C: Upgrading a version 1.0 agent to version 2.x


If you already have Version 1.0 agents deployed, you can upgrade these agents to Version 2.x. Upgrading your agents to the latest version allows you to use the new features to protect your computer.

Prerequisites
See Section A: Deployment scenarios on page 5. See Section B: Before you deploy an agent on page 7.

Upgrading methods
You can upgrade Server Protection for Windows agents in the following ways: v Heartbeat mechanism v Agent build mechanism

Choosing the right method


Choose the method suitable to you based on the following: v For systems that are difficult to access, use the heartbeat mechanism v For all other systems, use either mechanism

Topics
Migrating policy settings from version 1.0 to version 2.x on page 12 Migrating policy settings from version 2.0 or 2.1 on page 13 Process overview for upgrading use the heartbeat mechanism on page 14 Task 1: Upgrading the agent version on page 14 Task 2: Upgrading Policies on page 15 Process overview for upgrading using the agent build mechanism on page 17

Chapter 2. Deploying Server Protection for Windows agents

11

Upgrading Policies on page 18

Migrating policy settings from version 1.0 to version 2.x


Before you begin
See Section B: Before you deploy an agent on page 7.

About this task


Reference: See About policy inheritance on page 3 for a high-level overview of policy inheritance; see the SiteProtector Policies and Responses Configuration Guide for a more in-depth discussion on policy inheritance. Settings configured from the local console are not migrated as part of this upgrade. If you want an agent to continue using settings set from the local console after this upgrade, you must record the local settings before you upgrade the agent. After the upgrade, you must reapply the settings from the local console.

Procedure
1. In SiteProtector, select the Agent view. 2. In the navigation pane, right-click the applicable group, and then select Updates > Migrate Agent Version. The Migrate Agent Version window appears. 3. Click the Details icon, and then select Proventia Server for Windows from the Agent Type list. 4. 5. 6. 7. 8. Click the Upgrade Details icon. In the Migrate From Agent Version list, select 1.0. In the Update To Agent Version list, select a 2.x version. Click OK. Do one of the following: v To complete the upgrade using the heartbeat mechanism, go to Process overview for upgrading use the heartbeat mechanism on page 14 v To complete the upgrade using the Agent Build mechanism, go to Process overview for upgrading using the agent build mechanism on page 17

12

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Migrating policy settings from version 2.0 or 2.1


Before you begin
See Section B: Before you deploy an agent on page 7.

About this task


Important: If you are upgrading your agents to version 2.2, you must upgrade the agent versions before upgrading from Windows 2008 SP2 to Windows 2008 R2. Settings configured from the local console are not migrated as part of this upgrade. If you want an agent to continue using settings set from the local console after this upgrade, you must record the local settings before you upgrade the agent. After the upgrade, you must reapply the settings from the local console.

Procedure
1. In the SiteProtector System, select the Agent view. 2. In the navigation pane, right-click the applicable group, and then select Updates > Migrate Agent Version. The Migrate Agent Version window appears. 3. Click the Details icon, and then select Server Protection for Windows from the Agent Type list. 4. Click the Upgrade Details icon. 5. In the Migrate From Agent Version list, select 2.0 or 2.1. 6. In the Update To Agent Version list, select 2.1 or 2.2. 7. Click OK. 8. Do one of the following: v To complete the upgrade using the heartbeat mechanism, go to Process overview for upgrading use the heartbeat mechanism on page 14 v To complete the upgrade using the Agent Build mechanism, go to Process overview for upgrading using the agent build mechanism on page 17

Chapter 2. Deploying Server Protection for Windows agents

13

Process overview for upgrading use the heartbeat mechanism


This topic provides an overview of the tasks involved in using the heartbeat mechanism to upgrade agents from Version 1.0, Version 2.0, or Version 2.1.

Prerequisites
See Section B: Before you deploy an agent on page 7. See Migrating policy settings from version 1.0 to version 2.x on page 12. See Migrating policy settings from version 2.0 or 2.1 on page 13.

Process overview
Upgrading Server Protection for Windows agents to version 2.0, version 2.1, or version 2.2 using the heartbeat mechanism involves the following tasks:
Task 1 2 Description Upgrade the agent version Upgrade policies Reference Task 1: Upgrading the agent version Task 2: Upgrading Policies on page 15

Task 1: Upgrading the agent version


You must define the new server protection version to upgrade agents. The server protection version defines the version number of the agents and allows you to manage more than one version of agents from the same system.

About this task


The following table outlines the process for upgrading the server protection version:
h h h Task Update the server protection version Verify the agent version Reference Task 1: Updating the server protection version Task 2: Verifying the agent version

14

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Task 1: Updating the server protection version


Procedure
1. 2. 3. 4. 5. In the SiteProtector System, select the applicable group, and then select the Policy view. In the Agent Type list, select Server Protection for Windows. If you are migrating from version 1.0, in the Agent Version list, select 1.0. If you are migrating from version 2.0, in the Agent Version list, select 2.0 If you are migrating from version 2.1, in the Agent Version list, select 2.1

6. In the navigation pane, right-click Policy and then select Open. The Policy pane opens in the right pane. 7. Select Administrative Settings > Group Configuration. The Group Configuration pane appears. 8. For 1.0 agents migrating to version 2.0, in the Server protection version list, select the version that starts with 2.0. 9. For 2.0 agents migrating to version 2.1, in the Server protection version list, select the version that starts with 2.1. 10. For 2.1 agents migrating to version 2.2, in the Server protection version list, select the version that starts with 2.2. 11. In the Install path box, type the path to where the agents in this group are installed. 12. Save the policy. 13. Do one of the following to apply the updated policy: v Wait for the agents to send a heartbeat to the SiteProtector System v Select the applicable group, and then select Action > Refresh Agent to force the agents to send a heartbeat to the SiteProtector System immediately Task 2: Verifying the agent version: Procedure 1. In the SiteProtector System, select the applicable group, and then select the Agent view. 2. In the Version column, verify the agent version.

Task 2: Upgrading Policies


About this task
Policy settings determine the behavior of agents. As part of the upgrade you must, at a minimum, configure the Install and Update Settings policy. This policy defines the Agent Version setting which defines both the version of the agent software and the build version of that software. This policy also defines the installation directory for the agent on the host system. Tip: You can also configure the settings for any other policies. You can edit policies at the parent group level (settings are inherited by any child groups), or you can override the parent group policies and edit policies at the child-group level (settings are specific to the child group only).

Chapter 2. Deploying Server Protection for Windows agents

15

Editing policies
Procedure
1. 2. 3. 4. 5. In the SiteProtector System, select the Agent view. In the navigation pane, right-click the applicable group, and then select Manage Policy. In the Agent Type list, select Proventia Server for Windows. For version 2.0 agents, in the Agent Version list, select 2.0. For version 2.1 agents, in the Agent Version list, select 2.1.

6. For version 2.2 agents, in the Agent Version list, select 2.2. 7. In the navigation pane, select the group. The Active Deployments window for the group appears on the right pane. 8. In the right pane, right-click Install and Update Settings, and select Open. 9. In the Agent Version list, select the agent version. 10. In the Install path box, type the path to where the agents in this group are installed. 11. Click the Save icon. 12. From the Save Policy Version window, enter a Comment and then select the Deploy This New Version check box. 13. Click OK. The Deploy Policy window opens. 14. Click Targets. 15. Select the group you want to deploy this policy to. 16. Click OK and then close the tab. 17. To edit additional policies, do the following: v In the left pane, re-select the applicable group v In the right pane, right-click the policy to edit, and then select Open v Edit the policy Reference: See the Help for guidance. 18. Click the Save icon. 19. Repeat Step 12 through Step 18 for each policy you want to edit. 20. Do one of the following to apply the updated policy: v Wait for the agents to send a heartbeat to the SiteProtector System v Select the applicable group, and then select Action > Refresh Agent to force the agents to send a heartbeat to the SiteProtector System immediately

What to do next
v If you are performing the steps to upgrade from version 1.0 to version 2.0, you are finished. v If you are performing the steps to upgrade from version 2.0 to version 2.1 or version 2.2, you must complete the procedure for Migrating policy settings from version 2.0 or 2.1 on page 13.

16

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

System tray icon


After you upgrade agents using the heartbeat mechanism, the system tray icon is not visible on the host system. If you open the console from the Start menu, the icon will again be visible in the system tray. This only happens immediately following the update to 2.0.

Process overview for upgrading using the agent build mechanism


This topic provides an overview of the tasks involved in using the agent build mechanism to upgrade agents from Version 1.0, Version 2.0, or Version 2.1.

Prerequisites
See Section B: Before you deploy an agent on page 7. See Migrating policy settings from version 1.0 to version 2.x on page 12. See Migrating policy settings from version 2.0 or 2.1 on page 13.

What is an agent build?


An agent build is a pre-configured installation package that includes the following software and user-defined information: v The actual Server Protection for Windows software that the build installs on the computer v Information about how the agent should communicate with the SiteProtector System, including the IP address of the Agent Manager it reports to and the account name and password it should use when it contacts the Agent Manager v Policies that define the protection provided by the agent

Process overview
The following table provides an overview of the stages involved in upgrading agents using the agent build mechanism:
Task 1 2 3 Description Upgrade policies Generate an Agent Build Install agents using the Agent Build Reference Upgrading Policies Generating an Agent Build Installing an agent

Chapter 2. Deploying Server Protection for Windows agents

17

Upgrading Policies
About this task
Policy settings determine the behavior of agents. As part of the upgrade you must, at a minimum, configure the Install and Update Settings policy. This policy defines the Agent Version setting which defines both the version of the agent and the version of the agent build. This policy also defines the installation directory for the agent on the host system. Tip: You can also configure the settings for any other policies. You can edit policies at the parent group level (settings are inherited by any child groups), or you can override the parent group policies and edit policies at the child-group level (settings are specific to the child group only).

Editing parent level policies


Procedure
1. In SiteProtector, select the Agent view. 2. In the navigation pane, right-click the applicable group, and then select Manage Policy. 3. In the Agent Type list, select Server Protection for Windows. 4. For version 2.0 agents, in the Agent Version list, select 2.0. 5. For version 2.1 agents, in the Agent Version list, select 2.1. 6. For version 2.2 agents, in the Agent Version list, select 2.2. 7. In the navigation pane, select the group. The Active Deployments window for the group appears on the right pane. 8. In the right pane, right-click Install and Update Settings, and then select Open. 9. In the Agent Version list, select the agent version. 10. In the Install path box, type the path to where the agents in this group should be installed. 11. Click the Save icon. 12. From the Save Policy Version window, enter a Comment and then select the Deploy This New Version check box. 13. Click OK. The Deploy Policy window opens. 14. Click Targets. 15. Select the group you want to deploy this policy to. 16. Click OK and then close the tab. 17. To edit additional policies, do the following: v In the left pane, re-select the applicable group v In the right pane, right-click the policy to edit, and then select Open v Edit the policy Reference: See the Help for guidance. 18. Click the Save icon. 19. Repeat Step 12 through Step 18 for each policy you want to edit.

What to do next
See Section E: Creating and Deploying an Agent Build on page 22

18

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Section D: Configuring policies


Policies define how the agent protects your server. By configuring policies and deploying them to an agent, you determine the level of protection the agent provides.

Topics
Process overview for configuring policies Determining your policy inheritance needs Overriding parent policies on page 20 Configuring policies on page 20

Process overview for configuring policies


Configure agent policies so that the agent can protect your system as soon as the policies are deployed.

Process overview
The following table outlines the tasks to perform to configure policies:
Task 1 2 3 Description Determine your policy inheritance needs Implement your policy structure Configure the policies Reference Determining your policy inheritance needs Overriding parent policies on page 20 Configuring policies on page 20

Determining your policy inheritance needs


Policy inheritance provides an efficient way to use policies based on parent and child relationships. Child groups, by default, inherit policies from the nearest parent group. Child groups can, however, override the parent policies they inherit so that you have more granular control of settings. Policy settings in one child group do not affect policy settings in the parent group or policy settings in any sibling groups. Use policy inheritance to implement the policy structure that best meets your security needs. Agents can use the following policy structures: v All policies are inherited from a parent group v All policies override the parent group policies v A combination of inherited and overridden policies Reference: For more information about policy inheritance, see the SiteProtector Configuration Guide.

When to override parent policies


Override parent policies at the child level when you need a custom policy setting at the child level.

Considerations
To maximize efficiency, you should configure the parent policy, override the parent policy from the child, and then customize the child policy. To make configuration changes easier to implement, inherit as many policies as possible from the parent group. At a minimum, consider inheriting policies that define the software version and installation path information for the agent; you may want to override policies that define protection settings so you can have customized protection for child groups.
Chapter 2. Deploying Server Protection for Windows agents

19

Overriding parent policies


About this task
If you determine that you need custom policy settings at the child level, you must override the applicable parent level policy. You cannot implement policy customizations at the child level until you override the parent level policy. Tip: Inherit as many policies as possible from the parent group as this makes configuration changes easier to implement. Consider inheriting at least the Update Settings policy as this ensures you are running the same version of the agent on all systems and that the agent is installed in the same directory on each system.

Procedure
1. In the navigation pane, right-click a group and select Manage Policy. 2. In the Agent Type list, select Server Protection for Windows. 3. For agent version 2.0, in the Agent Version list, select 2.0. 4. For agent version 2.1, in the Agent Version list, select 2.1. 5. For agent version 2.2, in the Agent Version list, select 2.2. 6. In the navigation pane, select the child group you want custom policies for. Note: The Active Deployments reflects the current policies assigned to this child group. The Active Deployments window for the group appears in the right pane. 7. In the right pane, select the policy that you want to override. Tip: Use the Inheriting From column to determine which parent policies the child currently inherits. 8. Right-click the selected policies, and then click Deploy. 9. Click OK. The selected policy appears under the child group folder in the navigation pane.

Configuring policies
After you have implemented your policy inheritance structurethat is, you have decided which policies to inherit from the parent level and which policies to override at the child levelyou must configure the policies so the agent can protect your system.

About this task


Policy management in SiteProtector 2.0 Service Pack 7.0 changes significantly. You must now save and deploy each policy after you have finished configuring it. Important: While you can use the following procedure to guide you through the configuration of both parent level policies and child level policies, be aware that changes to parent level policies affect all child groups.

Procedure
1. 2. 3. 4. 5. In the navigation pane, right-click a group and select Manage Policy. In the Agent Type list, select Server Protection for Windows. For version 2.0 agents, in the Agent Version list, select 2.0. For version 2.1 agents, in the Agent Version list, select 2.1. For version 2.2 agents, in the Agent Version list, select 2.2.

6. In the navigation pane, expand the group you are configuring policies for.

20

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

7. If you are configuring policies for a parent group, select Default Repository. Important: If the group you are configuring does not contain an Update Settings policy (either by inheriting or by having a custom version), you must create this policy before you can deploy your agent. See Update Settings in the table below for more information. The default policy types appear in the right pane. 8. Configure the appropriate policies based on the following information: Tip: As you configure each policy, press F1 if you need Help.
For this policy... Administration Application Compliance BOEP Bypass Filters File Integrity Monitoring Firewall Group Settings do this... see Chapter 10, Configuring administrative settings, on page 71 see Chapter 6, Enforcing antivirus compliance, on page 47 and Chapter 7, Enforcing application control, on page 51 see Chapter 5, Protecting against buffer overflow exploits, on page 43 see Chapter 9, Configuring bypass filters, on page 69 see Section C: Auditing files and directories on page 64 see Chapter 3, Firewall configuration, on page 25 1. Select the Agent Manager List tab. 2. If the Agent Manager for this group is not in the list, click Add. 3. Click Choose an Agent Manager, select the Agent Manager for this group, and then click OK. 4. Select a trust level as follows: v Trust all SiteProtector trusts the server and does not try to validate the certificate v First time trust SiteProtector trusts the first certificate it receives from the server and stores this certificate locally. The client uses this certificate to validate all future communication with this server. v Explicit trust The certificate for the server must reside in a local SiteProtector directory before the agent can initiate communication with the server. Typically, the certificate is transferred to the client outside the standard communication channels. 5. Click OK. Update Settings Tip: Consider inheriting this policy from the parent to ensure all agents are running the same version and build of the software and that all agents are installed in the same directory. 1. If you do not see this policy, expand Policy Types Not Created, right-click the policy, and then select New Policy. 2. In the Agent Version list, select the agent version and build. 3. In the Install path box, type the location where the agent will be installed on the host system. Registry Integrity Monitoring Security Events System Integrity Monitoring see Section B: Auditing registry entries on page 62 see Chapter 4, Protecting against intrusions, on page 33 see Section A: Auditing operating system logs on page 59

9. Click the Save icon. 10. From the Save Policy Version window, enter a Comment and then select the Deploy This New Version check box. 11. Click OK. The Deploy Policy window opens. 12. Click Targets.
Chapter 2. Deploying Server Protection for Windows agents

21

13. Select the applicable group to which you want to deploy the policy. 14. Click OK. 15. Do one of the following: v If you are deploying policies using an agent build, go to Section E: Creating and Deploying an Agent Build Note: If you are deploying an agent for the first time, you must use the agent build mechanism to deploy the agent. v If you are deploying policies using the heartbeat mechanism, do one of the following: Wait for the agents to send a heartbeat to SiteProtector Select the applicable group, and then select Action > Refresh Agent to force the agents to send a heartbeat to SiteProtector immediately

Section E: Creating and Deploying an Agent Build


You can install Server Protection for Windows agents using an agent build created from SiteProtector. This section provides the process and procedures for creating an agent build and for installing Server Protection for Windows, Version 2.x using that agent build.

What is an agent build?


An agent build is a preconfigured installation package that includes the following software and user-defined information: v The actual Server Protection for Windows software that the build installs on the computer v Information about how the agent should communicate with SiteProtector, including the IP address of the Agent Manager it reports to and the account name and password it should use when it contacts the Agent Manager v Policies that define the protection provided by the agent

Prerequisites
Review the appropriate deployment scenario checklists before using the information in this section to create and deploy an agent build. Reference: Deployment scenario checklists on page 6

Topics
Generating an Agent Build on page 23 Installing an Agent on page 24

22

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Generating an Agent Build


About this task
An agent build is a preconfigured installation package that you can use to install Server Protection for Windows agents. The agent build contains the agent software and the policies you configured to define the behavior of the agent.

Generating an Agent Build


Procedure
1. Select Object > New Tab > Agent. 2. In the navigation pane, right-click the group, and then select Generate Agent Build. The Generate Agent Build window appears. 3. Click the Details icon. 4. In the Command Details section, select Server Protection for Windows from the Agent Type list. 5. Select the Agent Version. 6. Click the Build icon. 7. In the Agent Manager list, do one of the following: v Verify the Agent Manager listed is the correct Agent Manager for this group v Select an Agent Manager from the Agent Manager list 8. Type a Description for this build. 9. Click OK.

Checking the agent build generated successfully


Procedure
1. In the navigation pane, right-click the group, and then select Properties. The group properties page appears. 2. In the left pane, click Command Jobs. The Command Jobs pane appears. 3. Sort the Command Jobs by the Start Time, and then identify the latest agent build in the Command column. 4. Click the agent build command job. The details of the command job appear in the lower pane. 5. Confirm that the Status is Completed. Note: If the status is either Pending or Processing, then press F5 to refresh the status. Important: You should not distribute the agent build until the status is Completed.

Chapter 2. Deploying Server Protection for Windows agents

23

Installing an Agent
About this task
After you have created the agent build, you can access the build from the host system and install the agent on the server.

Procedure
1. On the computer where you want to install the Server Protection for Windows agent, start Internet Explorer. 2. Type the URL to the SiteProtector system using the following format: http://Agent_Manager_IP_address:8085 The Agent Manager Available Downloads Web page opens. 3. Select the agent build you want to install, and then click Download. 4. Save the agent build program file to the local system, and then double-click the file to begin the installation process.

What to do next
After you have installed Server Protection for Windows, you can use the following list to verify the installation completed successfully:
Look here... System tray Task manager to verify that... the Server Protection for Windows icon is glowing (if you installed the local user interface) v on Windows Server 2000 and 2003 32 bit systems, blackd, blackice, RapApp, and ph processes are running v on Windows Server 2000 and 2003 64 bit systems, blackd, blackice, and ph processes are running v on Windows Server 2008, blackd, blackice, and ph processes are running Server Protection for Windows Services the agent name, the IP address, and an Active status are showing in the Agent view v on Windows Server 2000 and 2003 32 bit systems blackice, RapApp, and IBM Proventia services have started v on Windows Server 2000 and 2003 64 bit systems blackice, and IBM Proventia services have started v on Windows Server 2008 blackice, and IBM Proventia services have started Events tab in the local UI an alert indicates agent-startup was successful (if you installed the local user interface)

If no services and processes start, look in the Windows directory for the AgentUpdateYour_System_Name.log for possible issues with the installation.

24

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 3. Firewall configuration


The firewall is the computer's first line of defense against a network-based attack. You can create firewall rules as part of the central policy, as part of the local policy (if your policy allows for local management), or automatically when the agent detects a threat.

Topics
Key concepts for firewall Protection levels on page 26 Firewall blocking on page 27 Firewall rules on page 28 Opening a port through the firewall on page 30 Advanced firewall configuration on page 31

Key concepts for firewall


The firewall is the computer's first line of defense against a network-based attack.

Disabling the agent firewall


Note: This feature applies to Version 2.1 and 2.2. If the Microsoft Windows firewall is running, the Server Protection for Windows agent disables the Windows firewall to avoid conflicts, and uses its own firewall instead. If the agent is stopped or fails, the agent restores the Microsoft Windows firewall until the agent is restarted. Any configuration settings for the Microsoft Windows firewall are preserved.

Level of configuration
The level of configuration required for the firewall varies depending on the level of protection you want to enforce. Choosing a protection level quickly implements perimeter security. Defining firewall rules provides more granular control.

Copyright IBM Corp. 2005, 2011

25

Protection levels
A protection level is a set of predefined firewall rules that provide a certain level of security. Protection levels allow or restrict access to the system based on port and protocol. For example, at the Paranoid level, the agent blocks incoming traffic on all TCP and UDP ports. Important: Protection levels do not block outbound traffic. To block outbound traffic, you must define outbound firewall rules.

Protection level settings


The following table describes the predefined protection levels:
Level Paranoid Nervous Description Blocks all TCP ports and all UDP ports; blocking all unsolicited inbound traffic. Blocks all TCP ports and blocks UDP ports 0-1023. This effectively blocks all unsolicited inbound traffic except for some interactive content on Web sites (such as streaming media and other application-specific Internet usage). Blocks TCP and UDP ports 0-1023, effectively blocking unsolicited network traffic that accesses the operating system and networking services. All ports remain open and unblocked allowing all inbound traffic.

Cautious

Trusting

Considerations
Keep the following in mind when you set up protection levels: v Ports 137 and 138 accept network traffic when Network Neighborhood lookup is enabled and they block traffic when the feature is disabled. v Ports 139 and 445 accept network traffic when file sharing is enabled and they block traffic when file sharing is disabled. v For computers running Windows Server 2000, Windows Server 2003, or Windows Server 2008 both ports 139 and 445 are used for the SMB file sharing protocol. Therefore, when default firewall settings from the management server are applied to the local agent, accept rules for both port numbers are automatically created. v If configuration sharing is enabled, the end user can enable or disable Network Neighborhood lookup and file sharing through the local agent console. The Protection Level may also be set through the local console.

Navigation
Locate the policy you want to edit and then click Firewall.

26

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Firewall blocking
A firewall can reduce, but not eliminate, exposure introduced by networked hosts. Firewall technology can help prevent attacks by limiting access to ports, IP addresses or ranges of IP addresses required for legitimate system activity. A firewall is one of several approaches you can use to block malicious traffic. See Chapter 4, Protecting against intrusions, on page 33 for a different approach.

Automatic blocking
When the agent detects an attack that poses an immediate threat to the system, it can automatically instruct the firewall to block the intruder. By default, automatically created blocks expire after 24 hours. Because the firewall controls transmission at the TCP/IP level, intruders cannot get around a block from the agent. Automatically created blocks have precedence over manually created firewall rules.

Manual blocking
Security managers can manually configure the firewall to block or accept traffic from any IP address or TCP/UDP port for a period of time or permanently.

Blocking
The agent can block traffic identified by IP address, port, or protocol: v For rules based on an IP address, the agent checks the source IP of inbound communications, and the destination IP for outbound communications. v For rules based on a port number or protocol, the agent checks the port the communication is destined for. Protocol numbers refer to the 8-bit field in the IP header that specifies protocols such as TCP, UDP, or ICMP.

Using firewall blocking for information access


You can use firewall rules to create access policies that prevent users from accessing applications advertised on specific ports. These access policies can block inbound or outbound traffic to these ports based on IP addresses or ranges of IP addresses.

Chapter 3. Firewall configuration

27

Firewall rules
You can create firewall rules to block inbound, outbound, or bidirectional traffic based on IP type or address, port number, or ICMP type or code. In addition, the agent can accept or reject inspected packets based on the source IP address.

Types of firewall rules


The following table lists the basic types of firewall rules:
Filter Option IP type Description An IP Type rule gives you an easy way to block a certain type of IP traffic. For example, you can block all UDP traffic by selecting UDP [17] from the IP Type list, Reject from the Action list, and BOTH from the Direction list. IP address UDP port An IP address rule specifies whether an agent should accept or reject traffic to or from a particular IP address or range of addresses. A UDP port rule specifies whether an agent should allow or block traffic through a particular User Datagram Protocol (UDP) port. You can associate the port with an IP address or address range. A TCP port rule specifies whether an agent should allow or block communication through a particular Transmission Control Protocol (TCP) port. You can associate the port with an IP address or address range. An Internet Control Message Protocol (ICMP) rule specifies whether an agent should allow or block traffic of a particular packet type or code. You can associate the packet type or code with an IP address or address range.

TCP port

ICMP type or code

Navigation
Locate the policy you want to edit and then click Firewall > Firewall Rules.

Firewall precedence
Use the Firewall Rules tab in the Server Protection for Windows agent's Firewall policy to add firewall rules based on IP type, IP address, UDP port, TCP port, and ICMP type and code. Note: Any Intrusion Prevention rules takes precedence over firewall rules.

Order of precedence
The agent applies firewall rules to network traffic based on traffic type and action requested. The agent processes rules in the order indicated in the following table:
Priority 1 Traffic Type IP Address Protocol IP Address and Port/ICMP/Protocol combination Action Accept

28

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Priority 2

Traffic Type IP Address Protocol IP and Port/ICMP/Protocol combination

Action Reject

3 4 5

Port/ICMP Port/ICMP All others

Reject Accept Accept

Examples of firewall precedence


This topic provides example scenarios to illustrate how you can make the application order of firewall rule precedence in the Server Protection for Windows agent work for you.

Scenario 1
You want to block all outbound UDP packets except for outbound UDP DNS packets, and you have the following UDP Rules in place: v UDP Rule 1: All, 53, ACCEPT, OUTBOUND v UDP Rule 2: All, All, REJECT, OUTBOUND Important: If the IP address setting for a UDP Rule or a TCP Rule is configured as All (and not specified numerically), the firewall interprets the rules as a Port/ICMP Traffic Type. In this scenario, Rule 1 would be priority 4 and Rule 2 would be priority 3. As a result, all outbound UDP packets would be rejected because a priority 3 rule (Rule 2) is processed before a priority 4 rule (Rule 1). Suggested approach Use the following rules: v UDP Rule 1: 0.0.0.0-255.255.255.255, 53, ACCEPT, OUTBOUND v UDP Rule 2: All, All, REJECT, OUTBOUND While this approach does not change the logic of the rule (0.0.0.0-255.255.255.255 is the same as All), it does change the priority of UDP Rule 1 from priority 4 to priority 1 because the firewall interprets the numeric IP addresses as an IP Address Traffic Type not a Port/ICMP Traffic Type. This syntax change allows outbound DNS queries while blocking all other outbound UDP packets.

Scenario 2
You want to block unused service ports on a computer. Your first thought might be to do both of the following actions: v Use IP accept rules (Priority 1) to allow access to the computer from your corporate network v Use IP and port reject rules (Priority 2) to block unused ports This approach is not effective because of the order in which the agent applies firewall rules. The IP allow rules are processed first, thereby allowing all IP traffic through. All packets would match the IP allow rules; therefore, the IP and port reject rules are never used. Suggested approach Use both of the following rules:
Chapter 3. Firewall configuration

29

v IP Port accept rules (Priority 1) for all services that you want to allow v IP reject rules (Priority 2) on all others This approach is effective because the IP Port accept rules cover only services that you specifically allow. Not all packets would match these rules, therefore the IP reject rules are applied, denying access to all others.

Managing conflicts between firewall rules


Use the Firewall Rules tab in the Server Protection for Windows agent to add firewall rules based on IP type, IP address, UDP port, TCP port, and ICMP type and code. A conflict exists when two or more rules of the same type specify different treatments for any parameter. The agent resolves such conflicts by applying a special set of priority levels that are internal to a rule type.
Table 1. Criteria for arriving at an internal rule set without overlaps Priority 1 Element Precedence Explanation Depending on the level of configuration sharing in effect, rules received from a management console may take priority over locally-created rules or vice versa. If two or more conflicting rules have the same precedence, the one with the later timestamp takes priority. If conflicting rules have the same precedence and timestamp, the latest (lowest) entry in the file takes priority.

Time

Position

Opening a port through the firewall


Some applications may require ports to be manually opened so that the application can work correctly. As the agent analyzes outbound packets, it can dynamically open the necessary ports for certain applications. This feature allows you to keep the agent set to a more secure protection level without manually opening and closing ports in the firewall.

Automatic port opening for certain applications and protocols


Agents can automatically open ports for the following applications and protocols: v Real Audio v v v v v v v Microsoft Windows Media Player RealPlayer Internet Relay Chat (IRC) Microsoft NetMeeting DNS FTP UDP

30

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Disabling automatic port opening


Each application has its own custom parameter that controls whether port opening is turned on or off. By default, port opening is turned on. You can turn it off by configuring the parameter for the specific application in the Advanced Configuration tab. Example: Add the tunnel.ftpserver parameter with a value of disabled to prevent port opening for FTP servers. Reference: See the Server Protection for Windows Custom Parameters Help for the list of parameters and values.

Opening ports for UDP traffic


The agent monitors all outgoing UDP frames and allows incoming frames from the same port and IP address for 10 seconds. Opening ports for UDP traffic enables the following protocols to operate properly in all protection modes:
Protocol DNS LDAP Kerberos SMTP pcANYWHERE client IPSEC Ports 53, 137 389 88 and 464 123 22, 5632 500

Manually opening a port


For some applications that use application ports, you must manually open ports when the agent is set to the Nervous or Paranoid protection level.

Advanced firewall configuration


The Server Protection for Windows agent supports a set of custom parameters that you can use to fine-tune how the agent firewall handles intrusions. Each parameter consists of the following elements: v Parameter name v Expected value v Description Important: Do not edit the parameters file manually. The Custom Firewall Parameters section of the policy provides an easy to use interface for adding custom parameters.

Navigation
Locate the policy you want to edit and then click Firewall > Advanced Configuration. Reference: See the Server Protection for Windows Custom Parameters Help for detailed information about firewall parameters.

Chapter 3. Firewall configuration

31

32

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 4. Protecting against intrusions


The Server Protection for Windows agent provides preemptive protection by examining a wide variety of protocols, including TCP, UDP, ICMP, and IGMP. The agent parses the protocols, analyzes them heuristically and deterministically, and analyzes them for harmful content.

Protection provided
The agent prevents the following from being used to attack a system: v Known exploits v Unknown exploits against known vulnerabilities v Unknown exploits that attempt to exploit a system by abusing a specific protocol

Topics
Understanding security events Intrusion prevention or intrusion detection Security events Trusted addresses on page 36 Tuning the agent on page 37 Event filtering on page 39Event filtering Advanced security event configuration on page 40 Back tracing on page 40 Packet logging on page 41 Evidence Logging on page 42

Copyright IBM Corp. 2005, 2011

33

About security events


This topic describes how the Security Events policy in the Server Protection for Windows agent works. Where most other protection products look at exploits through static signatures, Security Events understands the underlying protocol and the associated vulnerabilities. Security Events analyses the data stream. If the data contains a packet that is not typical for a particular protocol, Security Events flags the packet as suspicious, and reports or blocks it. If there are specific vulnerabilities associated with a specific type of traffic, Security Events identifies the traffic as malicious and protects the computer.

How security events work


The Server Protection for Windows agent analyzes packets as they enter or exit the system. Based on this analysis, the agent responds in one of the following ways: v If the agent detects no malicious activity, it lets the packets continue unhindered. v If the packets contain an attack that is not immediately threatening, the agent triggers a report to the SiteProtector System and the local user console (if installed). The agent can also log the traffic to evidence or packet logs if configured to do so. v If the attack poses an immediate threat to the computer, the agent sends a report to the SiteProtector System and to the local console (if installed). In addition, the agent stops subsequent incoming packets. The firewall can block the invading packet using any combination of the following methods: v Blocking the source IP address v Blocking the destination port number v Issuing a TCP reset

Event filter and bypass filter interaction


The agent processes bypass filters before it processes event filters, so it is possible to enter a bypass filter that makes an event filter redundant. Although the effect on the system is similar (traffic can circumvent the protection offered by security event rules), you might not see the expected behavior if you later clear the event filter. That is, when you clear the event filter, you would expect the security event rules to provide protection against malicious traffic, but, because the bypass filter is still in effect, packets might still be circumventing the protection offered by security event rules.

The Protocol Analysis Module (PAM)


The Protocol Analysis Module supplies the information that agents use to protect the host computer against intrusions. PAM uses a database that stores the signatures and handles specifications for a comprehensive list of intrusions.

34

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Intrusion prevention or intrusion detection


You can control whether the agent operates in intrusion prevention (IPS) mode or intrusion detection (IDS) mode.

Choosing IPS mode or IDS mode


The following table describes the settings for IPS and IDS modes:
To use... Intrusion prevention mode Intrusion detection mode Select... Block and alert Alert only Results When the agent detects an intrusion, it blocks the intrusion and alerts you. When the agent detects an intrusion, it alerts you, but does not block the intrusion. Intrusion detection is useful if you want to see what type of intrusions are occurring. Some security managers use this approach to monitor intrusions before they set up blocking. Important: If you select Alert only, no events detected by rules configured in the Security Events tab are blocked, even if block is the default action for the rule.

Navigation
Locate the policy you want to edit and then click Security Events.

Security events
The Security Events tab lists hundreds of rules that can protect your system against attacks. You can view security rules and edit their settings individually. You can control the following settings for each security check: v Whether the check is enabled or disabled v Severity level v Whether the event is blocked or only reported Important: The Block settings for events are enforced only when the agent is running in intrusion prevention mode. In intrusion detection mode, intrusions are only reported, regardless of the block status shown for events. See Intrusion prevention or intrusion detection for information.

Navigation
Locate the policy you want to edit and then click Security Events > Security Events.

Chapter 4. Protecting against intrusions

35

Descriptions for security event columns


This topic describes the column headings for the Security Events tab in the Server Protection for Windows agent's Security Events policy.
Table 2. Column descriptions for Security Events Column Name Enabled Tag Name Severity Protocol Block XPU Block Overridden Description Indicates whether the security check is enabled Name of the security check Current severity level of the event. Severity levels are High, Medium, and Low Communication protocol used by the attack or security event Indicates whether the security event is blocked X-Press Update version for this event Indicates whether the agent uses the default blocking behavior for this event, or whether the default blocking behavior has been overridden Indicates whether the severity for this event is set to the default, or whether the default has been overridden The unique identifier that X-Force has assigned to this security event Type of block the agent uses to protect against this security event Date the check for this event was implemented

Severity Overridden Issue ID Blocking Type Check Date

Trusted addresses
When you specify a trusted IP address, the Server Protection for Windows agent ensures that traffic associated with that addresses is not blocked by the security event rules. Trusting ensures that incoming traffic from useful or helpful addresses is not blocked automatically. Important: Consider trusting or ignoring only those addresses that do not pose a threat to your system. Keep in mind that an intruder can spoof, or fake, the IP addresses of an internal system. Note: Event filters give you more granular control over trusted addresses. The trusted addresses feature is provided to support trusted addresses configured in Version 1.0.

Interaction between firewall rules and trusted addresses


Firewall rules are processed before trusted address filters. If you enter a reject firewall rule that conflicts with a trust filter, the traffic will be rejected by the firewall rule.

Interaction among filters


The agent supports bypass filters, event filters, and trusted address filters. v Bypass filters allow packets from certain IP addresses to bypass analysis by the firewall and the security event rules. v Event filters ensure that traffic associated with useful or helpful addresses is not blocked by security event rules. v Trusted address filters ensure that traffic associated with useful or helpful addresses is not blocked by security event rules.

36

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Because there are several ways for the agent to filters traffic, it is possible to enter filters that make other filters redundant. Although the effect on the system is similar (traffic is allowed to circumvent the protection offered by the agent), you might not see the expected behavior if you later clear a filter. If you are unaware that there are redundant filters configured, you might expect the agent to resume processing certain traffic after you clear a filter; however, there might still be a filter configured that allows certain traffic to circumvent the protection offered by the agent. References: See Event filtering on page 39 and Chapter 9, Configuring bypass filters, on page 69. See Traffic seems to be bypassing analysis on page 81 for troubleshooting advice.

IP address or IP range
You can create trust rules for individual IP addresses or for a range of IP addresses.

Navigation
Locate the policy you want to edit and then click Security Events > Trusted Addresses Detail.

Tuning the agent


By default, the agent is configured to report and block many types of intruder activity, including port scans, worm infection attempts, and general suspicious activity. Depending on your network, this default can produce any number of events, from just a few to many.

Reducing the number of events you see


As a security manager, you might or might not be interested in monitoring port scan blocks and other less critical events. You can configure trust rules to help reduce the number of events you see. The following table describes trust rules:
Rule Type trust.pair Description Allows and ignores only specific events from specific computers. The agent does not allow any other events from trusted computers. Similarly, the agent does not allow the trusted events when they originate from a computer that is not trusted. Important: When implemented correctly, this type of rule is the safest trust rule you can use. trust. issue trust.myself.issue trust.myself Allows and ignores events you specify, regardless of where they originate. Allows and ignores only specific events that originate from the server. Allows and ignores all events that originate from the server. CAUTION: Use this type of rule carefully! If the server becomes infected, the agent does not report the outgoing attacks to SiteProtector.

Chapter 4. Protecting against intrusions

37

Rule Type trust.address

Description Allows and ignores all attacks from one or more IP addresses. CAUTION: Use this type of rule carefully! If you create a trust.address rule, the agent can no longer protect the computer if the trusted computer becomes infected by a virus or compromised by an attacker. The agent does not report the attack to SiteProtector.

Where to add rules


Add these rules using the Advanced Configuration tab in Security Events policy. See Advanced security event configuration on page 40.

Using SiteProtector to reduce the number of events you see


To further reduce the number of events you see, use the SiteProtector console to do the following actions: v Create event filtering rules in SiteProtector that allow agents to block less severe attacks locally, but prevent these events from being saved to the SiteProtector database by filtering events by severity at the Agent Manager. v Save all IPS events to the SiteProtector database, but create exceptions to automatically hide certain types of events from the analysis view and reporting.

Raising the visibility of an event


If you want to escalate the visibility of an event, you have the following options: v Create an incident in SiteProtector. v Request alerts (such as email messages or pages) through central responses in SiteProtector.

False positives
If you find that the agent is falsely identifying some types of network traffic in your environment, report the issue to IBM Internet Security Systems. For information about submitting reports, see Reporting a false positive on page 77.

38

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Event filtering
An event filter ensures that traffic associated with useful or helpful addresses is not blocked by rules configured on the Security Events tab. An event filter can reduce the number of alerts displayed in the Console and the number of false positives detected by the agent. Important: Consider filtering events for only those addresses or events that do not pose a threat to your system. Keep in mind that an intruder can spoof, or fake, the IP address of an internal system.

Example
A known user pings you frequently and the agent sends an alert when it detects the ping flood event on the host. As you know this is a harmless event, you configure an event filter for this IP address and this event. When the agent detects the ping flood event from any IP addresses defined in the event filter, it does not block the traffic or send an alert to the console.

How it works
After traffic passes through the firewall, the agent processes the rules configured on the Security Events tab against the traffic. If the traffic triggers a security event rule, the agent would typically take protective action and send an alert to the console to notify you of the potential threat to your system. If, however, you have configured an event filter for this type of event from this IP address, the agent takes no protective action, sends no alert to the console, and allows the packet to continue (even if the packet contains malicious content).

Interaction among filters


The agent supports bypass filters, event filters, and trusted address filters. v Bypass filters allow packets from certain IP addresses to bypass analysis by the firewall and the security event rules. v Event filters ensure that traffic associated with useful or helpful addresses is not blocked by security event rules. v Trusted address filters ensure that traffic associated with useful or helpful addresses is not blocked by security event rules. Because there are several ways for the agent to filters traffic, it is possible to enter filters that make other filters redundant. Though the effect on the system is similar (traffic is allowed to circumvent the protection offered by the agent), you might not see the expected behavior if you later clear a filter. If you are unaware that there are redundant filters configured, you might expect the agent to resume processing certain traffic after you clear a filter; however, there might still be a filter configured that allows certain traffic to circumvent the protection offered by the agent. References: See Trusted addresses on page 36 and Chapter 9, Configuring bypass filters, on page 69. See Traffic seems to be bypassing analysis on page 81 for troubleshooting advice.

Navigation
Locate the policy you want to edit and then click Security Events > Event Filters.

Chapter 4. Protecting against intrusions

39

Advanced security event configuration


The Server Protection for Windows agent supports a set of custom parameters that you can use to fine-tune how agents detect and prevent intrusions. Each parameter consists of the following elements: v Parameter name v Expected value v Description Important: Do not edit the parameters file manually. The Advanced Configuration tab in the policy provides an easy to use interface for adding custom parameters.

Navigation
Locate the policy you want to edit and then click Security Events > Advanced Configuration. Reference: See the Server Protection for Windows Custom Parameters Help for detailed information about IPS parameters.

Back tracing
Use the back trace feature to trace intrusions from other computers. The agent can trace the source of network traffic by analyzing information in the header of a packet that triggers an event. Back tracing also tries to identify other names for the intruder's computer and the hardware address, which makes viewing event information easier for you.

Where are back trace results?


The results of the back trace are available in the following locations: v Event Details in SiteProtector v The Intruder tab in the local console (if installed on the server)

Type of tracing
The agent can perform the following types of tracing: v Indirect tracing v Direct tracing You can configure an agent to use both direct and indirect tracing.

Indirect tracing
Indirect tracing does not make contact with the intruder's system, but identifies an intruder by querying such sources as NetBIOS and the DNS database. This option returns the intruder's host name and lists it in the Events log.

Direct tracing
Direct tracing follows the network path back to the intruder's system to find out the computer name. In general, direct traces gather more reliable information than indirect traces, but direct traces can alert the intruder of the agent's activity.

40

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Navigation
Locate the policy you want to edit and then click Security Events > Back Tracing.

Packet logging
The agent can store information about all system traffic in log files. When you enable packet logging, the agent generates and maintains a set of packet log files. Important: Before you use this option, decide how you want to collect the logs from servers. To collect the logs, you must have local access rights to the server, or you must have someone at the server send the logs to you.

How is packet logging different from evidence logging?


In contrast to evidence logging, which only collects information about intrusion attempts, packet logging captures information about all system traffic. Therefore, packet logs can grow very large and consume a great deal of hard disk space. Packet log files are filled until a maximum size is reached, and then the agent generates a new file. When the maximum number of files is reached, the agent overwrites the first log file with a new file. You can configure maximum file size and maximum number of files as part of the policy.

Locating packet logs


The agent stores packet logs in the directory where it is installed. The file prefix is log by default, or the prefix value you set. The file extension for all packet log files is .enc. Tip: When sorted alphabetically, all packet log files are grouped together according to the prefix you define.

Viewing packet log information


If you want to view the packet logs, you can use a trace file decoding application to open the packet log file. You can set a password to protect the log files. No one can access the log files without entering the password. Note: You can purchase trace file decoders on the Internet. If you are using Windows 2000 Server, you can install the Network Monitoring service, which includes Network Monitor, a decoding application. See the Windows 2000 documentation for more information. In addition, there are free open-source tools that you can use to view packet captures.

Navigation
Locate the policy you want to edit and then click Security Events > Packet Log.

Chapter 4. Protecting against intrusions

41

Evidence Logging
When the agent identifies an incoming packet as a security event, the packet is captured and encoded into an evidence file. Evidence files can provide proof of an intrusion and provide some indication of the intruder's intentions. Important: Before you use this option, decide how you want to collect the logs from servers. To collect the logs, you must have local access rights to the server, or you must have someone at the server send the logs to you.

How is evidence logging different from packet logging?


In contrast to packet logging, which collects information about all network traffic, evidence logging captures only information about intrusions. The agent can capture a single packet of an incoming communication that it identifies as a security event. If you need to, you can use the evidence in this file for an investigation.

Locating evidence logs


Evidence logs are stored in the directory where the agent is installed. The file prefix is evd by default, or the value you set on the Evidence Log tab. The file extension for all log files is .enc. Tip: When sorted alphabetically, all evidence log files are grouped together according to the prefix you define.

Viewing evidence log information


If you want to view the evidence logs, you can use a trace file decoding application to open the packet log file. Note: You can purchase trace file decoders on the Internet. If you are using Windows 2000 Server, you can install the Network Monitoring service, which includes Network Monitor, a decoding application. See the Windows 2000 documentation for more information. In addition, there are free open-source tools that you can use to view packet captures.

For help with analysis


You can send your log files to Technical Support for analysis. For help with interpreting logs, contact mailto://support@iss.net.

Navigation
Locate the policy you want to edit and then click Security Events > Evidence Log.

42

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 5. Protecting against buffer overflow exploits


The Server Protection for Windows agent can prevent intruders from using a buffer overflow to exploit a server. The BOEP component inspects memory as it is extracted in the execution space, helping to detect and prevent both known and unknown attacks that target unknown vulnerabilities on the host system. Note: Buffer Overflow Exploit Prevention is not currently supported on any 64-bit system, Windows Server 2008, or Windows Server 2008 R2.

Topics
Buffer Overflow Exploit Prevention Understanding buffer overflow exploit prevention (BOEP) on page 44

Buffer Overflow Exploit Prevention


Use the BOEP page in the Server Protection for Windows agent to monitor for programmatic behaviors that indicate that a buffer overflow is occurring. The agent needs no prior knowledge of specific exploits or vulnerabilities to protect computers from buffer overflow exploits. Note: Buffer Overflow Exploit Prevention is not currently supported on any 64-bit system, Windows Server 2008, or Windows Server 2008 R2.

What to expect from a buffer overflow


After a buffer overflow attack, the server operator may need to restart the target application to recover. If the compromise affects a core service like MS-RPC or LSASS, the server operator may need to restart the server, and the agent may not be able to log the event. Important: Until you address the root cause or vulnerability, these side effects of an attack may become common occurrences.

What BOEP does not prevent


The BOEP component prevents exploits that attempt to use buffer overflows to run; it does not prevent buffer overflows themselves. Vulnerability to buffer overflows is a characteristic of some applications. If a buffer overflow occurs, the computer may need to be restarted.

Default monitoring
When BOEP is enabled, the agent automatically protects the most commonly exploited applications from buffer overflow exploits. You have the option to include additional folders to monitor additional applications. Keep in mind that adding folders may increase the risk of false positives. Specify inclusions by referencing folders or files; specify exclusions by referencing applications.

Handling false positives


When you determine that a reported buffer overflow event is a false positive, you may want to add the associated application to your excluded applications list. Keep in mind, however, that you may risk future buffer overflow attacks in this associated application.

Copyright IBM Corp. 2005, 2011

43

BOEP and Data Execution Prevention (DEP)


By default, Data Execution Prevention (DEP) is enabled on Windows 2003 Server. DEP may block certain buffer overflow exploits before the BOEP component can analyze them and sent an alert. If you want the Server Protection for Windows agent to monitor for buffer overflow exploit events, disable DEP.

Understanding buffer overflow exploit prevention (BOEP)


Buffer Overflow Exploit Prevention (BOEP) can stop intruders from using a buffer overflow to exploit an end user's computer. BOEP monitors for programmatic behaviors that indicate that an overflow is occurring. When BOEP is enabled, the agent needs no prior knowledge of specific exploits or vulnerabilities to protect computers from buffer overflow exploits.

The importance of layered protection


Buffer overflow exploit prevention should be the last line of defense against packet-based attacks in a layered approach that includes a personal firewall and intrusion prevention. A personal firewall blocks known and unknown attacks against the ports and services you do not need. Intrusion prevention filters out known and unknown attacks against known vulnerabilities. Buffer overflow exploit prevention technology is not designed to be the first line of defense from an attack, as it allows some degree of collateral damage. In an attack, the buffer overflow succeeds; therefore, the targeted process memory may be corrupted and result in a denial of service.

What it does
The BOEP component prevents exploits that attempt to use buffer overflows to run; it does not prevent buffer overflows themselves. Vulnerability to buffer overflows is a characteristic of some applications. The Server Protection for Windows agent monitors system calls commonly used by malicious code. If it detects malicious code attempting to make such a system call, the agent blocks the call and prevents the intended operation. Depending on the response you configure the agent to take if it detects a buffer overflow exploit, the computer may need to be restarted. For example, if the agent is configured to terminate the process where the buffer overflow was detected, and the process is critical to system operation, the system may shutdown.

How it works
BOEP protects host computers from buffer overflow attacks through known and unknown vulnerabilities. As a general rule, a computer should never execute code from writable areas of system memory. By watching the use of Stack and Heap system memory, the agent identifies when a buffer overflow has succeeded and prevents the attack's payload from running. BOEP stops the worm from propagating and prevents the attacker from remotely executing code on the local computer.

Definitions: buffer overflow and buffer overflow exploit


A buffer is a section of memory used for storing data until the process that needs the data is ready to receive it. Some intruders attempt to send more data to the buffer than it can handle, causing a buffer overflow. Intruders may then be able to execute arbitrary commands in a program with administrator permissions, effectively taking control of the remote computer. This activity is the buffer overflow exploit.

44

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Protection for common applications


When buffer overflow exploit prevention is enabled, the agent automatically protects against buffer overflow exploits in common applications. For the most complete, up-to-date list of monitored applications, visit the Technote on the IBM Support portal at https://www.ibm.com/support/docview.wss?uid=swg21468637. Search for KBA #2736. IBM Security may update this list in security updates, and you can add additional folders or files for monitoring.

Protection modes
You can run buffer overflow exploit prevention in one of the following protection modes:
Setting Fail system call and send alert Terminate process and send alert Alert only Description Blocks the suspicious system call, but allows the program to continue running. Sends an alert to the console. Ends the program and sends an alert to the console. Allows the suspicious call and the program to continue running, but sends an alert to the console.

Tip: When you first start using BOEP, use the Alert only setting. This setting, sometimes called simulation mode, can help you identify false positives without interrupting normal business operations.

Chapter 5. Protecting against buffer overflow exploits

45

46

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 6. Enforcing antivirus compliance


With antivirus compliance enabled, the agent can enforce the requirement for antivirus software on a server. You can control which antivirus software version must be running on the host system and define the actions the agent should take if the computer is not running the required software.

Level of configuration
Antivirus compliance requires only minimal configuration, including selecting the required antivirus program, setting compliance parameters, and providing message text.

Topics
Understanding antivirus compliance Configuring antivirus compliance on page 48 Advanced antivirus compliance configuration on page 49

Understanding antivirus compliance


The Server Protection for Windows agent can enforce antivirus software compliance on servers. You can control which antivirus software is required and set actions to be taken if a computer is not in compliance.

How antivirus compliance works


Each time the system starts and periodically thereafter, the agent runs a series of checks to ensure that the server meets the antivirus compliance standard specified in the policy. If the server does not meet this standard, the agent takes the following actions: v Sends an alert to the console v Limits network access (if you have selected that option); the agent can block outbound connections to IP addresses listed in the IP Ranges tab in Group Settings

Supported antivirus software


The agent can enforce the presence of the following antivirus software: v Symantec/Norton Antivirus v McAfee Antivirus If the required antivirus software is not running on the computer, the computer is out of compliance. The agent responds by taking the action you have specified. Important: If you enable antivirus compliance for a group, each computer in that group must use the same antivirus program. If a computer is running an antivirus application other than the one specified in the policy, the agent may block the computer's access to the network.

Copyright IBM Corp. 2005, 2011

47

Other antivirus software


You can use custom settings for antivirus compliance to configure the agent to detect other third-party antivirus software. See the Server Protection for Windows Custom Parameters Help for detailed information.

Configuring antivirus compliance


The agent can enforce the presence of antivirus software on the server. You can control which antivirus software the system should be running and you can define the actions the agent takes if the server is not using the required software.

If you want to block network access


The agent blocks outbound connections to the network based on the IP address lists that are set in Group Properties in SiteProtector. You must define at least one IP address in SiteProtector before network blocking can take place. Important: Be sure computers can access the servers they need to update their virus software and definitions.

Out-of-compliance settings
You can define out-of-compliance limits for computers. In addition, you can set the agent to block or limit the computer's network access when the computer is out of compliance.

Compliance checking
You can control how frequently the agent checks the computer for compliance. You can define different frequencies depending on whether the computer is in compliance or out of compliance. In most cases, the agent should check the computer more frequently when the computer is out of compliance. Then, when the user takes appropriate steps to bring the computer back into compliance, the agent detects the update more quickly and drops the restrictions.

Navigation
Locate the policy you want to edit and then click Application Compliance > Antivirus Compliance.

48

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Advanced antivirus compliance configuration


The Server Protection for Windows agent supports a set of custom parameters that you can use to fine-tune antivirus compliance. Each parameter consists of the following elements: v Parameter name v Expected value v Description

Custom setup
The Server Protection for Windows agent is designed to automatically find the specified McAfee or Symantec antivirus software on the system. However, you can specify alternative methods of finding the information if the initial search is unsuccessful. If the agent does not find the expected antivirus software in the expected location, it checks for custom parameters: v executable_name The name of the program file that runs the antivirus scan program. v virus_definition_disk_folder The absolute path name to the folder where the virus definition files reside. v virus_definition_file_name The name of the virus definition file. Wildcard characters are accepted. Note: If you specify a value for virus_definition_disk_folder, you should also specify a value for virus_definition_file_name, and vice versa.

Navigation
Locate the policy you want to edit and then click Application Compliance > Antivirus Compliance.

Reference
See the Server Protection for Windows Custom Parameters Help for detailed information about antivirus compliance parameters.

Chapter 6. Enforcing antivirus compliance

49

50

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 7. Enforcing application control


Intruders can place unauthorized code on a system to gain control of the system or to exploit network vulnerabilities. When you enable application control, the agent can prevent unauthorized or modified applications from running or connecting to a network.

Topics
Key concepts for application control How to use application control on page 52 How to handle unknown applications on page 53 How to handle known applications on page 54 Preparing and importing known applications on page 56

Key concepts for application control


Intruders can place unauthorized code on a system to gain control of the system or to exploit network vulnerabilities. When you enable application control, the agent can prevent unauthorized or modified applications from running or connecting to a network. Note: Application control is not currently supported on any 64-bit system or on Windows Server 2008 32-bit systems.

Level of configuration
The level of configuration required for application control varies. Before you implement your application control strategy, you might want to monitor which applications run or connect to a network for a period of time so you can plan the best application control strategy. This approach can help you determine the applications that the system uses so that you can create a list of known application. You can also create or import a list of known applications, but this requires more effort.

Copyright IBM Corp. 2005, 2011

51

How to use application control


Application Control can prevent unauthorized applications from accessing the network. You can use Application Control to enforce communication control for servers. This approach provides protection from spyware, riskware, and peer-to-peer programs.

How?
To use this feature for communication control, you must create a list of authorized programs that should be allowed to access the network (using wildcard characters and system variables, as necessary). Then, block all other applications from accessing the network, or use the learning mode to see what other programs are trying to connect. Reference: How to handle known applications on page 54.

Preventing specific programs from running


Application Control can help enforce your corporate security policy. You can create a list of programs that are not allowed to run. You can block known peer-to-peer programs and any other programs that users should not be allowed to run.

How?
Create a known application rule that does not allow the program to run or access the network. If a user attempts to run a banned application, the agent sends an event to SiteProtector. Reference: How to handle known applications on page 54.

Monitoring for unapproved applications not in the corporate baseline


Before you set the policy to prevent all unknown applications from running in a production environment, consider having the agent send alerts each time an unknown application tries to run. You can review the alerts to see if you might need to run more applications than the baseline would allow. After you have collected data on the types of applications that are running, you can adjust policies as needed. Important: In a secure, locked-down environment, this approach may not be an option. Reference: How to handle unknown applications on page 53.

Preventing all unapproved applications from running


If your corporate environment requires servers to be locked down, you may want to block all unapproved applications so they cannot run. Reference: How to handle unknown applications on page 53.

Navigation
Locate the policy you want to edit and then click Application Compliance > Application Control.

52

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

How to handle unknown applications


Use the Application Control settings to configure the way the agent handles unknown applications. Important: Any application not on the known application list is considered unknown. Selecting don't let it connect or terminate may impact the operation of the server as the agent may prevent needed applications from running. Running an agent with a blank or incomplete known application list with the terminate option selected could even prevent the server from being able to start or run.

Startup parameters for unknown applications


You have several options for handling an unknown or modified application that tries to run. These options are as follows: v Let it run v Let it run, but send an event to SiteProtector v Terminate it

Access parameters for unknown applications


You have several options for handling an unknown or modified application that tries to connect to a network. These options are as follows: v v v v v Let it connect Let it connect, but send an alert to SiteProtector Prompt user before allowing application to connect Don't let it connect Terminate it

Learning mode
It is not always possible to include all possible approved applications to the known applications list. The Server Protection for Windows agent can operate in learning mode to allow you to refine your known applications list. In learning mode, the agent allows an unknown application to connect to a network, but that agent also sends an alert to notify you of the activity. You can use the alert to determine whether the application should be added to the known applications list.

Navigation
Locate the policy you want to edit and then click Application Compliance > Application Control.

Chapter 7. Enforcing application control

53

How to handle known applications


You can manage an application's ability to run or access a network on an application-by-application basis. You have several options for handling a known application. These options are as follows: v Allow application to run v Allow network access v Block network access v Do not allow the application to run Important: Any application that you have not added to the known applications list is considered unknown. If a known application is modified, it is no longer recognized and becomes unknown to the agent.

Adding known applications


You can add known applications in the following ways: v Manually enter the application information and settings in the user interface v Import a list of applications and settings from a file Note: If the agent prompts the user, the user can select Add my application to the known application list to add the application.

Field descriptions for known applications


The known application list can use any of the following information types to identify an application and control its behavior:
Field Rule Name Description Path Description Identifies the application rule you are creating. A description of the application rule. The directory path and file name of the program file. You can use wildcard characters and system variables. Examples: C:\WINDOWS\system32\program.exe %systemroot%\system32\program.exe %systemroot%\*.exe Note: Application control monitors only binary program files. MD5 A unique set of numbers and letters that identifies a program and version. The agent uses this value to determine if an application has been modified. Controls whether an application is allowed to run. Controls whether an application is allowed to connect to the network.

Allow application to run Allow network access

Note: Known application entries use AND logic; the more fields you define, the more specific the entry. For example, if you enter values for three fields, all of those values must match before an application is considered a match.

54

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Wildcard characters
The following table lists wildcard characters and values you can use as part of a file path for known applications:
Wildcard Character * ? [] Function Matches zero or more characters. Example: *.exe matches any file with the .exe extension, regardless of the file name. Matches exactly one character. Example: b?ss matches bass, bess, boss, b7ss. Used to designate a set of values. Matches exactly one character in the set specified within the bracket [ ] pair unless brackets include ! or ^ characters. Example: [r5.H_] matches r, 5, ., H, or _. Used only as the first character within bracket [ ] pair. It indicates that the characters within the bracket pair should be excluded. This character can be used interchangeably with ^. Example: [!a3B6] matches any character except a, 3, B, 6. Used only as the first character within bracket [ ] pair. It indicates that the characters within the bracket pair should be excluded. This character can be used interchangeably with !. Example: [^a3B6] matches any character except a, 3, B, 6. Used within bracket [ ] pair to indicate a range of numbers or letters. Example: [1-6a-d] matches any number 1 through 6 and any letter a through d.

System variables
You can use system variables as part of the file path for known applications. The following table lists the accepted system variables and their values:
System Variable %ProgramFiles% %systemroot% %systemdrive% %windir% %temp% %tmp% Value Program Files directory computer's root directory drive letter for the computer's system drive Windows installation directory Windows temporary directory Windows temporary directory

Including application subcomponents


You can choose to automatically include application subcomponents for known applications. Some applications require underlying programs to run and operate correctly. Automatically including the subcomponents of a program is a convenient option, but it is less specific and could be less secure. If you choose not to include subcomponents, you must include the underlying programs as part of the known application list in order for applications to function correctly.

Navigation
Locate the policy you want to edit and then click Application Compliance > Application Control.

Chapter 7. Enforcing application control

55

Preparing and importing known applications


One of the learning mode options requires the agent to send an alert to the central administrator. These alerts show up as events in the SiteProtector Console. You can use SiteProtector to export these events and the corresponding application information to a CSV file. Then, you can use the CSV file to import known applications instead of adding them manually.

Handling embedded commas, spaces, or quotes


After you have exported and formatted the CSV file, use a text editor to make any additional changes. The import functionality for known applications can handle embedded commas and quotes if you use the proper syntax. The following rules apply: v Data members that contain commas, spaces, or double quotes must be enclosed by a single set of double quotes. v Any embedded double quote must be preceded by a double quote. (The first double quote tells the program that the next character it encounters has a special meaning.) The following example illustrates how parsing works: Kazaa Block,Block all unauthorized network access for Kazaa, but allow it to run, C:\Program Files\Kazaa\Kazaa.exe, fc3c926442cc85a469268651bd04c186,true,false This statement is parsed as follows:
Element Rule Name Description Path MD5 Allow To Run Allow Network Access Description Kazaa Block Block all unauthorized network access for Kazaa, but allow it to run C:\Program Files\Kazaa\Kazaa.exe fc3c926442cc85a469268651bd04c186 true false

56

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Exporting event details from SiteProtector


Procedure
1. 2. 3. 4. In the Enterprise Groups pane, select a group for which you want to view events. Select the Analysis view. Select the events you want to export. From the Action menu, select Data Export > Export.

5. In the Files of type box, select CSV File. 6. Choose a file name and a location, and then save your CSV file.

Editing the file


Before you begin
See the SiteProtector help.

Procedure
1. Open the CSV file using a spreadsheet tool, such as Microsoft Excel. 2. Delete all columns except Path and MD5. 3. Add the following columns before the Path and MD5 columns: v Rule Name v Description 4. Add the following columns after the MD5 column: v Allow to run v Allow to access network When you are done, the columns should be ordered as follows:
Rule Name Description Path MD5 Allow to run Allow to access network

5. Add the appropriate data for each entry in the table, leaving blank any item you do not want to specify. 6. Delete the row that contains the column headers and any blank rows before the first row of data. 7. Save, and then close the file.

Chapter 7. Enforcing application control

57

Importing known applications


Before you begin
Make sure that you have a CSV file prepared from SiteProtector.

Procedure
1. Select the appropriate group, and then select Application Compliance > Application Control. The application control settings appear in the details pane. 2. Select Enable general application compliance. 3. Click Import. A browse box appears. 4. Locate the CSV file. 5. Select the file, and then click Open. The information in the file is imported into the list of known applications.

58

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 8. Auditing system integrity and policy compliance


Suspicious activity in system log files, system registry keys, or system directories may indicate a threat to the integrity of the host system or a violation of your security policy. The Server Protection for Windows agent can detect unauthorized changes to the system before they can impact availability.

Sections
Section A: Auditing operating system logs Section B: Auditing registry entries on page 62 Section C: Auditing files and directories on page 64

Section A: Auditing operating system logs


Server Protection for Windows can audit the Windows Event Log for suspicious activity that may indicate a threat to the integrity of the host system or a violation of your security policy.

About event logs


The Windows event logs gather the following types of information: v Security-related events, such as information about valid and invalid logon attempts v Application-related events, such as an error opening an application file v Operating system-related events, such as the failure of a system component to load Note: You can view these event logs in the Windows Event Viewer (Administrative Tools > Event Viewer).

Topics
Auditing Windows event logs on page 60 Coalescing events on page 60 Identifying a suspicious series of events on page 61 Creating exceptions to audit rules on page 62

Copyright IBM Corp. 2005, 2011

59

Auditing Windows event logs


The Windows event logs can hold important clues to problems you may be having with your computer or the applications that run on your computer. Auditing the event logs can help you identify, diagnose, and minimize potential threats to system integrity. With the Server Protection for Windows agent you can do the following: v Configure rules that audit the event logs for frequently encountered events that may indicate your system is under threat v Configure custom rules that audit the event logs for specific threats to your system

How it works
The agent monitors the event logs for events that you have specified as events of interest. When it detects one of these events, the agent sends an alert to the Console to notify you of the log activity.

Enforcing the policy rules


Before the agent can monitor for the audit events you specify in the policy, you must ensure that the server is configured correctly. The enforce audit policy setting ensures that the server has the correct auditing configuration by integrating the agent audit alert system with the operating system audit subsystem. When a change is detected to the auditing configuration on the operating system, the agent ensures that the auditing configuration defined by the audit policy is enforced and combines any additional changes detected.

Navigation
Locate the policy you want to edit and then click System Integrity Monitoring > Audit Rules. Locate the policy you want to edit and then click System Integrity Monitoring > User Defined Audit Rules.

Coalescing events
The Server Protection for Windows agent, by default, generates one alert every time an event you monitor for occurs on a system. This means that the number of alerts sent to the Console can be significant. The agent can reduce the number of alerts sent to the console by coalescing (combining) several instances of a similar event within a specified time frame in to one alert. This can speed up the analysis of alerts.

Which events can you coalesce?


The Server Protection for Windows agent supports coalescing for events detected by the following rules: v Audit rules v User-defined audit rules

How it works
When you configure an audit rule to monitor the event logs for a specific event, you can set a coalescing threshold. If the audit rule is triggered more than one time within this threshold, the agent reports all instances of the event in one alert. The alert specifies the details of the first occurrence of the event and the total number of times the event occurred. Tip: Use coalescing to notify you of many occurrences of lower severity events where timely notification is less critical.

60

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Tip: Study the timing of events on your system to determine the best threshold setting.

Navigation
Locate the policy you want to edit and then click System Integrity Monitoring > Audit Rules. Locate the policy you want to edit and then click System Integrity Monitoring > User Defined Audit Rules.

Identifying a suspicious series of events


Certain threats to the integrity of your system are best identified by correlating a series of events. For example, three failed login attempts followed shortly by a successful login may indicate that a user forgot their password; however, this may also indicate an attempt to break the system's password. Event correlation allows you to identify suspicious patterns of events that might signify a threat to your system; thereby allowing you to quickly pinpoint and address potential issues.

How it works
You enable certain audit rules to monitor your Windows event log. You then create a correlation rule that defines a relationship between certain events. If these events occur within a certain timeframe and in a certain order (if you specified the order of the events was important), the correlation rule is satisfied and the agent notifies you.

Rules you can correlate


The agent can correlate any combination of predefined or user-defined audit rules you have enabled up to a maximum of five rules. Note: A correlation rule cannot contain another correlation rule.

Alerts associated with a correlation rule


The agent sends an alert for each event it is monitoring for. In addition, the agent sends another alert when the criteria of the correlation rule is met; the name you specify for the correlation rule is the alert name that displays in the console.

Navigation
Locate the policy you want to edit and then click System Integrity Monitoring > Correlation Rules.

Chapter 8. Auditing system integrity and policy compliance

61

Creating exceptions to audit rules


As you configure protection, you create rules that define how agents should protect host systems. You may be interested in a particular event unless the event meets a specific criteria.

How it works
You enable an audit rule to monitor and add an exception to that rule. You then edit the exception and specify pre-defined fields in an event that you want exception to be checked. If the field that you want an exception to be checked on is not pre-defined you can then specify a custom exception expression. Using custom exception expressions requires that you understand the Windows event data structure. Because a Windows event data structure can differ from one version of the operating system to another, you might have to add multiple exceptions.

Example
You want to audit all logons to the system except logons by administrators. Using exceptions, you can create a rule to audit the event log and then create exceptions to the rule for any administrators that are allowed to access the system.

Navigation
Locate the policy you want to edit and then click System Integrity Monitoring > Audit Rules. Locate the policy you want to edit and then click System Integrity Monitoring > User Defined Audit Rules.

Section B: Auditing registry entries


The registry contains highly sensitive data, such as user profiles and the ports used by the system, so it is essential that you secure the registry. You can provide a certain level of security by setting the appropriate access rights to subkeys and registry entries; however, you cannot completely lock down the registry because users and programs need access to certain areas to function correctly. Because you must allow some access to the registry, it is essential to monitor activity in the registry so that you can quickly identify potential threats to your system.

Topics
Monitoring the registry on page 63 Creating exceptions to registry monitoring on page 63

62

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Monitoring the registry


Because you cannot completely lock down the registry while still allowing users and programs to access the areas they need to function correctly, it is essential to monitor activity in the registry so that you can quickly identify potential threats to your system.

What to monitor for


The registry integrity monitoring feature can monitor the registry for events that include the following: v Creation of subkeys (this feature is valid for version 2.0 only) v Attempts to determine who owns a key v Creation or modification of the value of a key v Deletion of keys

How registry integrity works


When you monitor registry entries, you define the activity that is of interest to you for the each key. The agent monitors the registry keys for the activity you specify and notifies you if that activity occurs.

Importing a list of keys to monitor


If you already have a list of keys the agent should monitor, you can import this list so that you do not have to enter each key individually. You can only import content from Registration Entries files (files with a .reg extension). Note: The system may take several minutes to import a file. Depending on the size, it could take up to an hour to import the list.

Navigation
Locate the policy you want to edit and then click Registry Integrity Monitoring.

Creating exceptions to registry monitoring


As you configure registry integrity monitoring, you create rules that define how agents should protect host systems. You may be interested in a particular event unless the event meets a specific criteria. Using exceptions, you can create general rules to audit the registry and specific exception rules to ignore an event if specific criteria are met.

Example
You want to audit registry entries for successful queries unless the query was initiated by a security manager. You create a registry integrity rule to monitor the registry entries of interest and then you create an exception rule to exclude queries initiated by the security manager that you do not want to monitor.

Navigation
Locate the policy you want to edit and then click Registry Integrity Monitoring.

Chapter 8. Auditing system integrity and policy compliance

63

Section C: Auditing files and directories


By auditing directories you can detect unplanned and unauthorized changes to files on a system. This allows you to identify and troubleshoot changes that may pose a threat to the system.

Topics
Monitoring directories and files File attributes the agent can monitor on page 65 Importing a list of files to monitor on page 66 Scheduling a baseline comparison on page 67 Updating the baseline on page 67 Excluding directories and files from monitoring on page 68

Monitoring directories and files


The file integrity monitoring component automates the detection of unplanned and unauthorized changes to files. The agent can monitor certain system files by default. You can specify additional files to monitor or you can remove monitored files from this list to customize your protection.

Realtime vs non-realtime monitoring


The agent can monitor files in the following modes: v Realtimethe agent sends an alert to the Console as soon as a monitored file is modified Consider realtime monitoring for the following situations: You have critical files to monitor (the agent can notify you of changes as soon as they occur) It is important to know who made changes to a file (only realtime monitoring can determine the user associated with a change) v Non-realtimethe agent sends an alert to the Console when a scheduled comparison indicates that a monitored file has been modified Note: A file may have been changed days before you receive the alert, depending on how frequently you perform scheduled comparisons.

How it works
The agent creates a baseline of attribute values for the files you want to monitor. The baseline is stored in an embedded SQLite database called iss_fim.db. The file integrity monitoring component detects differences between the current environment and the baseline stored in the database. In realtime mode, the comparison happens as soon as a monitored file is modified; in non-realtime mode the comparison happens at an interval scheduled by you. Note: The initial baseline is of the current system. The agent cannot detect files that are already compromised; the agent will be able to notify you only of file integrity issues from this point forward.

When is the initial baseline created?


To create the initial baseline, configure the file integrity monitoring policy to define the files and file attributes you want the agent to monitor, and then specify a complete baseline update. When the agent receives the policy, it creates the baseline based on your settings.

64

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Navigation
Locate the policy you want to edit and then click File Integrity Monitoring.

File attributes the agent can monitor


Global versus local attributes
The Server Protection for Windows agent supports auditing of both global and local attributes. v Global attributesthe agent monitors for changes to the specified attributes for all files and folders it is auditing v Local attributesthe agent monitors for changes to the specified attributes only for the specified file or folder Support for this granular configuration allows you to define attributes of interest for all files in one place while also allowing you to add custom protection for certain files by defining local attributes for a specific file.

Which attributes can the agent monitor?


The agent can monitor the following file attributes:
Attribute Add Delete Description Monitors for files added to the monitored directory or folder Monitors for files deleted from the monitored directory or folder

Modification Time Time that the monitored file or directory was modified Note: This option is no longer available in Version 2.1 or Version 2.2. Read Note: In Version 2.1 and 2.2 the Read option is only supported at the local attribute level. Monitors for any accessing of information related to a file or directory Note: You do not have to open a file or directory to read information from it; whenever the system must interpret data related to a file or directory, such as when you open the properties window or when you cause a pop-up window to display information about the file, you have initiated a read.

Chapter 8. Auditing system integrity and policy compliance

65

Attribute Modify

Description Monitors for the following modifications to the monitored file or directory: v Content Checksum Monitors for changes to the unique identifier for a file. Note that the agent uses the SHA1 checksum algorithm. v File Type Monitors for changes to the file format associated with the file v Discretionary Access Control List (DACL) Controls which users and groups (trustees) are allowed or denied access to a file/folder v File Size Monitors the size of the file v System Access Control List (SACL) A list used to specify which attempts to access a system object are recorded in the security event log. v Owner Monitors for changes in the Owner of a file/folder. For example, if C:\Dir has "Administrator" as the Owner and then it is modified to "someuser", an event is triggered.

Importing a list of files to monitor


If you already have a list of files the agent should monitor, you can import this list so that you do not have to enter each file individually.

File format
You can only import content from text files (files with a .txt extension), the content must contain the full path to the file or directory, and each item must be on a new line.

Navigation
Locate the policy you want to edit and then click File Integrity Monitoring > Inclusions.

66

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Scheduling a baseline comparison


A scheduled comparison specifies when the agent should compare the current environment to the baseline stored in the database.

Why schedule a comparison?


When the agent monitors a file in realtime, it sends an alert as soon as the monitored file is modified; however, when the agent monitors a file in non-realtime it can only send an alert when the scheduled comparison reveals that the file was modified.

Alerts associated with a scheduled comparison


An agent generates an alert for each modified file it detects during the scheduled comparison, regardless of whether that file is also being monitored in realtime. The agent will generate an alert every time the scheduled comparison runs until you update the baseline with the modified attribute value.

Agent unavailable for scheduled comparison


If for some reason the agent is not available at the time of a scheduled comparison, the agent will run the comparison as soon as it is available. If the file integrity monitoring policy changes while the agent is unavailable, the agent does not perform the scheduled comparison when it becomes available, but it does update the baseline to reflect the policy changes.

Navigation
Locate the policy you want to edit and then click File Integrity Monitoring > Scheduled Comparisons.

Updating the baseline


When you update the baseline, the agent performs a one-time, immediate baseline of the monitored attribute values for monitored files. The agent can create either a complete baseline or an incremental baseline. v Incremental baselineupdates the baseline to reflect only those changes since the last baseline was created Tip: Use an incremental baseline update when policy changes are minor; the agent can complete an incremental baseline in a shorter period of time. v Complete baselinerecreates the baseline based on the new policy settings Tip: Use a complete baseline update when policy changes are substantial. Note: A baseline update does not initiate a database comparison; database comparisons occur only as scheduled on the Scheduled Comparisons tab.

When to update
Update the baseline whenever you make changes to the file integrity monitoring policy. Specify an incremental or complete baseline update as part of any changes you make to the file integrity monitoring policy. Example: You install new software on the host system and you want to audit the integrity of the new files. On the Inclusions tab, you add the new files to the list of files to monitor, you then specify an incremental baseline update on the Reestablish Baseline tab to update the baseline.

Chapter 8. Auditing system integrity and policy compliance

67

Important: If the agent is configured for non-realtime monitoring, consider running a scheduled comparison before you make policy changes and update the baseline to reflect those changes. If you do not run a scheduled comparison before you update the baseline, be aware of the following: v Any changes made to monitored files since the last scheduled comparison will never be reported because, as soon as the baseline is updated, the agent can no longer report changes based on the values in the previous baseline v Any attribute values that have changed will be saved to the baseline, whether the changes were legitimate or not

Notification of baselining
The agent sends an alert to the console when baselining begins and when it completes.

Navigation
Locate the policy you want to edit and then click File Integrity Monitoring > Reestablish Baseline.

Excluding directories and files from monitoring


Server Protection for Windows allows you to configure one inclusion rule to monitor many files and then a few exclusion rules to exclude the few files that you are not interested in monitoring. As you configure protection, you create rules that define how agents should protect host systems. Occasionally you need to protect several files in a directory or a folder but not all files in the directory or folder. You can support exclusions to include only the items that you want to protect.

Navigation
Locate the policy you want to edit and then click File Integrity Monitoring > Exclusions.

68

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 9. Configuring bypass filters


Bypass filtering allows you improve system performance by allowing certain packets to bypass the protection offered by the firewall and the security event rules.

Topic
Bypass filtering

Bypass filtering
You can configure agents to allow packets from certain IP addresses to bypass analysis by the firewall and the security event rules. For example, you do not need to analyze traffic related to a system backup; you can configure a bypass filter to avoid processing this known data and slowing down the backup process.

How it works
When the agent processes a packet, it checks to see if there is a bypass filter set for packets associated with this IP address or this protocol. If there is a bypass filter configured, the agent does not process any firewall rules or security event rules against the packet.

Consideration
The more bypass filters the agent must process, the greater the impact on performance; consider configuring no more than 32 bypass filters.

Interaction among filters


The agent supports bypass filters, event filters, and trusted address filters. v Bypass filters allow packets from certain IP addresses to bypass analysis by the firewall and the security event rules. v Event filters ensure that traffic associated with useful or helpful addresses is not blocked by security event rules. v Trusted address filters ensure that traffic associated with useful or helpful addresses is not blocked by security event rules. Because there are several ways for the agent to filters traffic, it is possible to enter filters that make other filters redundant. Though the effect on the system is similar (traffic is allowed to circumvent the protection offered by the agent), you might not see the expected behavior if you later clear a filter. If you are unaware that there are redundant filters configured, you might expect the agent to resume processing certain traffic after you clear a filter; however, there might still be a filter configured that allows certain traffic to circumvent the protection offered by the agent. References: See Event filtering on page 39 and Trusted addresses on page 36. See Traffic seems to be bypassing analysis on page 81 for troubleshooting advice.

Navigation
Locate the policy you want to edit and then click Bypass Filters.

Copyright IBM Corp. 2005, 2011

69

70

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 10. Configuring administrative settings


The Server Protection for Windows agent offers additional features to enhance protection and make setting up and managing agents easier.

Topics
Configuring management settings Enabling agent protection on page 73 Protecting during system start-up and shutdown on page 74 Setting up configuration sharing on page 75 Configuring event caching on page 76

Configuring management settings


The management settings control how local agents interact with SiteProtector.

Setting the heartbeat interval


Server Protection for Windows agents communicate with SiteProtector through periodic communications called heartbeats. Agents use heartbeats to check for updated policies. The heartbeat interval indicates the amount of time that elapses between heartbeats. If your environment is stable, you might consider setting this value higher, such as once or twice a day. If you are testing policy changes, you might want to temporarily set this value to as low as 30 seconds. Important: Heartbeat settings do not affect the transmission of event information. The agent always sends event information to SiteProtector in real time.

Enabling SiteProtector management


Even if you want to share configuration between SiteProtector and the local console, you may want to limit certain actions from the local console. You can disable the management tab on the local console so that users cannot disconnect the agent from SiteProtector or change which console has configuration priority.

Responding to traffic overload


The Server Protection for Windows agent monitors all traffic to and from the server. During overload conditions, traffic may be delayed as the agent tries to process the high load. You can configure the agent to allow traffic to pass-through the agent to prevent any possible delays. The agent uses a buffer to pass packets between user space and kernel space. For each adapter on the system, there is one buffer. The buffer is a circular queue, so, as the agent processes packets, buffer space is made available to subsequent packets. If the agent cannot read packets from the buffer as quickly as they are written to the buffer, the buffer fills. In these overload situations, the agent fails open and forwards packets to their destination without processing them against firewall rules or security event rules.
Copyright IBM Corp. 2005, 2011

71

Note: This feature applies to versions 2.1 and 2.2. Important: Allowing traffic to bypass the protection offered by the agent may impact the integrity of your server.

Network monitoring
You can configure the Server Protection for Windows agent to let all network traffic pass through without being processed against firewall rules or security events (IPS and IDS mode). Note: This feature applies to versions 2.1 and 2.2.

Configuring Agent Manager failover settings


You can configure the Server Protection for Windows agent to report to a secondary Agent Manager if the primary Agent Manager becomes unavailable. Specify the backup Agent Manager from the central SiteProtector Console.

How the backup management server works


The backup management server feature works as follows:
Stage 1 2 Description The central administrator configures a backup management server with port and account settings that are identical to the main server. The central administrator defines a list of available management servers in the Group Settings policy. Note: The first URL on the list appears in the URL text box in the Management tab on the agent. The agent attempts to report to the first management server on the list. If communication with the first management server fails, the agent attempts a connection to the next management server on the list.

3 4

Reference: For detailed information about configuring a backup management server, see the online Help.

Agent Manager failover settings


The following table describes the failover settings:
Option Number of connection attempts to current Agent Manager before failover Initial retry interval The retry interval doubles with each failed attempt until a maximum of x seconds Description Number of times the agent should try to connect to the primary Agent Manager before it tries to connect to a secondary Agent Manager Number of seconds the agent should wait after a failed connection attempt before it tries to reconnect Maximum interval (in seconds) allowed between connection attempts Note: The interval doubles after each failed attempt so that multiple agents do not send frequent, repeated communication requests. Interval (in seconds) after which the agent should try to connect to the primary Agent Manager. Note: The interval starts following a successful connection to a secondary Agent Manager.

Attempt to reconnect to primary Agent Manager after x seconds

72

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Advanced configuration
The Server Protection for Windows agent supports a set of custom parameters that you can use to override default parameters. Each parameter consists of the following elements: v Parameter name v Expected value v Description Important: Do not edit the parameters file manually. The Management section of the policy provides an easy to use interface for adding custom parameters. Reference: See the Server Protection for Windows Custom Parameters Help for detailed information about management parameters.

Navigation
Locate the policy you want to edit and then click Administration > Management.

Enabling agent protection


The Server Protection for Windows agent can protect its own directories, services, processes, and registry keys from tampering. Note: If this feature was enabled in 2.0 or 2.1 it will be disabled when upgrading. Agent protection is disabled by default, to enable for 2.1 or 2.2 agents, see the Help. The following list details the agent protection implementation: v 32-bit versions of Windows Server 2000 and Windows Server 2003 agent protection restricts all users on the system, including administrators, from accessing agent directories, services, processes and registry keys v 32-bit version of Windows Server 2008 and Windows Server 2008 R2 for users with administrative rights, agent protection does not provide process-level protection v 64-bit versions of Windows Server 2003, Windows Server 2008, and Windows Server 2008 R2 for users with administrative rights, agent protection does not provide process-level protection

Users with administrator permissions


The Stop Agent menu option is always available in the local console for users who have administrator permissions for that server, regardless of the agent protection settings. If you enable agent protection and set a password, the user must enter the password before he or she can stop the agent. Important: When agent protection is enabled, you can only stop the agent only from the local user interface. You cannot stop the agent from the Windows control panel or by using a command prompt. If the local user interface is not installed, you cannot stop protected services.

Other users
The Stop Agent option is always disabled for users without administrator permissions for that server, regardless of agent protection settings.

Chapter 10. Configuring administrative settings

73

Best practice
Consider requiring a password. The password option works with the other agent protection options to secure the agent. If you set a password, only a person who knows the password can stop agent services from the local server. If you set a password, even users with administrator permissions must know the password to bypass agent protection settings.

Navigation
Locate the policy you want to edit and then click Administration > Agent Protection.

Protecting during system start-up and shutdown


Some protection products leave a computer vulnerable to attack between system startup and the time the protection utility starts. The Server Protection for Windows agent can protect a computer during system startup, during system shutdown, or any time that Server Protection for Windows protection is disabled for any reason.

How it works
The agent computer loads the driver at system startup, even before TCP is loaded. If the Block all network traffic when Server Protection for Windows is not running option is enabled, the driver blocks network traffic until the agent is active. When agent services are disabled, the agent applies a special firewall to temporarily allow the following UDP traffic: v DHCP (67, 68) v NETBIOS neighborhood (137, 138)

Deciding whether to use this option


If there are problems with the agent or protection is stopped, the agent blocks inbound traffic and prevents the server from accessing the network. Consider the implications for your corporate environment before you select this feature.

Navigation
Locate the policy you want to edit and then click Administration > Agent Protection.

74

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Setting up configuration sharing


When you install an agent on a server, you can specify whether the agent is visible to a user on the system. You can configure agents to allow no local control, partial local control, or almost complete local control. By default, SiteProtector has total control over all agents in the group.

Local user interface


The local interface shows events, intruders, and network traffic information on the local computer. The interface also provides tools that allow the local user to manage security settings and to control how the agent protects the computer.

Silent agents
An agent that does not include the local user interface is called a silent or headless agent; these agents are always controlled from SiteProtector. Silent agents are appropriate for environments that need host-based protection without relying on local input.

Sharing control
An agent that includes the local user interface may offer the local user a degree of control over agent settings as follows:
Control Level Total management control Shared management control Result SiteProtector has complete control over the agent. The local user has partial control over configuration settings, and can alter any parameters that you have not explicitly set. Settings configured from SiteProtector override settings created locally. The local user has control over all configuration settings. Although SiteProtector distributes configuration settings to all agents in the group, the local user can override any of those settings.

Shared local control

Changing from headless to local console (or vice versa)


You can accomplish most configuration sharing changes by changing the configuration section of the policy. However, you cannot change to or from a headless agent with only a policy update. If you want to change from a headless agent to an agent with a local console or vice versa, you must prepare a new agent build.

Navigation
Locate the policy you want to edit and then click Administration > Shared Configuration.

Chapter 10. Configuring administrative settings

75

Configuring event caching


Event caching controls whether local agents store a record of events when the server is not connected to SiteProtector.

How it works
If you enable this feature, the agent saves event alerts when the server is not connected to SiteProtector. When the agent reconnects to SiteProtector, the agent sends saved alerts to SiteProtector. If you choose to have event alerts cached, you can set the cache size. If you clear this option, the agent does not save event alerts when the server is not connected to SiteProtector and alerts for events that occur during this time are not sent to SiteProtector.

Cache size
If you choose to have event alerts cached, set the cache size (in megabytes). If the cache fills up, the agent overwrites older alerts with newer alerts; set the cache size to minimize the loss of alerts.

Navigation
Locate the policy you want to edit and then click Administration > Event Caching.

76

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Chapter 11. Troubleshooting


This chapter describes issues you may encounter as you use Server Protection for Windows agents and provides information on how to troubleshoot those issues.

Topics
Reporting a false positive Not seeing any BOEP alerts on page 78 Is Version 2.0 supported on ISA Server? on page 78 How can I restore custom policy settings for a child group? on page 79 Overriding parent policies does not give child policies with the parent policy on page 79 Under heavy traffic conditions traffic seems to be bypassing analysis on page 80 Not seeing any file integrity monitoring alerts on page 80 Traffic seems to be bypassing analysis on page 81 System tray icon disappeared after upgrade on page 83 Refresh agent feature in the SiteProtector System not functioning on page 82 Agent showing as offline on page 82

Reporting a false positive


Problem
You find that a Server Protection for Windows agent is falsely identifying traffic, events, or applications as threats to your system.

Background
Sometimes false positives are a result of a misconfiguration of the agent; however, sometimes, because it is difficult to reproduce all possible network configurations when IBM Security tests its software, an agent reports behavior as malicious or suspicious when it is not.

Solution
Report the issue to IBM Security Solutions Technical Support. Please reference the IBM Software Support handbook for information on "Getting IBM Support." v A screen capture of the false positive event (or events) v A brief summary of why you think this false positive happens v A description of the software and version information or network configuration if the false positive is being triggered by a specific software product or network configuration in your environment

Copyright IBM Corp. 2005, 2011

77

Frequently, the following information is absolutely necessary for IBM Security Solutions Technical Support to fix the false positive problem. If you can provide the following information in your report, it would be extremely helpful: v A capture file containing a frame by frame record of network traffic over a specific period of time. v Explicit instructions on how to reproduce the false positive v The name, phone, and email address of someone we can contact if we need assistance to reproduce the false positive

Not seeing any BOEP alerts


Problem
You have enabled Buffer Overflow Exploit Protection but you are not seeing any BOEP alerts.

Background
By default, Data Execution prevention (DEP) is enabled on Windows 2003 Server. DEP might block certain buffer overflow exploits before the BOEP module of the Server Protection for Windows agent can analyze them and send an alert.

Solution
Disable DEP if you want the Server Protection for Windows agent to monitor for BOEP events.

Is Version 2.0 supported on ISA Server?


Problem
You want to install Server Protection for Windows, Version 2.0 on ISA Server but you are not sure if this environment is supported.

Solution
Server Protection for Windows will run in Intrusion Detection mode on ISA Servers after you manually configure a registry key.

Configuring the agent to run on ISA Server


To configure the agent to run on ISA Server, complete the following steps: 1. Install the agent. 2. Stop the agent from the local console. 3. From the command line, stop the driver using net stop issnet. 4. Set the registry value for HKLM/System/CurrentControlSet/Services/issnet/Parameters/ ControlFlags to 0x80000. 5. Start the agent from the local console.

78

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

How can I restore custom policy settings for a child group?


Problem
You want to temporarily override child group policies and use settings from the parent group policies but then you want to restore the custom child group settings.

Example
To test a new configuration, you temporarily need to have each child group send a heartbeat to SiteProtector more frequently than usual; however, if you change the heartbeat setting in each child group manually, you must make a note of the heartbeat setting for each child group so that you can restore the setting after the testing completes.

Background
If a child group is using custom policies and you remove the overrides to the parent policies, the child group uses the parent group policy settings. Reinstate the custom child policies by overriding the parent policies from the child group.

Solution
Reestablish the custom settings for each child group by simply overriding the parent policy at each child group this restores the previous custom policy settings.

Overriding parent policies does not give child policies with the parent policy
Problem
You want to create new custom policies for a child group but, when you override the parent group policy, the child group policy has custom settings from a previous customization of the child group policy not settings from the parent group policy.

Background
Overriding parent policies from a child group reinstates the previous custom policy of the child group (if one exists).

Solution
To create a new custom child policy based on the current parent group policy settings, you must first copy the parent policy and then paste it in the child group. After you paste the parent group policy into the child group, you can customize the policy to create your new child group policy.

Creating a new custom child group policy


Complete the steps below to create a new custom child group policy: 1. Right-click the parent group and select Copy. 2. Right-click the child group and select Paste. 3. Select the newly pasted policy in the child group and configure your custom settings.

Chapter 11. Troubleshooting

79

Under heavy traffic conditions traffic seems to be bypassing analysis


Problem
Traffic seems to be reaching the system without being processed by the agent.

Background
The agent uses a buffer to pass packets between user space and kernel space. For each adapter on the system, there is one buffer. The buffer is a circular queue, therefore, as the agent processes packets, buffer space is made available to subsequent packets. If the agent cannot read packets from the buffer as quickly as they are written to the buffer, the buffer fills. In these overload situations, the agent fails open and forwards packets to their destination without processing them against firewall rules or security event rules. When the agent recovers from the overload condition, it resumes normal processing of packetspackets are again processed against firewall rules and security event rules.

Solution
If the fail open behavior is not acceptable in your environment, for agent Version 2.1, on the Management tab of the Administration policy, clear the Allow traffic to pass through (fail open) check box. This will cause the agent to drop packets when the capture buffer is full. If the fail open behavior is not acceptable in your environment, for agent Version 2.0, in the Advanced Configuration tab of the Security Events policy, set the packet.DropIfBufferFull parameter to TRUE. This will cause the agent to drop packets when the capture buffer is full.

Not seeing any file integrity monitoring alerts


Problem
You updated your file integrity monitoring policy and now you are not seeing any alerts for file integrity events.

Background
Before an agent can monitor for file integrity issues based on the settings you configured in the file integrity monitoring policy, it must update entries in the baseline to reflect the files and values you want to monitor.

Solution
Update the baseline whenever you make changes to the file integrity monitoring policy. Specify an incremental or complete baseline update as part of any changes you make to the file integrity monitoring policy.

Updating the baseline


To update the baseline, complete the following steps: 1. In the Navigation pane, select the group you updated the file integrity monitoring policy for, and then select File Integrity Monitoring. 2. Click the Reestablish Baseline tab. 3. Click one of the following options:

80

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

v Complete baselineoverwrites the existing baseline with a new baseline Tip: Use this option if you made significant changes to the policy. v Incremental baselineupdates the current baseline with only the new changes to the policy Tip: Use this option if you made minor changes to the policy. 4. Do one of the following to apply the updated policy: v Wait for the agents to send a heartbeat to the SiteProtector System v Select the applicable group, and then select Action > Refresh Agent to force the agents to send a heartbeat to the SiteProtector System immediately

Traffic seems to be bypassing analysis


Problem
Traffic seems to be reaching the system without being processed by the agent.

Background
The agent supports bypass filters, event filters, and trusted address filters. v Bypass filters allow packets from certain IP addresses to bypass analysis by the firewall and the security event rules. v Event filters ensure that traffic associated with useful or helpful addresses is not blocked by security event rules. v Trusted address filters ensure that traffic associated with useful or helpful addresses is not blocked by security event rules.

Scenario 1
It is possible that a bypass filter, an event filter, or a trusted address filter is allowing traffic to pass to the system.

Scenario 2
It is possible that you disabled a bypass filter but you still have an event filter or a trusted address filter that is preventing the security event rules from blocking malicious or suspicious traffic.

Scenario 3
It is possible that you disabled an event filter but you still have one or both of the following problems: v A bypass filter that is preventing the firewall and the security event rules from blocking malicious or suspicious traffic v A trusted address filter that is preventing the security event rules from blocking malicious or suspicious traffic.

Solution
Check for filters that might be preventing the agent from inspecting all the traffic you want inspected. Note: Bypass filters and event filters can only be set from the SiteProtector System, where as trusted addresses can be set from either the SiteProtector System or from the local console (if you are allowing configuration sharing between the SiteProtector System and the local console). If, after checking polices at the SiteProtector System level traffic is still not being processed as you expected, check the settings
Chapter 11. Troubleshooting

81

configured at the local console.

Refresh agent feature in the SiteProtector System not functioning


You initiate a Refresh Agent request from the SiteProtector System to the Server Protection for Windows agent, but the agent does not initiate a heartbeat to the SiteProtector System.

Problem
You initiate a Refresh Agent request from the SiteProtector System to the agent, but the agent does not initiate a heartbeat to the SiteProtector System.

Background
The Refresh Agent feature in the SiteProtector System uses an ICMP message to contact the agent. If you have an ICMP firewall rule that includes your SiteProtector System address as a source to block, the agent cannot detect the ICMP message and answer the request to heartbeat in to the SiteProtector System.

Solution
If you have an ICMP firewall rule that is blocking the SiteProtector System, you can do one of the following actions: v Redefine your firewall rules to allow ICMP traffic from the SiteProtector System to your agent v Use only the regular agent heartbeat to communicate with the SiteProtector System

Agent showing as offline


The Server Protection for Windows agent is showing as offline in the SiteProtector System, but the agent is still sending alerts.

Problem
The agent is showing as offline in the SiteProtector System, but the agent is still sending alerts.

Background
The Unresponsive Agent Threshold setting specifies the number of minutes that can elapse between agent heartbeat signals before the agent is considered unresponsive. If the Unresponsive Agent Threshold setting is shorter than the heartbeat interval for an agent, SiteProtector shows your agent as offline when in fact it is available but it has not sent a heartbeat within the threshold period. For example, if your Unresponsive Agent Threshold is set to two hours (the default) and your heartbeat interval is set to six hours, the agent status in the SiteProtector System changes to offline when two hours have passed because the agent has not sent a heartbeat within those two hours. The agent will not send a heartbeat for another four hours based on the heartbeat interval setting.

Solution
Do one of the following actions: v Change the Unresponsive Agent Threshold setting so that it is longer than the heartbeat interval v Send a Refresh Agent command to initiate a heartbeat from the agent to SiteProtector System

82

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

System tray icon disappeared after upgrade


Problem
You have just upgraded from Version 1.0 to Version 2.0 using the heartbeat mechanism and now the system tray icon is not visible on the host system.

Solution
Open the local console from the Start menu; the icon will again be visible in the system tray. This only happens immediately following the update to 2.0.

Chapter 11. Troubleshooting

83

84

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Notices
This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd. 1623-14, Shimotsuruma, Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Copyright IBM Corp. 2005, 2011

85

Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: IBM Corporation Project Management C55A/74KB 6303 Barfield Rd., Atlanta, GA 30328 U.S.A Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee. The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Copyright and trademark information at www.ibm.com/ legal/copytrade.shtml. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.

86

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

Index A
advanced configuration antivirus compliance 49 firewall 31 management settings 73 security events 40 agent password protection 73 agent build definition 22 Agent Manager backup 72 failover settings 72 primary 72 secondary 72 update 9 Alert only (IDS mode) 35 alerts associated with scheduled comparison 67 caching 76 not in console 76, 78, 80 antivirus compliance advanced configuration 49 custom parameters 49 overview 47 supported antivirus software 47 application control 52 handling known applications 54 learning mode 53 system variables 55 unknown applications 53 application rules 28 applications adding to known list 54 audit events coalescing 60 correlating 61 audit rules exceptions 62 monitoring event log 60 predefined 60 user defined 60 automatic blocking 27 automatic port opening 30 buffer overflow exploit prevention (BOEP) (continued) false positives 43 Buffer Overflow Exploit Prevention (BOEP) and Data Execution Prevention (DEP) 78 not seeing alerts 78 overview 44 simulation mode 45 bypass filter interaction with other filters 69 number of 69 when to use 69 bypassing protection 71 deployment (continued) scenarios 6 deployment scenario checklists deployment scenarios 5 double quotes in CSV file 56 6

E
enforce audit policy 60 event filter 39 interaction with other filters 39 event log auditing 60 events coalescing 60 reducing number of 39 exceptions audit rules 62 registry integrity rules 63 system integrity monitoring policy 62 exclusions file integrity monitoring 68

C
caching alerts 76 coalescing audit events 60 combining, coalescing 60 comma in CSV file 56 communication control 52 control application 52 communication 52 correlating audit events 61 correlation rule alert name 61 maximum number of rules 61 critical files monitoring 64 CSV file handling commas 56 handling double quotes 56 handling spaces 56 custom parameters antivirus compliance 49 firewall 31 management settings 73 security events 40 customer support vii

F
fail open 71, 80 failover settings Agent Manager 72 false positives reporting 77 file integrity monitoring attributes monitored 65 creating initial baseline 64 database name 64 exclusions 68 global attributes 65 importing list of files 66 local attributes 65 non-realtime 64 not seeing alerts 80 overview 64 realtime 64 scheduling and non-realtime monitoring 67 scheduling baseline comparison 67 update baseline 67 when to update baseline 67 files or folders adding for file integrity monitoring 64 filters clearing had no effect 81 interaction among 36, 39, 69, 81 firewall advanced configuration 31 automatic port opening 30 blocking 27 custom parameters 31

B
backup Agent Manager 72 how it works 72 baseline start and stop notification 68 updating 67 Block and alert (IPS mode) 35 BlockWhileAgentStopped option 74 BOEP (Buffer Overflow Exploit Prevention) 43 buffer overflow exploit prevention (BOEP) data execution prevention (DEP) 43 Copyright IBM Corp. 2005, 2011

D
Data Execution Prevention and BOEP 78 disable 78 DEP (Data Execution Prevention) deployment agent build 22 before you deploy 5 prerequisites Agent Manager update 9 create groups 10 database update 9 verify licenses 8 43

87

firewall (continued) interaction with Microsoft firewall 25 manual port opening 31 opening port through 30 opening UDP port 31 protection levels 26 rule processing order 28 types of rules 28 Firewall policy 28, 30 application order 29 firewall rules application order 28 conflicts 30 interaction with trusted addresses 36 processing order 28 Firewall Rules tab 28, 29, 30

L
layered protection 1 learning mode 53 licenses 8

S
scheduled comparison alerts associated with 67 secondary Agent Manager 72 security events advanced configuration 40 custom parameters 40 rules to detect 35 Security Events policy 36 security events tab 36 silent agent 75 simulation mode for BOEP 45 space in CSV file 56 system tray icon not visible 17, 83 system variables for Application Control 55

M
management settings advanced configuration 73 custom parameters 73 manual blocking 27 manual port opening 31 Microsoft Windows firewall 25

N G
group settings policy 21 groups adding 11 advantages of 10 network monitoring 72

O
overload condition 80

T
traffic bypassing analysis 81 troubleshooting 82 filter interaction 81 trust rules 37 trusted address filter interaction with other filters trusted IP address 36, 39

H
headless agent 75 changing to or from 75 heartbeat guidelines for setting 71 understanding 71

P
password protection for agents 73 policies Firewall 28, 29, 30 Security Events 34, 36 policy apply updates immediately 3 group settings 21 inheritance 3 policy inheritance 3 parent settings not at child 79 restore custom policy settings 79 policy updates when applied 3 port opening automatic 30 manual 31 UDP traffic 31 primary Agent Manager 72 protection levels 26

36

U
UDP traffic opening ports 31 unknown applications 53 unresponsive agent threshold setting update settings policy 21 upgrading 11 and local console settings 12, 13 before you upgrade 5 82

I
IBM Security Solutions technical support vii IDS mode 35 intrusion detection mode 35 intrusion prevention mode 35 IP addresses trusting 36, 39 IPS mode 35 ISA Server 78 iss_fim.db 64

V
variables for Application Control verify licenses 8 55

K
known applications accepted wildcard characters adding 54 55

R
realtime monitoring when to use 64 refresh agent 82 registry integrity exceptions to rules 63 monitoring 62 reporting false positives 77

W
wildcard characters for known applications Windows event log auditing 60 55

88

Security Server Protection for Windows: Administrator Guide for Security Server Protection for Windows

You might also like