Network Fault Detection: Classifier Training Method for Anomaly Fault Detection in a Production Network Using Test Network

Information
Jun Li and Constantine Manikopoulos Electrical and Computer Engineering Department, New Jersey Institute of Technology University Heights, Newark, New Jersey 07102, USA jxl1727@oak.njit.edu, manikopoulos@njit.edu

Abstract
We have prototyped a hierarchical, multi-tier, multiwindow, soft fault detection system, namely the Generalized Anomaly and Fault Threshold (GAFT) system, which uses statistical models and neural network based classifiers to detect anomalous network conditions. In installing and operating GAFT, while both normal and fault data may be available in a test network, only normal data may be routinely available in a production network, thus GAFT may be ill-trained for the unfamiliar network environment. We present in detail two approaches for adequately training the neural network classifier in the target network environment, namely the re-use and the grafted classifer methods. The re-use classifier method is better suited when the target network environment is fairly similar to the test network environment, while the grafted method can also be applied when the target network may be significantly different from the test network. Keywords: Anomaly detection, Fault Management, Internet, Network Management

1. Introduction
Network faults can be classified into two types: hard failures, in which the network, or some of its elements, are not able to deliver any traffic at all; soft failures, network/service anomaly or performance degradation in various performance parameters, i.e., decrease in bandwidth, increase in delay, etc. A hard fault can be easily noticed by all, the system administrators as well as the users. However, defining and otherwise characterizing and detecting soft faults is more difficult. In the past few years, some modest amount of research progress has been made in soft fault detection [1, 2]. A proactive fault management system is presented

in [3]. In [4] a study was carried out of path failure detection. In [5], neural networks are applied in billing fraud detection in telecommunication voice networks. Performance anomaly detection in Ethernet is discussed in [6]. In [7], Cabreta et. al. described their practice of using Kolmogorov-Smirnov (K-S) statistics to detect Denial-of-Service and Probing attacks. In previous work [8], we have proposed a hierarchical, multi-tier, multi-window, soft fault detection system for wireless and wired networks, namely the Generalized Anomaly and Fault Threshold (GAFT), which operates automatically, adaptively and proactively. The system uses statistical models and neural network classifiers to detect anomalous network conditions. The statistical analysis bases its calculations on probability density function (PDF) algebra, in departure from the commonly used isolated sample values or perhaps their averages. The use of PDFs is much more informative, allowing greater flexibility than just using averages. The sample PDF of some network performance parameter, shown in Fig 1, helps illustrate that point. The figure depicts a hard service limit at some parameter value, drawn as a vertical line. Such a limit might apply for a real-time service, for example, the total packet delay parameter for packetized voice. From the point of view of the hard limit, the network condition described by the PDF in this figure would represent a service failure, in that some packets exceed the limit. However, the PDF shows that the totality of such failing packets, those in the tail of the PDF to the right of the limit, may be small. Thus, if in fact, this total is smaller than the maximum packet loss rate specification, the service may remain in compliance to the delay limit, simply by discarding the laggard packets. Our system generalizes further by combining information of the PDFs of the monitored performance parameters, either all of them or subgroups of them, in one integrated and unified decision result. This combining is powerful in that it achieves much higher discrimination capability

1
Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE

that enables the monitoring of individual service classes in the midst of general traffic consisting of all other classes. It is also capable of discriminating against intrusion attacks, known or novel. In fact, network intrusion detection is one of the promising applications of this technology. Others, include transaction oriented e-commerce networks, that typically support large numbers of service classes concurrently, and, as previously indicated, various wireless networks, where monitoring the service classes may be challenging due to adverse network conditions.
0.045

The rest of this paper is organixed as follows: Section 2 introduces the re-use and grafted training approaches; Section 3 outlines the algorithms of fault modeling, estimation and simulation for the grafted classifier; Section 4 describes the testbed and the fault schemes simulated; Section 5 reports the results; and Section 6 summarizes the conclusions of this work.

2. Classifier training methods

In a production network, during its intended operation, anomaly detection is expected to find deviant activity. The classifier processes the current pattern of activity and rates it with a similarity score as to whether it is more similar to normal or anomalous activity. The GAFT classifier has employed supervised training that utilized labeled data collected during normal and anomalous activity periods, in some test network. The GAFT classifier, like most such classifiers, requires data records of the typical (normal) and fault-labeled (anomalous) variety in sufficiently large amounts. Moreover, if one desires to distinguish between different classes of faults or perhaps individual faults, data for all such different faults need to be made available for training. Such data will enable the neural net classifier to learn the difference between normal and anomalous patterns of activities, thus achieving high detection and low false alarm rates. In a test or laboratory network the experimenter or developer can investigate and record the launching of all desired known faults, selecting from a library of faults. Such investigations of faults can be carried out and recorded for various background traffic types and levels. The challenge that is faced is that while both normal and fault data are easily and conveniently available for a test network, only normal data are routinely available for a production network. The classifier of the network fault detection system will probably be ill-trained for the new network environment and thus unsuitable for use without changes if (1) the unfamiliar network is sufficiently different from the test network, so the new background will not be close enough to that of the test network, or (2) it lacks training that includes fault data for the unfamiliar network. One or the other, or more likely both conditions hold in most realistic cases of installations of new network fault detection system (NFDS) and thus the NFDS classifier will need to be re-trained at some stage during or soon after installation. A network fault detection tool that is to be installed in an unfamiliar real network environment can reasonably expect some initialization period that will provide sufficient estimates of the normal behavior of the network. In a typical setting, by far most of the traffic that the NFDS sees is normal, with only a sprinkling of fault traffic. Thus, algorithms could conceivably be used

0.04

0.035

0.03

probability

0.025

hard limit

0.02

0.015

0.01

0.005

0.5

1.5

network parameter

Fig. 1. PDF of a network performance parameter GAFT gathers data from network traffic, the system log and hardware reports; it statistically processes and analyzes the information, detects network anomaly and failures, for each parameter individually, or in combined groups using neural network classification; it generates the system alarms and event reports; finally, GAFT updates the system profiles based on the newly observed network patterns and the system outputs (see Fig 2). A more detailed description of the architecture of GAFT can be found in [8].
Inputs GAFT Outputs

Network Data Network Traffic Information (IP, UDP, TCP, ...) Higher Layer Log Information Hardware reports and information from some other sources

Event Probe and Analysis

Anomaly Alarm Network Reports Event Log

Fault Detection

System Update

Fig. 2. The inputs and outputs of GAFT

2
Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE

to separate out the normal traffic from any faults, whenever they might occur in the unfamiliar network, based on intensity comparison considerations alone. Such algorithms, employing clustering techniques, have been investigated elsewhere [9]. Additionally, it may be that during the initialization phase, all the traffic in the new setting is in fact normal. Even if some fault traffic is found in the unfamiliar network, it may be difficult, at this stage, to characterize what type of fault it is. This leaves us with an estimate of normal traffic in the unfamiliar network setting, which is possible to carry out, but no clear description of network faults. Although theoretically possible, it would be unreasonable and unsafe to launch faults in the production network to be protected, solely for the benefit of the learning phase of the NFDS. Most likely it would not be permitted to take place. Thus, somehow, while using the normal and fault data from a test network, as well as the normal data from the production network, it is needed to ensure that a properly trained classifier is generated. In this paper, we propose two alternative solution approaches to this challenge, the re-use classifier and the grafted classifier: The re-use classifier method is straightforward and consists of employing the test-network classifier in the new network, as is after training only in the test network, that is without any modifications, at the beginning. This approach is expected to perform adequately if the unfamiliar network is similar in architecture and usage to the test network, but will likely increasingly deteriorate in performance the more dissimilar the two settings are. The grafted classifier method essentially consists of abstracting an estimate of the fault model, in pure form, from the test network and then grafting this fault onto the normal model in the new network, thus transplanting the fault model from the test network to the production network. In this way, the grafted classifier method will provide fault models as well a normal models in the unfamiliar network setting, thus enabling the training of the classifier in this new network. We have used simulation measurements to investigate the effectiveness of these techniques in an experimental setting, where both have been found to be effective. The grafted classifier approach will work adequately for the class of parameters where the effect of the fault is of additive character to that of the background. Nevertheless, it should be noted that the expectation regarding these techniques is not that they generate the final best model of a fault, but that they will provide a good seed of such a model, allowing the system to adapt the model by learning from then on, thus bootstrapping onto an accurate fault model from an

initially only adequate estimate. GAFT employs an adapting algorithm that adjusts the normal as well as the fault models, by using the descriptions of the current normal and fault data, after they are detected to be so. It is noteworthy that the re-use and grafted classifier we proposed in this work can be also applied to training the anomaly-based intrusion detection system [10]. The grafted classifier approach described above, may, in general, depend on the nature of the monitored parameter as well as the type of fault. Basically, it relies on the normal background model to vary most from network to network while the fault models vary least. This is intuitively reasonable and has been observed to be the case for many types of networks and faults. The normal background network traffic changes in time and from network to network. On the other hand, the behavioral profiles of various kinds of faults are comparatively stationary and exhibit similarities from network to network. In more detail, this approach involves extracting fault profiles, for each monitored parameter, from a test network and then applying them to a new production network. Firstly, we extract the models of faults and background traffic from the data collected in a test network. Secondly, the fault profiles are calculated by removing the components of normal traffic from the fault models. Thirdly, the fault models for the new network can be estimated by combining and merging the models of normal traffic of the real network with the models of faults that have been extracted from the test network. Finally, the fault traffic itself can be simulated according to the estimated fault models, thus enabling the NFDS classifier to be trained using the data thus generated. Below, it is desired to provide and differentiate the definitions of fault models and fault profiles: Fault model is the statistical representation of the network traffic pattern when both the background traffic and the fault traffic are observed in a network. Fault profile is the statistical representation of the network traffic pattern when only the fault traffic is observed in a network.

3. Fault modeling, estimation and generation

Most of this section deals with the grafted classifier algorithms, with some descriptions and algorithms of common utility with the re-use classifier method, as well. Here, in Subsection 3.1, we will first introduce the traffic model representation of GAFT and the overall approach of fault modeling and estimation algorithms; the algorithms to measure the reference models are described in Subsection 3.2, the algorithm of calculating the fault profiles is given in Subsection 3.3; fault

3
Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE

estimation is discussed in Subsection 3.4; and Subsection 3.5 introduces the algorithm to generate the fault PDFs to train the classifier.

3.1. Overall approach

In GAFT, the network traffic observations are summarized in a set of Probability Density Functions (PDFs), where each PDF represents the distribution of a network parameter, e.g. packet length, packet rate, etc., of the traffic observed during a time window. Let S be the sample space of a network measurement, events E 1 , E 2 , , E k a mutually exclusive partition of S, N the total number of events of the monitored parameter, and p1 , p 2 , l , p k the probabilities of an event falling within the partition

Thus, it may be feasible to estimate fault models, in a production network, based on the knowledge collected from a test network, at least well enough to serve as seeds in the unfamiliar network setting. The fault modeling and estimation involves the following steps (see fig. 4):
Background and Fault Models Measurement Background Models Measurement Network Traffic Measurement Data

Test Network

Fault Profile Extraction Fault Instances PDF Generation

Production Network

Fault Model Estimation

GAFT

Classifier Results

Ei s during the observation. Thus,

here, the PDF of the network parameter measurement is represented as the organization of N events into E i s and

Fig. 4. The phases of fault estimation Fault and background models measurement in the test network: the reference models of both the normal traffic and the fault traffic need to be calculated from the test network. Afterward, we will call the test model of normal background traffic M Bt and the test fault traffic M Ft for simplicity. Fault profiles extraction: based on the reference models built up in the above phase, we can calculate the profiles of the fault, PF , by removing the components of normal traffic traffic model

pi s . A more detailed comparative study of various

promising partition schemes can be found in [11]. A sample PDF figure is shown in Fig. 3. The partition algorithm used in this experiment is a uniform partition with 64 bins in the histogram.

M Bt from the fault

M Ft .

Background measurement in the production network: in this phase, we calculate the models of background traffic in the unfamiliar production network, M Bd , for use in the fault model estimation, subsequently. Fault model Estimation in the production network: the fault model, M Fd , is estimated by combining and merging appropriately the fault profiles

Fig. 3. A sample PDF diagram The normal network traffic is generally bursty and varies considerably in a network from time to time and from network to network, therefore the models of normal traffic are hard to predict or estimate in advance. However, because the schemes and patterns of a particular fault often have their own specific character in how they affect the normal background traffic, their effect on the normal traffic may be fairly consistent.

PF

with the background model

M Bd .

Fault PDF generation: at this point the fault models have been calculated, resulting in one PDF estimate for each parameter that describes the average behavior for that parameter. However, what are needed are the numerous presumed contributors to this average PDF, to be used as training, test and validation data that describe the faults. Explicitly, using this PDF as the generator for each parameter,

4
Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE

we can randomly generate presumed fault instances that are PDFs that statistically average to the calculated PDF model. Subsequently, for each presumed fault instance generated, we carry out the similarity measurement to the normal background, thus finding a similarity score for each parameter. The scores of all the parameters are then assembled into a similarity vector record. That is how the many required fault records can be constructed. The totality of these records will be apportioned into training, test and validation fault sets for use by the classifier of GAFT.

PF (i ) =
where: i: The

N F M Ft (i ) N B M Bt (i ) NF NB NP = NF NB
i th bin of PDF.

(2)

PF : The profile of the particular fault traffic at hand. M Ft : The reference model of the particular fault
traffic at hand. M Bt : The model of the normal background traffic.

3.2. Reference model measurement

The measurement of the reference model (M) is at the foundation of the fault estimation algorithms presented here, therefore an accurate but efficient approach in calculating the reference models is crucial. We use the following equation to construct the reference model: M (i, t ) = (1 r.s(t )) M (i, t 1) + r.s(t ) K (i, t ) (1) Here M(t): The reference model at time t. i: The i bin of the PDF that represents the model. t: The simulation time window. r: The predefined learning rate. Generally, r is assigned a small value (i.e., 0.01). K(t): The newly observed network traffic pattern at time window t. s(t): The score of traffic(t). s equals to 1 when the type of traffic(t) (either background or fault)is the same as that of M (background/fault). Otherwise s is set to 0. The basic idea here is to enable the reference model learn, at a small learning rate, when the traffic is normal while keeping it unchanged when the traffic is anomalous.
th

N F : The number of samples comprising M Ft . N B : The number of samples comprising M Bt .

N P : The number of samples comprising PF . The basic idea here is that an estimate of the change that the faults impose to the normal traffic PDF models, for some and perhaps most parameters, can be inferred by subtracting out the contributions of the normal model from the fault model, bin-by-bin. Even if this works well for only a few of the parameters, the combining power of the classifier assures that the classification results will be better than those of any single parameter by itself. This is sufficient to seed correctly the GAFT system into its learning mode, so that it improves its models using actual data, from then on. Our experiments suggest that many parameters can in fact be modeled correctly in this manner.
3.4. Fault model estimation
The fault models for an unfamiliar production network can be estimated as the sum of the model of the background traffic in the production network ( M Bd ) and the fault profiles derived from the test network ( PF ). The equations of estimation are:

3.3. Fault profile extraction

The reference models

M Ft contain both fault traffic

M Fd (i ) =
where: i: The

N B M Bd (i ) + N P PF (i ) NB + NP NF = NB + NP
i th bin of PDF.

(3)

and the background traffic, because both kinds of traffic may be observed within a time window. This is the reason why the fault models collected in the test network cannot be applied directly to an unfamiliar production network. The profiles of faults PF are calculated by removing the components of the background traffic M Bt from the fault models M Ft . The equations to calculate the profiles of the pure faults are given below:

M Fd : The estimated reference fault model in the

unfamiliar production network. N F : The number of samples comprising the estimated fault model

M Fd .

5
Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE

M Bd : The measured model of the background

traffic in the production network. N B : The number of samples comprising M Bd .

4.1. Network model architecture

4.1.1. Test network simulation model. We built the test network using the Optimized Network Engineering (OPNET) tool, as shown in Fig. 5. In this network model, there are 3 servers, namely, Main Server (responsible for file access and mail service), HTTP Server, and TELNET Server, as well as 11 clients, connected by a hub.

PF : The fault profile derived from the test network. N P : The number of samples comprising PF .

3.5. Fault PDF generation

At this point we have computed an estimate of the fault model in the unfamiliar production network. Then fault PDFs can be generated as the sum of the estimated fault models and the background traffic collected from the production network. The equations are:
K F , d (i, t ) = N K B, d (t ) * K B, d (i, t ) + N P * PF (i ) N K B, d (t ) + N P
N K F ,d (t ) = N K B ,d (t ) + N P

(4)

where: i: The i bin of the PDF. t: The time window K F , d (t ) : The simulated fault traffic in the production network. K B , d (t ) : The measured background traffic in the production network at time window t. PF : The extracted fault profile. Fig. 5. Test network simulation model
th

N K F ,d (t ) : The number of samples of K F , d (t ) . N K B ,d (t ) : The number of samples of K B , d (t ) .

4. Simulation Experiments
In order to investigate the performance of the re-use and grafted classifiers methods, we carried out extensive simulation experiments under various conceivable production network environments, i.e., variations in network topology, typical background traffic and anomaly traffic characteristics and load. The rest of this section is organized as follows: Subsections 4.1 and 4.2 present the test and production network model architectures and the corresponding background network traffic configurations used in the various simulation scenarios; Subsection 4.3 describes the specific network fault model used in the experiments, while in Subsection 4.4 we describe the statistical parameters used and their interrelation. Numerical results and additional discussions regarding the effectiveness of these two classifier methods are presented in Section 5.

4.1.2. Production network simulation model. The production network model, shown in Fig. 6, consists of four subnetworks connected by four routers. The links that connect the network components are assumed to be T1 links. The clients are located in all the four subnetworks as follows: Subnet_1 (Ethernet Network), Subnet_2 (Token Ring Network), Subnet_3 (Fast Ethernet Network) and Subnet_Server (FDDI Network). The clients located in each of the four subnetworks communicate with the server located in Subnet_Server. In our simulation experiments, in order to comparatively evaluate the performance of the two proposed classifier methods under the variation of the production network topology, we changed the number of the clients located in each subnet, which establish the communication with the server located in the Subnet_Server. Three simulation scenarios are investigated, i.e., 10 clients, 15 Clients and 20 Clients (see Section 5).

4.2. Network background traffic configuration

In our simulation experiments, we model the four most popular TCP/IP services: HTTP (TCP), TELNET (TCP), FTP (TCP) and SMTP (UDP). The application-

6
Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE

layer workload characteristics of these four Internet services, as used in our simulations, derive from the literature, as in [12]; such work indicates that the workload characteristics utilized closely resemble network conditions.

4.4. Statistical parameters monitored by GAFT

For detection purposes, GAFT monitors 21 parameters. Specifically in the UDP layer detection, it collects statistics for the following five parameters: UDP Packet Length, UDP packet Ratio, UDP Byte Ratio, UDP Packet Rate and UDP Byte Rate. Here, the UDP Packet Ratio and UDP Byte Ratio are defined as the ratio of packets and bytes received from different clients. The UDP Detection module collects the information of both the background traffic and the fault traffic. In TCP layer detection, GAFT will collect statistics for the following eight parameters: TCP Received Packet Length, TCP Received Packet Rate, TCP Received Byte Rate, TCP Connection Open Rate, TCP Connection Close Rate, TCP Connection Duration, TCP Received Packet Port Ratio and TCP Received Packet Host Ratio. Here, the TCP Received Packet Port Ratio and TCP Received Packet Host Ratio are defined as the ratio of packets received from different ports and clients, respectively. In the IP layer Detection module, GAFT collects statistics for the following five parameters: IP packet length, IP Packet Ratio, IP Byte Ratio, IP Packet Rate and IP Byte Rate. Here, the IP Packet Ratio and the IP Byte Ratio are defined as the ratio of packets and bytes, respectively, that are received from different clients. The IP layer detection module collects all the traffic information which passes through the IP layer. In Heartbeat Detection, GAFT collects statistics for the following two parameters: Packet Loss Rate and End to End delay parameters. The heartbeat signal detection technique is a useful construct for tolerating faults in computer networks. It is used by distributed programs to ensure that if a process in a program terminates or fails, then the remaining processes in the program also terminate. Here, we use the concept of the heartbeat protocol to detect network faults. Heartbeat packets will be sent, with small packet length and short period. By collecting the information of the packet loss rate and end-to-end delay of the heartbeat packets, GAFT can evaluate the relationship between network congestion and network faults. In the experiment, we use constant packet length of 100 bytes and period 0.5 s for heart beat packet signaling. These 21 network parameters, are monitored using constant 5.27 s size time windows and then organized as PDFs with 64-bin partition histograms.

Fig. 6. Production network simulation model

4.3. Network fault model

In a real networking environment, a network/service degradation or failure may be caused by equipment malfunctions, malicious external or internal faults, software execution runaways on workstations, etc. However, these problems usually result in the generation of other additional common faults, such as the persistent packet storm, leading to router utilization overloads, significant packet delays and packets being dropped from the routers. This in turns may lead to clients being disconnected from the communicating server, etc. The end result, from a service viewpoint, is that services may be denied to clients. In this work, we simulated such network fault conditions by injecting various volumes of TCP fault traffic into the simulation testbed, i.e. the TCP small packet flooding fault traffic. Specifically, in the test network, a client (workstation) is assumed to generate and send TCP flooding packets to the HTTP server periodically, and in the production network, a client located in the Subnet_1 will send the TCP flooding fault packets to the server located in Subnet_Server. As presented in detail in Section 5, we performed various experiments, and tested the performance of re-use and grafted classifier methods under many different scenarios, where we varied several characteristics regarding the network topology and the anomalous traffic and background traffic, such as the number of clients located in each subnets, and the background and anomalous traffic patterns.

5. Experimental results and discussion

In this section we present and discuss the numerical results that we obtained performing the various experiments, described in the previous section. The objective of the results presented, in the remaining of this section, is to evaluate the effectiveness of the re-use and grafted classifier methods, and demonstrate that the

7
Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE

grafted method performs significantly better than the reuse method, in most simulation scenarios. As mentioned, we have carried out extensive simulation experiments. The purpose of these simulation experiments is to investigate the performance of re-use and grafted classifier methods, when the network topology and the background and anomaly traffic of the production network vary. The production network configurations, for these simulation experiments, are given in Table 2. Three scenarios are carried out. In each scenario, the data sets collected from the test network (the test network configuration presented in Table 1) are used as training data for training the test classifier. This test classifier is used in the investigation of the efficacy of the re-use classifier. This data are also used for the calculation of the fault profiles, by using the algorithms described earlier that are part of the grafted classifier method. The data sets collected from the production network are used to estimate fault models and to simulate fault data for that environment. The classification rates of GAFT using the simulated data are compared with those using the actual data, to carry out performance comparisons.

3. The PDF data collected from the test network are used for both training and testing of the GAFT classifier. Subsequently, the classifier is used to validate the production network setting PDF data. Thus, this corresponds to examining the efficacy of the re-use method, to be referred to as the re-use method experiment. The misclassification rates of GAFT, for the baseline, grafted classifier and re-use classifier experiments, are listed in Table 3. The ROC (Receiver Operating Characteristic) diagrams are presented in Figs. 7, where the x-axis is the false alarm rate and the y-axis is the detection rate. The false alarm rate is the rate of the typical traffic events being classified as faults or anomalies, while the detection rate, is calculated as the ratio between the number of correctly detected faults/anomalies to their total number. The values seen in the table follow what was observed from Figs. 7. Table 2. The production network configuration
*The combination of the background traffic in the production network is the same for the three simulation scenarios, which is HTTP: 50%, FTP: 20%, SMTP: 20%, TELNET: 10%, but different from that of the test network.

Table 1. Test simulation network configuration Network Topology Background Traffic 11 clients connected Total Traffic in bits/sec: 234k Combination: Http: 65%, FTP: 15%, SMTP: 15% and TELNET: 5% Total Traffic in bits/sec: 7k

Fault Traffic

For each simulation scenario, three different experiments are performed to examine the classification performance outcomes of GAFT for the production networks at hand for the two approaches, the re-use classifier and the grafted classifier. 1. The actual PDF data collected from the production network are used for the training and testing/validation of the GAFT classifier. The results are used as the base line of the system performance, in that they should be the best that can be achieved for the actual production network. For simplicity, we call this experiment as the baseline for experimental outcomes. 2. The generated PDF data, using the grafted classifier method, are used for training the GAFT classifier, while the actual data are used for testing. The results reflect the performance of this method and is referred to as the grafted method experiment.

# of clients in Subnet 1 3 # of clients in Subnet 2 3 # of clients in Subnet 3 3 # of clients in Subnet Server 1 Total # of clients 10 Background 213 k bps* Fault Traffic 4.5 kbps # of clients in Subnet 1 4 # of clients in Subnet 2 4 Network # of clients in Subnet 3 4 Topology # of clients in Subnet Server 3 Total # of clients 15 Background 320 kbps* Fault Traffic 4.86 kbps # of clients in Subnet 1 5 # of clients in Subnet 2 5 Network # of clients in Subnet 3 5 Topology # of clients in Subnet Server 5 Total # of clients 20 Background 427 kbps Fault Traffic 6.48 kbps Network topology

From the simulation experiments, we can observe that the misclassification rates of the grafted experimental measurements perform significantly better than the re-use classifier experiments. This may be due to the significant dissimilarity between the test network configuration and the production network configuration. In fact, the grafted classifier performance is close to that found for the baseline experiments; the results of the

Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE

Scenario1 Scenario2 Scenario3

latter are, of course, the best that can be achieved for the production network. This indicates that the grafted modeling and estimation algorithms can be used as effective starting points for the case which the network topology of the production network is significantly different from the topology of the test network, when migrating an anomaly NFDS from the test network to an network setting, where accurate fault models are not available. Table 3. The misclassification rate of the simulation experiments Baseline Grafted Re-use Scenario 1 0.0117 0.0474 0.0864 Scenario 2 0.0160 0.0871 0.2483 Scenario 3 0.0127 0.0983 0.1665

References
[1] S. Aidarous and .T. Plevyyak, Eds, Telecommunications network management, in IEEE Series on Network Management. New York: IEEE Press, 1998. [2] C. Hood and C. Ji, Proactive Network Fault Detection, IEEE Trans. Reliability, Vol. 46, No. 3, 1997. [3] L.L. Ho, D.J. Cavuto, S. Papavassiliou, A.G. Zawadzki, Adaptive and Automated Detection of Service Anomalies in Transaction-oriented WANs: Network Analysis, Algorithms, Implementation, and Deployment, IEEE Journal on Selected Areas in Commun., vol. 18, May 2000, pp. 744-757. [4] N. Dawes, J. Altoft, and B. Pagurek, Network Diagnosis by Reasoning in Uncertain Nested Evidence Spaces, IEEE Trans. Commun., vol. 43, 1995, pp. 466. [5] J.R. Dorronsoro, F. Ginel, and C. Sanchez, Neural fraud Detection in Credit Card Operations, IEEE Trans. Neural Networks, vol. 8, 1997, pp. 827. [6] R. Maxion and F.E. Feather, A Case Study of Ethernet in a Distributed Computing Environment, IEEE Trans. Reliability, vol. 39, Oct. 1990, pp. 433-443 [7] Joao B.D. Cabrera, B. Bavichandran, R.K. Mehra, Statistical Traffic Modeling for Network Intrusion Detection, Proceedings of 8th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication systems, Aug. 2000, pp. 466-473. [8] Constantine Manikopoulos, Symeon Papavassiliou, Jiang Jun, Jun Li, and Zheng Zhang, Generalized Anomaly Detection in Next Generation Internet: Architecture and Evaluation, submitted to Computer Communications (Elsevier publication) Special Issue on Performance Evaluation of IP Networks and Services. [9] L. Portnoy, E. Eskin, and S. Stolfo, Intrusion detection with unlabeled data using clustering, in Proceedings of ACM Workshop on Data Mining for Security Applications, pp. 114, November 2001. [10] Zheng Zhang and Constantine Manikopoulos, Methods of Classifier Training for Anomaly Intrusion Detection in a Deployed Network Using Test Network Information, in Proc. 2002 IEEE Workshop on Information Insurance, United States Military Academy, West Point, NY, 17-19 June 2002, pp. 306314. [11] Z. Zhang, C. Manikopoulos, and J. Jorgenson, Experimental comparisons of binning schemes in representing network intrusion detection data, in accepted by CISS2002, March 2002. [12] Vern Paxon and Sally Floyd, Wide-area traffic: the failure of poisson modeling, IEEE/ACM Transactions on Networking, 3(3) pp226-244, June 1995.

6. Conclusions
The classifier of our prototype NFDS, GAFT, while well trained in a test network, is ill-trained in an unfamiliar network setting. This is because while typical background traffic data is available in the new network, fault data usually is not. To solve this problem, we have designed two techniques, the re-use classifier and the grafted classifier methods. We have carried out several experiments that compare the baseline (best possible) outcomes (when the classifier knows the faults in the unfamiliar network) to the results for the grafted and reuse classifier methods, at various background patterns and network topologies. The classification results show that the re-use classifier method can achieve modest performance, but cannot perform reliably. However, the classification performance of the grafted classifier methods is quite satisfactory, being only a little inferior to the baseline results. This indicates that the grafted modeling and estimation algorithms can be used as effective starting points, when migrating an anomaly NFDS from the test network to an unfamiliar network setting. GAFTs adaptation and learning algorithm can then bootstrap the system from adequate fault models to excellent ones. Additional experimentation with a greater variety of test-target network combinations, in both simulated and real network environments, will be helpful in further investigating the applicability of these two methods.

Acknowledgement
Our research was partially supported by a Phase II SBIR contract with US Army. We would also like to thank OPNET TM Technologies, Inc., for providing the OPNET simulation software.

9
Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE

Fig. 7. The ROC curves

10
Proceedings of the 27th Annual IEEE Conference on Local Computer Networks (LCN02) 0742-1303/02 $17.00 2002 IEEE