TERM PAPER

OF

PRINCIPLE OF OPERATING SYTEM

Topic: - SYSTEM CALL

SUBMMITTED TO:- Miss. Simrat ( Mam).

SUBMITTED BY:- Capt. Gulshan Khan

Roll No:- RJ1801A59

Regn.No:- 10803495

Class:- B.C.A(5th) sem. {Backlog}

Section:- J1801
 Contents

1. Introduction.

2. What Are System Calls?

3. How Does System Call Interception Work?

4. How Does McAfee Entercept Use System Call

5. Interception to Protect Servers?

6. What Is the Performance Impact Associated with

7. System Call Interception?

8. Conclusion
 INTRODUCTION
In computing, a system call is how a program requests a service from an operating
system's kernel that it does not normally have permission to run. System calls provide
the interface between a process and the operating system. Most operations interacting
with the system require permissions not available to a user level process, e.g. I/O
performed with a device present on the system, or any form of communication with
other processes requires the use of system calls.

A system call is used by application (user) programs to request service from the
operating system. The following statements illustrate why system calls are needed. An
operating system can access a system's hardware directly, but a user program is not
given direct access to the hardware. This is done so that the kernel can keep the
system safe and secure from malicious user programs. But often, a user program
requires some information from the hardware (e.g., from a web camera to show you
the picture), but it cannot get the information directly. So, it requests the operating
system to supply it the information. This request is made by using an appropriate
system call.

A system call executes in the kernel mode. Every system call has a number associated
with it. This number is passed to the kernel and that's how the kernel knows which
system call was made. When a user program issues a system call, it is actually calling
a library routine. The library routine issues a trap to the Linux operating system by
executing INT 0x80 assembly instruction. It also passes the system call number to the
kernel using the EAX register. The arguments of the system call are also passed to the
kernel using other registers (EBX, ECX, etc.). The kernel executes the system call and
returns the result to the user program using a register. If the system call needs to
supply the user program with large amounts of data, it will use another mechanism
(e.g., copy_to_user call).

Introduction

System call interception enables many McAfee Entercept proactive server protection
capabilities. This paper addresses the followin questions:

• What are system calls?

• How does system call interception work?

• How does McAfee Entercept use system call interception to protect

Servers ?

• What is the performance impact associated with system call interception?

What Are System Calls?

In order to protect the core of the operating system from damage by errant or
malicious programs, modern operating systems separate code executed by users from
code executed by the operating system itself. To achieve this, modern processors
include a mode bit that specifies whether the processor is executing kernel-mode code
or user-mode code. If the mode bit is set (i.e., user-mode code is executing), the
processor hardware prevents all access to the kernel memory space. If a user-mode

program attempts to access anything in the kernel memory space, the processor
generates an illegal access exception. Thus, no user-mode program can access kernel
memory directly.

User-mode programs need to utilize the functionality provided by the kernel in order
to access disk drives, network connections, and shared memory. Since the processor
prevents direct access to kernel-mode functions, usermode programs must use system
calls, which form the only permitted interface between user-mode and kernel-mode.
System calls expose all kernel functionality that user-mode programs require. System
calls, such as “fopen,” which opens a file, are implemented inside the OS using a
system call table. The system call table relates each system call to a specific function
address within the OS kernel.
 Conceptually, the structure of a
system call table is as follows:
System Call Kernel Function Address

Fopen 0x0000A1F2*

Unlink 0x00003F16*

Rmdir 0x00009C57*
The following C-language program illustrates how system calls are used:

#include <stdio.h>

void main()

FILE* handle;

handle = fopen

("explorer.exe", "w");

When the above C-language program is executed, the processor encounters the
“fopen” instruction, looks-up “fopen” in the system call table, and transfers control to
the kernel-mode function at 0x0000A1F2.

Figure 2 graphically illustrates this concept.

Each system call has an entry in the system call table, which then points to a
corresponding function in the OS kernel.
 How Does System Call Interception Work?
McAfee Entercept adjusts the entries in the system call table, pointing them at the kernel-
mode driver. This makes the above system call table look like this.

System Call Kernel Function Address

Fopen (McAfee Entercept
Driver Address)
Unlink (McAfee Entercept
Driver Address)
Rmdir (McAfee Entercept
Driver Address)
This inserts McAfee Entercept into the command chain anytime a system call is made.
System call interception allows McAfee Entercept to intercept and, if necessary, block access
to any system resources by any program. If McAfee Entercept determines that access should
be allowed, the McAfee Entercept driver calls the original kernel function.

As shown in Figure 3,
McAfee Entercept does not modify the kernel. It simply inserts itself into the command
execution chain. Several commercial products, including most anti-virus products, use system
call interception for various purposes. McAfee Entercept applies this wellunderstood
technique to protecting servers from intrusions and misuse.

How Does McAfee Entercept Use System Call.

Interception to Protect Servers?

McAfee Entercept behavioral rules determine whether a system call is allowed or blocked.
The intricate details of all
the rules are beyond the scope of this paper, but in general, McAfee Entercept asks three
main questions when a
system call is made:

• What process is making the call?

• What user authority is the process running under?

• What is the call trying to access?

One of McAfee Entercept’s many behavioral rules can be summarized as follows:

Rule 1—The Web server can only access Web files and Web-server resources. All other
accesses will be blocked.

The following case examples illustrate how McAfee Entercept enforces this behavioral rule:

Case 1—The Web-server process attempts to access the Web file “index.html.” McAfee
Entercept intercepts the call to open the file and determines the following:

• Process making the call: inetinfo.exe

• User authority: IUSR_<machine>

• Resource accessed: index.html.

With the above information, McAfee Entercept determines that this call involves the Web
server running under the proper user authority and accessing a Web file. Since this
matches Rule 1 above, McAfee Entercept allows the call.

Case 2—An attacker uncovers a new, previously undiscovered Web-server security

vulnerability. This new vulnerability, like so many before it, allows a remote user to access
arbitrary files on the Web server. The attacker exploits this vulnerability, attempting to access
the file “credit_cards.mdb,” which contains the credit card numbers of the users of a
particular e-commerce site. When the Web server attempts to access “credit_cards.mdb,” the
system call to open the file is intercepted.
McAfee Entercept then determines the following:

• Process making the call: inetinfo.exe

• User authority: IUSR_<machine>

• Resource accessed: credit_cards.mdb.

Since “credit_cards.mdb” is not a Web file, this violates Rule 1. McAfee Entercept blocks the
call to open the file, and the exploit is prevented. Another McAfee Entercept behavioral rule,
the converse of Rule1, is:

Rule 2—Only the Web server can access Web files and Web-server resources. Any other
process or user that attempts to access Web files and/or resources will be blocked.

The following example illustrates how McAfee Entercept enforces this behavioral rule:
Case 3—An attacker obtains the administrator’s account password to the Web server, using
social engineering. He or she then logs in to the server as the administrator, opens the
company’s homepage in Notepad and attempts to modify it. McAfee Entercept intercepts the
call to modify the file “company_hompage.html” and determines the following:
• Process making the call: notepad.exe.

• User authority: Administrator.

• Resource accessed: company_homepage.html.

Since “company_homepage.html” is a Web file, but the process and user accessing it are not
the Web-server process and user, McAfee Entercept blocks the call to open the file, and the
defacement is prevented.

What Is the Performance Impact Associated with System Call Interception?

System administrators are rightly concerned about any performance impact introduced by
security software loaded on their servers. McAfee Entercept has minimal impact on CPU
utilization, and its impact on disk utilization and overall latency is negligible.
McAfee Entercept runs entirely in memory, occupying less than 10MB of RAM. As a result,
it can make system call decisions quickly, without accessing the disk. As illustrated in the
examples, the amount of information needed by McAfee Entercept to decide whether to allow
or disallow a system call is small, and, consequently, the number of CPU cycles consumed is
also small. During performance testing with customers who have the heaviest-use profiles,
the percentage of CPU utilized by McAfee Entercept has typically been 1 to 5 percent. The
value of preventing known and unknown attacks far outweighs this minimal impact. Other
host-security products use much more of the system resources. A traditional host-based IDS
can easily use 50 percent of the CPU if all its functionality is enabled. Fileintegrity monitors,
such as Tripwire, can use large amounts of CPU and are constantly accessing the disk,
causing poor disk-throughput response latency. McAfee Entercept generally does not access
the disk once it has been loaded. In these days of gigahertz processors, most servers are I/O
bound, meaning they spend most of their time waiting on disk and network I/O and have
CPU cycles to spare. Web servers, for example, do little processing on the data they serve,
spending most of their time accessing disk and network resources, and transferring data.
Since McAfee Entercept does not usually access the disk, the overall system latency and
response time is unaffected.
 conclusion
McAfee Entercept uses system-call interception and behavioral rules to protect servers from
both known and unknown attacks, and does so without modification to the kernel.
System-call interception allows McAfee Entercept to intercept and, if necessary, block
accesses to any system resources by any program. Because all programs running on servers
must use system calls to access system resources, system-call interception is an excellent way
to protect system resources. Additionally, the performance impact of McAfee Entercept
versus traditional IDS systems is minimal. McAfee Entercept blocks attempted attacks before
they can compromise the system, which allows for maximum security with minimal
performance impact.