91.580.

203
Computer & Network Forensics
Xinwen Fu
Linux Logging Mechanisms

Outline

Log files

What need to be logged

Logging policies
Finding log files

Syslog: the system event logger

CS@UML

Who logs data?

The accounting system
The kernel
Various utilities

All produce data that need to be logged

Most of the data has a limited useful lifetime,
and needs to be summarized, compressed,
archived and eventually thrown away

CS@UML

Logging policies
1.
2.
3.
4.

Throw away all data immediately

Reset log files at periodic intervals
Rotate log files, keeping data for a fixed
amount of time
Compress and archive to tape or other
permanent media

CS@UML

Which policy to choose

Depends on:

how much disk space you have

how security-conscious you are

Whatever scheme you select, regular

maintenance of log files should be
automated using cron

CS@UML

1. Throwing away log files

Not recommend

Security problems (accounting data and log

files provide important evidence of break-ins)
Helpful for alerting you to hardware and
software problems

In general, keep one or two months

CS@UML

In a real world, it may take one or two weeks

for SA to realize that site has been
compromised by a hacker and need to review
the logs

2. Reset log files at periodic

intervals

Most sites store each days log info on

disk, sometimes in a compressed format
These daily files are kept for a specific
period of time and then deleted
One common way to implement this policy
is called rotation

CS@UML

3. Rotating log files

Keep backup files that are one day old,

two days old, and so on.

logfile, logfile.1 , logfile.2, logfile.6

Linux: /etc/logrotate.conf

Specify the frequency with which the files are reused

Each day rename the files to push older

data toward the end of the chain

CS@UML

Script to archive 4 days files

#! /bin/sh
cd /var/log
mv logfile.2 logfile.3
mv logfile.1 logfile.2
mv logfile logfile.1
cat /dev/null > logfile

Some daemons keep their log files open all the time, this
script cant be used with them. To install a new log file, you
must either signal the daemon, or kill and restart it.
In Unix-like operating systems, /dev/null or the null device is
a special file that discards all data written to it, and provides
no data to any process that reads from it. In Unix
programmer jargon, it may also be called the bit bucket or
black hole.

CS@UML

4. Archiving log files

Some sites must archive all accounting
data and log files as a matter of policy, to
provide data for a potential audit
Log files should be first rotated on disk,
then written to tape or other permanent
media

CS@UML

10

Finding log files

To locate log files, read the system startup

scripts : /etc/rc* or /etc/init.d/*

If logging is turned on when daemons are run

Where messages are sent

Some programs handle logging via syslog

(syslogd or rsyslogd)

CS@UML

Check /etc/syslog.conf (or rsyslog.conf on

Fedora Core 9) to find out where this data goes

11

Finding log files (default

configuration)

Different operating systems put log files in

different places:

/var/log/*
/var/cron/log
/usr/adm
/var/adm

On Linux, all the log files are in /var/log

directory

CS@UML

12

Outline
Log files
Syslog: the system event logger

CS@UML

how syslog works

its configuration file
debugging syslog
the software that uses syslog

13

What is syslog
A comprehensive logging system, used to
manage information generated by the
kernel and system utilities
Allow messages to be sorted by their
sources and importance, and routed to a
variety of destinations:

CS@UML

Log files, users terminals, or even other

machines

14

Syslog: three parts

1.

Syslogd: daemon that does the actual

logging

2.

API: openlog, syslog, closelog

3.

Configuration file: /etc/syslog.conf

Library routines that programs use to send
data to syslogd

logger

User-level command for submitting log entries

CS@UML

15

syslog-aware programs
Using syslog library routines
write log entries to a special file

/dev/log

/dev/klog

reads

syslogd

consults

dispatches
Log
files
CS@UML

Userss
terminals

Other
machines

/etc/syslog.conf
Most system logging
daemons listen on one or
more Unix sockets, the most
typical being /dev/log;
/dev/klog is kernel log socket

16

http://www.calpoly.edu/cgi-bin/man-cgi?syslogd

Configuring syslogd
The configuration file /etc/syslog.conf
controls syslogds behavior
It is a text file with simple format, blank
lines and lines beginning with # are
ignored (comment).

CS@UML

selector <TAB> action

for example
mail.info
/var/log/maillog

17

Configuration file - selector

Identifies

Program facility that is sending a log message

Messagess severity level
eg. mail.info

Syntax

facility.level
Facility names and severity levels must be
chosen from a list of defined values

CS@UML

18

Configuration file - Facility

Names
FACILITY
kern
user
mail
daemon
auth
lpr
news

CS@UML

PROGRAMS THAT USE IT

the kernel
User process, default if not specified
The mail system
System daemons
Security and authorization related
commands
the BSD line printer spooling system
The Usenet news system

19

Configuration file - Facility

names (Cont.)
FACILITY
uucp
cron
mark
local0-7
syslog
authpriv
ftp
*

PROGRAMS THAT USE IT

Reserved for UUCP
the cron daemon
Timestamps generated at regular intervals
Eight flavors of local message
syslog internal messages
Private or system authorization messages
the ftp daemon, ftpd
All facilities except mark
UUCP stands for Unix to Unix CoPy.

CS@UML

20

Configuration file - Facility

names
(Cont.)
Facility - Mark: Timestamps can be used to log time

at regular intervals (by default, every 20 minutes),

so you can figure out that your machine crashed
between 3:00 and 3:20 am, not just sometime last
night. This can be a big help if debugging problems
occur on a regular basis
Start at command line: syslogd m 1
Use syslog.conf

Start syslog daemon: syslogd

Add the line to syslog.conf: mark.* /var/log/messages

CS@UML

21

Configuration file - severity

level
severe

not
severe
CS@UML

LEVEL
emerg (panic)
alert
crit
err
warning
notice
info
debug

APPROXIMATE MEANING
Panic situation
Urgent situation
Critical condition
Other error conditions
Warning messages
Unusual things that may need
investigation
Informational messages
For debugging
22

Configuration file - selector

Levels indicate the minimum importance that a

message must have in order to be logged
mail.warning - would match all the messages
from the mail system, at the minimum level
of warning
Level of none will exclude the listed facilities
regardless of what other selectors on the same
line may say.
*.info;mail.none
action

All the facilities, except mail, at the minimum level info

will subject to action

CS@UML

23

Configuration file selector

(Cont.)
Can include multiple facilities separated with ,

commas
e.g., daemon,auth,mail.info
action
Multiple selectors can be combined with ;
e.g. daemon.level1;mail.level2action

Selectors are | -- ORed together, a message matching

any selector will be subject to the action

Can contain

CS@UML

* - meaning all
none - meaning nothing

24

Configuration file - action

(Tells what to do with a message)
ACTION

MEANING

filename

Write message to a file on the

local machine
Forward messages to the syslogd on
hostname
Forward messages to the host at IP address

@hostname
@ipaddress
user1, user2,
*
CS@UML

Write messages to users screens if they

are logged in
Write messages to all users logged in
25

Configuration file - action

(Cont.)

If a filename action used, the filename must be

absolute path. The file must exist since syslogd
will not create it
e.g. /var/log/messages
If a hostname is used, it must be resolved via a
translation mechanism such as DNS or NIS
While multiple facilities and levels are allowed in a
selector, multiple actions are not allowed.

CS@UML

26

Config file examples (1)

# Small network or stand-alone syslog.conf file
# emergencies: tell everyone who is logged on
*.emerg
*
# important messages
*.warning;daemon,auth.info
# printer errors
lpr.debug

CS@UML

/var/adm/messages

/var/adm/lpd-errs

27

Config file examples (2)

# network client, typically forwards serious messages to
# a central logging machine
# emergencies: tell everyone who is logged on
*.emerg;user.none
*
#important messages, forward to central logger
*.warning;lpr,local1.none
@netloghost
daemon,auth.info
@netloghost
# local stuff to central logger too
local0,local2,local7.debug

@netloghost

# card syslogs to local1 - to boulder

local1.debug

@ialab.cs.uml.edu

# printer errors, keep them local

lpr.debug

/var/adm/lpd-errs

# sudo logs to local2 - keep a copy here

CS@UML
local2.info

/var/adm/sudolog

28

Sample syslog output

1. Mar 27 09:10:02 tcb-ia-lab-inst sshd[4100]: Accepted
password for cis418 from ::ffff:216.254.235.105 port 61940
ssh2
2. Mar 27 18:10:00 tcb-ia-lab-inst sshd[9332]: Failed password
for root from ::ffff:216.254.235.105 port 62817 ssh2
3. Mar 27 18:10:08 tcb-ia-lab-inst sshd[9332]: Accepted
password for root from ::ffff:216.254.235.105 port 62817
ssh2
4. Mar 27 20:08:27 tcb-ia-lab-inst sshd[10629]: Accepted
password for root from ::ffff:10.0.0.111 port 42172 ssh2
5. Mar 27 20:09:48 tcb-ia-lab-inst sshd[10649]: Failed
password for root from ::ffff:10.0.0.111 port 48233 ssh2

CS@UML

29

Syslogd
A hangup signal (HUP, signal 1) cause
syslogd to close its log files, reread its
configuration file, and start logging again
If you modify the syslog.conf file, you
must HUP syslogd to make your changes
take effect

CS@UML

ps -ef | grep syslogd

Kill -1 pid-of-syslogd

30

Software that uses syslog

PROGRAM
amd
date
ftpd
gated
gopher
halt/reboot
login/rlogind
lpd

CS@UML

FACILITY
auth
auth
daemon
daemon
daemon
auth
auth
lpr

LEVELS
err-info
notice
err-debug
alert-info
err
crit
crit-info
err-info

DESCRIPTION
NFS automounter
Display and set date
ftp daemon
Routing daemon
Internet info server
Shutdown programs
Login programs
BSD line printer daemon

31

Software that uses syslog

PROGRAM FACILITY
named
daemon
passwd
auth

LEVELS
err-info
err

sendmail
rwho
su
sudo
syslogd

debug-alert
err-notice
crit, notice
notice, alert
err-info

CS@UML

mail
daemon
auth
local2
syslog, mark

DESCRIPTION
Name sever (DNS)
Password setting
programs
Mail transport system
romote who daemon
substitute UID prog.
Limited su program
internet errors,
timestamps

32

Syslog 's functions

Liberate programmers from the tedious
mechanics of writing log files
Put SA in control of logging

Before syslog, SA had no control over what

information was kept or where it was stored

Can centralize the logging for a network

system

CS@UML

33

Debugging syslog -- logger

Useful for submitting log entries from

shell scripts

Can also use it to test changes in

syslogds configuration file.

CS@UML

For example..

34

Add line to syslog.conf:

local5.info

/var/log/test.log

verify it is working, run

logger -p local5.info test messages
a line containing test messages should be written to /tmp/test.log
If this doesnt happen:
forgot to create the test.log file
or forgot to send syslogd a hangup signal

CS@UML

35

Remote logging

On a central logging server: 10.0.0.192

syslogd -r

On a local server: 10.0.0.45

CS@UML

authpriv.*;auth.* @10.0.0.192
Question: where are those events written?

36

Process Accounting

accton is used to turn on or turn off process

accounting
lastcomm tracks commands each user uses

ac prints out statistics about users' connection

times in hours based on the logins and logouts in
the current /var/log/wtmp file

touch /var/log/pacct
/sbin/accton /var/log/pacct
lastcomm -f /var/log/pacct

ac -p -d

sa summarizes accounting information from

previously executed commands, software I/O
operation times, and CPU times, as recorded in
the accounting record file /var/log/pacct

CS@UML

sa /var/log/pacct

37

Process Accounting (Cont.)

last goes through the /var/log/wtmp file and

prints out information about users' connection
times

lastb is the same as last, except that by default it

shows a log of the file /var/log/btmp, which
contains all the bad login attempts.

CS@UML

38

Using syslog in programs

openlog( ident, logopt, facility);

syslog( priority, messge, parameters);

Messages logged with the options specified by

logopt begin with the identification string ident.
Send message to syslogd, which logs it at the
sepecified priority level

close( );

CS@UML

39

/ * c program: syslog using openlog and closelog */

#include <syslog.h>
main ( ) {
openlog ( SA-BOOK, LOG_PID, LOG_USER);
syslog ( LOG_WARNING, Testing . );
closelog ( );
}

On the host, this code produce the following log entry:

Apr 4 15:21:57 tcb-ia-lab-inst SA-BOOK[7762]: Testing ...

CS@UML

40

Summary

On linux, check following files:

/etc/syslog.conf : syslog configuration file

/etc/logrotate.conf : logging policy, rotate
/etc/logrotate.d/*
/var/log/* : log files

try following commands to find out

more...

CS@UML

man logrotate
man syslogd

41

References
1.

2.
3.

4.

Chris Prosise, Kevin Mandia, Matt Pepe, Incident Response and

Computer Forensics, Second Edition (Paperback), ISBN:
007222696X
Brian Hatch, Preventing Syslog Denial of Service attacks,
http://www.hackinglinuxexposed.com/articles/20030220.html
Albert M.C. Tam, Enabling Process Accounting on Linux HOWTO,
02/09/2001,
http://www.faqs.org/docs/Linux-mini/Process-Accounting.html
Keith Gilbertson, Process Accounting, 12/01/2002,
http://www.linuxjournal.com/article/6144

CS@UML

42

Notes

Change host name

CS@UML

/etc/hosts # add the host to the end of

127.0.0.1
/etc/sysconfig/network

43

#! /bin/sh
cd /var/log
mv logfile.2.Z logfile.3.Z
mv logfile.1.Z logfile.2.Z
mv logfile logfile.1
cat /dev/null > logfile
kill -signal pid
compress logfile.1
signal - appropriate signal for the program writing the log file
pid - process id
CS@UML

44