Professional Documents
Culture Documents
ACM SIGCOMM Computer Communication Review 43 Volume 42, Number 2, April 2012
Trace Analysis Format Input Output
Program Pcap file X X
Pcap interface X X
Libtrace API
ERF file X X
DAG capture hardware X X
Native Linux interface X X
Format Modules Protocols Native BSD interface X ✗
DAG PCAP ERF Metadata TSH / FR+ X ✗
Legacy DAG files X ✗
Linux BSD Legacy Link Layer
ATM cell header file X ✗
IP Layer RT network protocol X ✗
Libtrace I/O Transport
Capture Source dump and this can have unintended consequences. For in-
stance, converting an ERF trace to pcap loses timestamp
Figure 1: The architecture of libtrace. precision (the pcap timestamp format only has microsecond
precision whereas ERF timestamps have nanosecond resolu-
tion [6]), the loss counter and the interface that the packet
Performance: In addition to performance improvements was captured on, which often describes the packet direction.
from the threaded I/O, libtrace is optimised to avoid copy- While direction can sometimes be inferred from MAC or IP
ing packets in memory wherever possible and to cache packet addresses within the recorded packet itself, this requires that
properties to avoid decoding the same packet header multi- the addresses have not been modified or anonymised subse-
ple times. In practice, libtrace is notably faster than other quent to the original capture.
protocol header decoding libraries. Therefore, we have designed libtrace to support multiple
trace formats, enabling the user to analyse a trace in its
Libtrace is not the first trace processing library that at- native format and removing the need for potentially lossy
tempts to address the weakness of libpcap; previous efforts format conversions. Each supported capture format is im-
include libcoral [3], libnetdude [4] and scapy [5]; but we plemented as a separate module within libtrace. The module
shall demonstrate that libtrace offers a better compromise provides all of the capture format-dependent functionality,
between the effort required to develop a working analysis such as opening capture sources, reading packet records and
program and the subsequent performance of that program. accessing information stored within the capture header such
as timestamps, wire lengths and loss counters.
The libtrace API provides format-neutral functions that
2. DESIGN AND ARCHITECTURE developers can call within their program to act upon a cap-
Over the past five years, libtrace has evolved primarily ture source or a packet record. When the API function is
to support the research activities conducted by the Univer- called, the matching function for the capture format asso-
sity of Waikato, with an emphasis on both ease of develop- ciated with the trace or packet parameter is then invoked.
ment (especially for student projects) and performance (as For example, if trace read packet is called on a pcap trace
we work with large trace sets). A high-level view of the cur- file, libtrace will call the ‘read packet’ function provided by
rent architecture of the libtrace library is depicted in Figure the pcapfile format module and return the result. This is
1. The arrows describe the communication between the user entirely transparent to the programmer; they simply call the
program, library components and the capture source. The libtrace API function and the library handles the rest.
libtrace library consists of 12,842 lines of C and C++ code Because the libtrace API is entirely format-neutral, a lib-
and comes bundled with a set of tools that perform common trace program is “capture format agnostic”, i.e. the trace
trace manipulation and analysis tasks including splitting, format and location can be specified as a command-line ar-
anonymising and merging trace files. gument and adjusted by the user as needed, but the same
In the following subsections, we shall describe each of the code will read and write any of the supported trace formats
library components in turn and discuss the reasoning behind without modification and recompilation. Libtrace can au-
their current implementation. tomatically detect the format for many input sources, so
specifying the capture format is often unnecessary.
2.1 Capture Formats One benefit of this approach is that a program can be
Many capture formats exist and each differs slightly from developed and tested against off-line trace data before be-
the others. The de-facto standard is the pcap format, pri- ing deployed (without any code changes) on a live capture
marily due to the ubiquity of both the libpcap library (for device, such as a DAG card. Libtrace will also implicitly
writing analysis software) and the tcpdump tool (for cap- convert packets from one format to another if the user se-
turing and examining trace files). Another common format lects a different format when writing packets to an output
is the ERF format employed by Endace high-performance trace. The conversion is performed internally by libtrace
capture hardware [2], which we use to capture many of the and converts between formats as faithfully as possible.
trace files we use for our research. Table 2.1 lists the trace formats that are supported by
Traces using formats other than pcap must be converted libtrace. Output is not supported for many of these formats
to pcap before they can be analysed using libpcap or tcp- because the formats are either obsolete (in the case of the
ACM SIGCOMM Computer Communication Review 44 Volume 42, Number 2, April 2012
legacy DAG formats) or have not been requested by users. Similar problems can arise with IP fragmentation, as novice
The structure of libtrace enables programmers to easily add programmers often assume that the bytes following an IPv4
support for new trace formats and capture devices to future header will always be a transport header.
versions of the library without having to change the libtrace The libtrace API provides functions which allow direct
API. Once the code for the new format is added (typically as access to the packet headers at the metadata, link, IP and
a separate source file), the only other necessary changes are transport layers. These functions will implicitly decode and
to ensure the format constructor is called by the trace init skip any preceding headers. For example, the trace get tcp
function inside libtrace and to update the build system to function will find and return the TCP header for a packet (or
compile and link the new module into libtrace. NULL if no TCP header is present), without the user hav-
ing to concern themselves with the rest of the packet. The
2.2 File Compression and Libtrace I/O Libtrace protocol decoders also detect and handle truncated
Packet trace files used in Internet measurement research headers and IP fragments appropriately.
are often large and are usually compressed to reduce storage Libtrace recognises many protocol headers, including (but
requirements. The typical approach for reading compressed not limited to) Ethernet, 802.11, VLAN, MPLS, PPPoE,
trace files is to use a separate process to decompress the IPv4, IPv6, TCP, UDP and ICMP. Libtrace can also de-
trace, piping the decompressed packets into the analysis pro- code meta-data headers, such as RadioTap, Linux SLL and
gram via standard input. This allows the decompression to Prism. New libtrace releases often add support for new
occur in parallel with the analysis, improving performance headers and link types. Libtrace programs linked against
on multi-core systems. However, the size of the pipe buffer new versions of the library will automatically be able to de-
(64 KB in Linux) limits the rate at which data can be passed code the new headers without code changes.
between the decompression and analysis processes.
Instead, I/O in libtrace is implemented using separate 2.4 API
threads rather than separate processes. For example, when The libtrace programming API is the means by which de-
decompressing a trace, the I/O thread writes the decom- velopers access the trace processing capabilities of libtrace
pressed data into one of fifty 1 MB buffers. Once a buffer and is the only component of libtrace which they interact
is full, a signal is sent to the main thread to indicate there with directly. The C program shown in Listing 1 is a libtrace
is data available for processing. The main thread reads the program that reads packets from a capture source and ex-
packets from the filled buffer as they are requested by the tracts the standard flow 5-tuple fields (IP addresses, ports
analysis, while the I/O thread moves on to filling the next and the transport protocol) and the IP total length. The
buffer. When the main thread has emptied the buffer it is main function opens the capture and reads each packet in
reading from, it sends a signal to the decompression thread turn using trace read packet. For brevity, we have omitted
to indicate that the buffer is available for writing again. from this example much of the error checking that should
Libtrace supports three compression formats: raw (i.e. normally be performed when opening and reading the cap-
uncompressed files), zlib and bzip2. When a trace file is ture. Each packet is passed to the per packet function
first opened for reading by a capture format module, the which finds the IPv4 header (skipping any packets that are
compression format that was used to create the file is deter- not IPv4) and extracts the desired information from the
mined by examining the magic number at the start of the packet.
file and the appropriate I/O module is then associated with As we are only interested in data from the IP header on-
the file. Writing compressed trace files is similar, except wards, we use trace get ip to skip directly to the IPv4
that the compression format must be specified by the caller header. Libtrace handles any preceding link layer headers,
when the file is opened. so our only responsibility is to check that a valid IPv4 header
has been returned. The IP addresses and ports are retrieved
2.3 Protocol Decoding directly using the appropriate libtrace API functions. The
When using libpcap, the programmer must develop code IP length and the protocol are determined by reading the
to decode the protocol headers and find the data that they values from fields in the IP header returned by the call to
are interested in. For example, an analysis of TCP source trace get ip. If further analysis at the transport layer is
and destination ports must include code to search for the required, trace get transport can be used to retrieve a
TCP header, starting from the beginning of the packet and pointer to the transport header.
skipping over any other headers which may be present. Be- Because the intricacies of capture formats and compres-
cause of the extra development effort required, tools written sion libraries are handled internally within libtrace itself,
using libpcap are seldom capable of parsing header combi- this program will cope equally well with any trace format
nations that were not encountered by the author, limiting supported by libtrace. Similarly, the differences between
their usefulness to a wider audience. link layer protocols are dealt with transparently by the call
Another problem that arises when decoding protocol head- to trace get ip, so the properties of the network that the
ers is the variety of edge-cases that occur when examining input trace was captured from does not matter. By using
real Internet traffic. If not detected and handled correctly, libtrace, this is all achieved using only 41 lines of code.
these can have a major (and often undetectable) effect on The libtrace API also allows for asynchronous reading
the analysis results. For example, we commonly encounter from input sources using the trace event function, mean-
incomplete packet headers in trace files, where a captured ing that the main program will not block while waiting for a
packet is truncated mid-way through a header. If care is packet to arrive. Instead, the libtrace API will return either
not taken, it can be easy to read data from beyond the end a time to wait or a file descriptor to wait on if no pack-
of the capture record. This may lead to program crashes ets are available. This is particularly useful for developing
or invalid data being incorrectly treated as packet contents. interactive or GUI-based measurement applications.
ACM SIGCOMM Computer Communication Review 45 Volume 42, Number 2, April 2012
In addition, there are Ruby bindings for many of the lib-
trace API functions that can be used to quickly script pro- Listing 1: An example libtrace program.
totypes of trace analysis tools which use all the features of 1
2
# include < libtrace .h >
libtrace. However, trace processing with Ruby takes sig- 3 void per_packet ( l i b t r a c e _ p a c k e t _ t * pkt ) {
4 uint16_t src_port , dst_port , ip_len ;
nificantly longer than an equivalent C libtrace program (as 5 struct sockaddr_storage src_stor , dst_stor ;
shown in Section 3.1), so Ruby is not suitable for large or 6
7
struct sockaddr * src_ip , * dst_ip ;
uint8_t proto ;
complicated analysis tasks. 8 libtrace_ip_t * ip ;
9
10 ip = trace_get_ip ( pkt );
2.5 Performance 11
12
if ( ip == NULL )
return ;
The trace data sets that we deal with in our research ac- 13
14 src_ip = t r a c e _ g e t _ s o u r c e _ a d d r e s s ( pkt ,
tivities are typically large and can take multiple days to 15 ( struct sockaddr *)& src_stor );
16 dst_ip = t r a c e _ g e t _ d e s t i n a t i o n _ a d d r e s s ( pkt ,
process. For example, the Waikato I trace set in the WITS 17 ( struct sockaddr *)& dst_stor );
18 src_port = t r a c e _ g e t _ s o u r c e _ p o r t ( pkt );
archive is 1.3 TB compressed and spans 620 days of capture 19 dst_port = t r a c e _ g e t _ d e s t i n a t i o n _ p o r t ( pkt );
[7]. When dealing with such large datasets, even minor per- 20
21 ip_len = ntohs ( ip - > ip_len );
formance enhancements can save hours of processing time. 22 proto = ip - > ip_p ;
23
As a result, libtrace incorporates a number of techniques 24 /* Analysis code goes here... */
25 }
designed to maximise the performance of the library. 26
Firstly, as mentioned earlier, all I/O operations in libtrace 27
28
int main ( int argc , char * argv []) {
libtrace_t * trace ;
are conducted using a separate thread. This allows any com- 29 l i b t r a c e _ p a c k e t _ t * packet ;
30
pression and decompression operations to be off-loaded onto 31 packet = t r a c e _ c r e a t e _ p a c k e t ();
32 trace = trace_create ( argv [1]);
another CPU core, without the bottleneck that is created 33 if ( trace_start ( trace )) return 1;
by piping to and from separate gzip processes. Any analysis 34
35 while ( t r a c e _ r e a d _ p a c k e t ( trace , packet ) >0) {
that is I/O-bound, i.e. where the I/O operations require 36 per_packet ( packet );
37 }
more processing power than the analysis itself, will benefit 38 trace_destroy ( trace );
39 t r a c e _ d e s t r o y _ p a c k e t ( packet );
most from this approach. 40 return 0;
When reading from live capture devices, libtrace utilises 41 }
ACM SIGCOMM Computer Communication Review 46 Volume 42, Number 2, April 2012
Libtrace Libpcap Libcoral Libnetdude Scapy
First Released 2004 1994 1999 2003 2005
Language C C C C Python
Version Evaluated 3.0.12 1.1.1 3.8.6 0.12 2.1.0
Released Sep 2011 Apr 2010 Jun 2009 Mar 2010 Dec 2009
Supports Pcap X X X X X
Other Formats X ✗ X ✗ ✗
Reads Stdin X X X ✗ X
Native Gzip X ✗ X ✗ X
Decodes IPv4 X ✗ X X X
Decodes IPv6 X ✗ X ✗ X
Decodes VLANs X ✗ X X X
ACM SIGCOMM Computer Communication Review 47 Volume 42, Number 2, April 2012
Waikato ISP MAWI grams. Libtrace deals with the intricacies of capture for-
Libtrace 00:08:51 00:20:01 00:04:21 mats, network protocols and edge-cases internally, leaving
Libtrace (no threads) 00:12:15 00:25:54 00:06:12 the programmer free to concentrate on measuring the rele-
Libpcap 00:12:09 00:25:45 00:06:12 vant network properties instead. Using libtrace required less
Libcoral 00:12:12 00:29:50 00:04:24 development effort (measured using lines of code) than any
Libnetdude 00:30:43 00:49:19 00:14:41 of the C libraries that we evaluated. The scripted language
Ruby 03:53:39 06:00:44 01:43:45 libraries were faster to develop with, but the performance
(a) Ports Analysis. of the resulting programs proved to be unsuitable for large-
scale trace analysis. By contrast, the libpcap programs re-
Waikato ISP MAWI quired the most lines of code, due to the lack of protocol
Libtrace 00:19:39 00:32:49 00:09:09 decoding functions in libpcap and the need to detect and
Libtrace (no threads) 00:22:45 00:35:05 00:09:52 handle special cases explicitly.
Libpcap 00:19:15 00:31:07 00:08:35 A libpcap program may perform better in some instances,
Libcoral 00:21:35 00:35:35 00:09:42 but significant expertise and time are required to develop a
Libnetdude 00:47:23 01:16:34 00:22:10 libpcap program that delivers accurate and reliable results.
Ruby 04:33:16 06:43:05 02:07:39 Libtrace allows researchers to quickly and easily develop pas-
sive network analysis tools that “just work” regardless of the
(b) Scans Analysis. capture format or network configuration, which we regard
as a valuable contribution to the measurement community.
Table 5: Time taken to run the evaluation programs Libtrace remains under continuous development and sup-
(HH:MM:SS). port for new protocols and capture formats is added on a
frequent basis. There are also ongoing efforts aimed at im-
proving performance; for instance, we have developed an
that packets can be read from disk. The difference between experimental JIT compiler that may allow for BPF filtering
libtrace and other libraries was significant for the Waikato that is faster than the standard libpcap implementation.
and ISP traces: libpcap, which was next fastest, required The latest version of libtrace can be freely downloaded
40% more time to complete the same analysis. However, from http://research.wand.net.nz/.
libcoral was only three seconds slower than libtrace when
processing the MAWI traces; libcoral forks an additional
process to read pcap trace files, which improves the I/O
5. REFERENCES
speed, but this does not occur when reading ERF trace files. [1] “Libpcap,” http://www.tcpdump.org/.
When performing the Scans analysis, we found that libp- [2] Endace Measurement Systems Ltd.,
cap was slightly faster than libtrace. Due to the processing- http://www.endace.com/.
heavy task of having to maintain a flow table, the Scans [3] K. Keys, D. Moore, R. Koga, E. Lagache, M. Tesch,
analysis was not I/O-bound and therefore libtrace’s threaded and k. claffy, “The Architecture of CoralReef: an
I/O offered little advantage over the piped I/O used by the Internet Traffic Monitoring Software Suite,” in Passive
libpcap program. Instead, the speed of the protocol de- and Active Network Measurement Workshop (PAM),
coders was the critical factor: profiling showed that the in- Apr 2001.
creased function call overhead from using the libtrace API [4] “Netdude,” http://netdude.sourceforge.net/.
functions was sufficient to give libpcap a slight performance [5] “Scapy,” http://www.secdev.org/projects/scapy/.
edge. However, the caching of header locations enabled lib- [6] W. John, S. Tafvelin, and T. Olovsson, “Review:
trace to outperform the libcoral analysis. Passive Internet Measurement: Overview and
Guidelines based on Experiences,” Comput.
4. CONCLUSION Commun., vol. 33, pp. 533–550, March 2010.
In this paper we have described libtrace, an open-source li- [7] WAND Network Research Group, “WITS: Waikato
brary for reading, processing and writing packet traces that Internet Traffic Storage,”
we believe offers the best compromise between the effort http://www.wand.net.nz/wits/index.php.
required to develop a trace analysis program and the sub- [8] V. Paxson, “Bro: a System for Detecting Network
sequent performance of that program. We have discussed Intruders in Real-Time,” Computer Networks, vol. 31,
the problems typically encountered when working with pas- no. 23-24, pp. 2435–2463, 1999.
sive network traces, especially when using libpcap, and the [9] M. Allman, V. Paxson, and J. Terrell, “A Brief
methods used by libtrace to solve or alleviate them. History of Scanning,” in Proceedings of the 7th ACM
We have evaluated our library by implementing two trace SIGCOMM conference on Internet measurement, ser.
analysis tasks using several trace analysis libraries, including IMC ’07, 2007, pp. 77–82.
libtrace and libpcap, and measuring both the development [10] “MAWI Working Group Traffic Archive,”
effort required and the time taken to run the analysis over http://tracer.csl.sony.co.jp/mawi/samplepoint-F/
some sample trace sets. We found that libtrace offered the 2010/201004261400.html.
best performance for I/O-bound analysis tasks (40% faster [11] D. Wheeler, “SLOCCount,”
than an equivalent libpcap program) and was second-fastest http://www.dwheeler.com/sloccount/.
when running an analysis-heavy task, where libpcap proved
slightly faster due to less function call overhead.
However, the real strength of libtrace lies in the program-
ming API, which allows easy development of analysis pro-
ACM SIGCOMM Computer Communication Review 48 Volume 42, Number 2, April 2012