CERT/CC
background
background
CERT NetSA Security Suite 
Open Source Tools for Network Monitoring 
News | Documentation | Downloads
YAF 0.8.1 | NAF 0.6.0 | SiLK 1.0.1 | RAVE 1.9.9
fixbuf 0.7.3 | ipa 0.2.1 | airdbc 0.2.2 | airframe 0.7.2 | Portal 0.8.0
SiLK - Documentation
Documentation | Downloads | Release Notes | FAQ | License | Credits | Reference Data | Live CD

Handbooks

Analysts' Handbook: Using SiLK for Network Traffic Analysis (pdf only)
tutorial on the SiLK tools and on using them for analyzing network traffic
PySiLK: SiLK in Python (pdf)
reference guide for manipulating SiLK Flow data from within Python
The SiLK Reference Guide (pdf)
the manual page for each tool in a single document
SiLK Installation Handbook (pdf)
instructions on configuring, building, and installing SiLK at your site

Alphabetized Index of Manuals

addrtype
ccfilter
flowcap
mapsid
num2dot
pmapfilter
rwaddrcount
rwappend
rwbag
rwbagbuild
rwbagcat
rwbagtool
rwcat
rwcount
rwcut
rwdedupe
rwfglob
rwfileinfo
rwfilter
rwflowappend
rwflowpack
rwgeoip2ccmap
rwgroup
rwidsquery
rwip2cc
rwipaexport
rwipaimport
rwipfix2silk
rwmatch
rwnetmask
rwp2yaf2silk
rwpackchecker
rwpcut
rwpdedupe
rwpmapbuild
rwpmapcat
rwpmatch
rwptoflow
rwrandomizeip
rwreceiver
rwresolve
rwscan
rwscanquery
rwsender
rwset
rwsetbuild
rwsetcat
rwsetintersect
rwsetmember
rwsettool
rwsetunion
rwsilk2ipfix
rwsort
rwsplit
rwstats
rwswapbytes
rwtotal
rwtuc
rwuniq
sensor.conf
silk.conf

Analysis Suite

The SiLK Analysis Suite is a collection of command-line tools for querying packed NetFlow data. The most important tool is rwfilter, an application for querying the central NetFlow data repository for NetFlow records that satisfy a set of filtering options. The tools are intended to be composed in various ways to perform an analysis task. A typical analysis uses unix pipes and intermediate data files to share data between invocations of tools.

The Analysts' Handbook: Using SiLK for Network Traffic Analysis (pdf only) gives a tutorial on the tools and describes using them for analysis.

Each tool is distributed with its own UNIX manual page (available through the links below). In addition, all the manual pages are available in a singe document: The SiLK Reference Guide (pdf).

Filtering, sorting, and display

rwfilter

Partition SiLK Flow records into one or more 'pass' and/or 'fail' output streams.

silk.conf

Configuration file for Classes, Types, and Sensors.

rwsort

Read SiLK Flow records, sort them by specified, and write the records to the named output path or to the standard output.

rwcut

Print SiLK Flow records in a |-delimited, columnar, human-readable format.

Counting and statistics

rwuniq

Summarize SiLK Flow records into user-defined keyed bins specified with the --fields switch.

rwcount

Summarize SiLK Flow records across time, producing textual output with counts of bytes, packets, and flow records for each time bin.

rwstats

Summarize SiLK Flow records by one of a limited number of key/value pairs and display the results as a Top-N or Bottom-N list.

rwtotal

Summarize SiLK Flow records by a specified key and print the byte, packet, and flow counts for flows matching the key.

rwaddrcount

Summarize SiLK flow records by source or destination IP

IPset, Bag, and Prefix Map manipulation

rwset

Read SiLK Flow records and generate binary IPset file(s).

rwsetbuild

Read IP addresses in dotted-quad or CIDR notation from input-file and writes a binary IPset file to output-file.

rwsetcat

Print the IPs and other information about IPset files.

rwsettool

Generate a new IP-set by tooling all --add-set binary IP-set files, then removing all --remove-set binary IP-set files from the new IP-set.

rwsetmember

Generate a new IP-set by membering all --add-set binary IP-set files, then removing all --remove-set binary IP-set files from the new IP-set.

rwsetintersect

Generate a new IP-set by intersecting all --add-set binary IP-set files, then removing all --remove-set binary IP-set files from the new IP-set. This command is deprecated; use rwsettool instead.

rwsetunion

Merge the input binary IPSet files into the output IPSet; an IP in any input file will be in the output file. This command is deprecated; use rwsettool instead.

rwbag

Read SiLK Flow records and builds binary Bag(s) containing key-count pairs.

rwbagbuild

Create a binary Bag file from binary IPset file or a textual input file.

rwbagcat

Print binary Bag files as text.

rwbagtool

Perform operations on bag files.

rwpmapbuild

Read textual input and creates a binary prefixmap file for use with the Address Type (addrtype) and Prefix Map (pmapfilter) Plug-Ins.

rwpmapcat

Print information about a prefix map (pmap) file. By default, print each IP range in the pmap and its label.

rwipaimport

Import a SiLK IPset, Bag, or Prefix Map file into the IP Address Association (IPA) library.

rwipaexport

Export a set of IP addresses from the IP Address Association (IPA) library to a SiLK IPset, Bag, or Prefix Map.

Run time plug-ins

addrytpe

The Address Type plug-in provides a way to map an IP address to an integer denoting the IP as internal, external, or non-routable.

ccfilter

The Country Code plug-in provides a mapping from an IP address to two-letter, lowercase abbreviation of the country that "owns" the IP address.

pmapfilter

The Prefixmap plug-in provides a way to map field values to string labels based on a user-defined map file.

Record grouping and masking

rwgroup

Group flows together by specified id-fields and delta-field; marks the group ID in next hop IP; requires pre-sorting.

rwmatch

Group records as queries and responses.

rwnetmask

Read SiLK Flow records from STDIN, mask-off the lower (32-N) bits of the specified IP address(es), and write the resulting records to stdout.

Packet and external flow-format processing

rwpcut

Output a tcpdump file as ASCII, in a form similar to rwcut.

rwpdedupe

Detect and eliminate duplicate records.

rwpmatch

Filter a tcpdump file by outputting only packets whose 5-tuple and timestamp match corresponding flows in a rw-file.

rwptoflow

Read a tcpdump file and generate a SiLK Flow record for every packet.

rwp2yaf2silk

Convert a tcpdump file to a single file of SiLK Flow records; rwp2yaf2silk assumes that the yaf and rwipfix2silk commands are available on your system.

rwipfix2silk

Convert a stream of IPFIX (Internet Protocol Flow Information eXport) records to the SiLK Flow record format.

rwsilk2ipfix

Convert a stream of SiLK Flow records to an IPFIX (Internet Protocol Flow Information eXport) format.

Scan detection

rwscan

Attempt to detect scanning activity from SiLK Flow records. rwscan can produce files that can be loaded into a database and queried with rwscanquery.

rwscanquery

Query the scan database that has been populated from database load files generated by rwscan.

Utilities

mapsid

Map between sensor names and sensor IDs.

num2dot

Read pipe (|) delimited text from the standard input, convert integer values in the specified column(s) (default first column) to dotted-decimal IP addresss, and print result to standard output.

rwappend

Append the SiLK Flow records contained in the second through final filename arguments to the records contained in the first filename argument.

rwcat

Read SiLK Flow records from the FILES named on the comamnd line, or from the standard input when no FILES are provided, and write the SiLK records to the specified output file or to the standard output if it is not connected to a terminal.

rwdedupe

Read SiLK Flow records from files named on the command line or from the standard input and write the records to the named output path or to the standard output, removing any duplicate flow records. Note that the order of records is not maintained as the records are sorted.

rwfglob

Print to the standard output the list of files that rwfilter would normally process for a given set of file selection switches.

rwfileinfo

Print information (type, version, etc.) about a SiLK Flow, IPset, or Bag file.

rwgeoip2ccmap

Create the country code prefixmap required by ccfilter from the MaxMind GeoIP database.

rwidsquery

Invoke rwfilter to find flow records matching Snort signatures.

rwip2cc

Map a (textual) list of IP addresses to country.

rwrandomizeip

Generate a new flow file by substituting a non-routable IP address for the source and destination IP addresses in given input file.

rwresolve

Read pipe (|) delimited text from the standard input, attempt to resolve the IP addresses in the specified column(s) to host names, and print result to the standard output.

rwsplit

Generate a set of subfiles from the input. The subfiles can be limited by flow, byte, packet, or unique IP count, and the subfile may contain all the flows or only a sample of them.

rwswapbytes

Generate a new flow file by changing the byte order of values in a given input file.

rwtuc

Generate SiLK flow records from textual input; the input should be in a form similar to what rwcut generates.

SiLK Packing System

The SiLK Packing System are daemon applications that collect flow data (IPFIX flows from yaf or NetFlow V5 PDUs from a router) and convert them into a more space efficient format, recording the packed records into service-specific binary flat files. Files are organized in a time-based directory heirarchy with files covering each hour at the leaves.

Installation and set up is described in the SiLK Installation Handbook (pdf).

The tools that make up the SiLK Packing System are:

flowcap

Listen to devices which produce flow data (flow sources) and store it a incremental files in a flat directory.

rwflowappend

Watch a directory for files containing small numbers of SiLK flow records (incremental files) and append those records to hourly files stored in a directory tree.

rwflowpack

Read flow data from a socket or from a file and pack the flow records into hourly flat-files organized in a time-based directory structure.

rwpackchecker

Read SiLK Flow records and look for "unusual" patterns that may indicate data file corruption.

rwreceiver

Accept files transferred from one or more rwsender processes and store them in a destination directory. Either rwsender or rwreceiver may act as the server with the other acting as the client.

rwsender

Watch an incoming directory for files, move the files into a processing directory, and transfer the files to one or more rwreceiver processes. Either rwsender or rwreceiver may act as the server with the other acting as the client.

sensor.conf

Configuration file for sensors and probes.