Contents

Analysis Suite

The SiLK analysis suite is a collection of command-line tools for processing SiLK Flow records created by the SiLK packing system. These tools read binary files containing SiLK Flow records and partition, sort, and count these records. The most important analysis tool is rwfilter, an application for querying the central data repository for SiLK Flow records that satisfy a set of filtering options. The tools are intended to be combined in various ways to perform an analysis task. A typical analysis uses UNIX pipes and intermediate data files to share data between invocations of the tools.

The tools and plug-in modules that make up the analysis tools are listed below, roughly grouped by functionality.

Filtering, Displaying, and Sorting

rwfilter

Select SiLK Flow records form the data repository and partition the records into one or more 'pass' and/or 'fail' output streams.

rwcut

Print the attributes of SiLK Flow records in a delimited, columnar, human-readable format. Users can define new printable attributes using plug-ins written in C or PySiLK.

rwsort

Sort SiLK Flow records using a user-specified key comprised of record attributes, and write the records to the named output path or to the standard output. Users can define new key fields using plug-ins written in C or PySiLK.

SiLK Python Extension (PySiLK)

PySiLK: SiLK in Python

Read, manipulate, and write SiLK Flow records, IPsets, and Bags from within Python. PySiLK may be used in a stand-alone Python program or to write plug-ins for several SiLK applications. This document describes the objects, methods, and functions that PySiLK provides. The next entry describes using PySiLK from within a plug-in.

silkpython

Use PySiLK to define new partitioning rules for rwfilter, new key fields for rwcut, rwgroup, and rwsort, and new key or value fields for rwstats and rwuniq.

Counting, Grouping, and Mating

rwcount

Summarize (aka group or bin) SiLK Flow records across time, producing textual output with counts of bytes, packets, and flow records for each time bin.

rwuniq

Summarize SiLK Flow records by a user-specified key comprised of record attributes and print columns for the total byte, packet, and/or flow counts for each bin. rwuniq can also count the number of distinct values for a field. Users can define new key fields and value fields using plug-ins written in C or PySiLK.

rwstats

Summarize SiLK Flow records just like rwuniq, but sort the results by a value field to generate a Top-N or Bottom-N list, and print the results.

rwtotal

Summarize SiLK Flow records by a specified key and print the sum of the byte, packet, and flow counts for flows matching the key. rwtotal uses a fixed amount of memory, and it will faster that rwuniq, but it has a limited set of keys.

rwaddrcount

Summarize SiLK flow records by the source or destination IPv4 address and print the byte, packet, and flow counts for each IP.

rwgroup

Group SiLK flow records by a user-specified key comprised of record attributes, label the records with a group ID that is stored in the next-hop IP field, and write the resulting binary flows to the specified output path or to the standard output. rwgroup requires that its input is sorted by the user-specified key.

rwmatch

Match (mate) records as queries and responses, mark mated records with an ID that is stored in the next-hop IP field, and write the binary flow records to the output. rwmatch requires that its input files are sorted.

IPsets, Bags, Aggregate Bags, and Prefix Maps

An IPset is a data structure and a binary file format that contains a list of IP addresses where each IP appears once (a mathematical set).

A Bag is a data structure and a binary file format where a key is mapped to a counter (similar to a hash table or Python dictionary). The key is either a 32-bit number or an IPv6 address, and the counter is a 64-bit number. Usually the key represents an aspect of a flow record (an IP address, a port number, the protocol) and the counter is a volume (the number of flow records, the sum of the packet counts) for the flow records that match that key.

An Aggregate Bag is similar to a Bag except the key and/or the counter may be comprised of multiple fields. Aggregate Bags were introduced in SiLK 3.15.0.

A prefix map is a data structure and file format that maps every IP address to string. An example prefix map gives the two-letter country code for any IP address.

rwset

Read SiLK Flow records and generate binary IPset file(s) containing the source IP addresses or destination IP addresses seen on the flow records.

rwsetbuild

Read (textual) IP addresses in canonical form or in CIDR notation from an input file or from the standard input and write a binary IPset file.

rwsetcat

Print the contents of a binary IPset file as text. Additional information about the IPset file can be printed.

rwsetmember

Determine whether the IP address or CIDR block specified on the command line is contained in an IPset.

rwsettool

Perform union, intersection, difference, and sampling functions on the input IPset files, generating a new IPset file.

rwbag

Read SiLK Flow records and build binary Bag(s) containing key-count pairs. An example is a Bag containing the sum of the byte counts for each source port seen on the flow records.

rwbagbuild

Create a binary Bag file from a binary IPset file or from a textual input file.

rwbagcat

Print binary Bag files as text.

rwbagtool

Perform operations (e.g., addition, subtraction) on binary Bag files and produce a new binary Bag file.

rwaggbag

Read SiLK Flow records and build a binary Aggregate Bag containing key-count pairs. An example is a Aggregate Bag containing the sum of the byte counts for each source port seen on the flow records. Since SiLK 3.15.0.

rwaggbagbuild

Create a binary Aggregate Bag file from a textual input file. Since SiLK 3.15.0.

rwaggbagcat

Prints binary Aggregate Bag files as text. Since SiLK 3.15.0.

rwaggbagtool

Performs operations (e.g., addition, subtraction) on binary Aggregate Bag files and produces a new Aggregate Bag file. Since SiLK 3.15.0.

rwpmapbuild

Read textual input and create a binary prefix map file for use with the Address Type (addrtype) and Prefix Map (pmapfilter) utilities.

rwpmapcat

Print information about a prefix map file as text. By default, print each IP range in the prefix map and its label.

rwpmaplookup

Find information about specific IP address(es) or protocol/port pair(s) in a binary prefix map file and print the result as text.

rwipaimport

Import a SiLK IPset, Bag, or Prefix Map file into the IP Address Association (IPA) data store.

rwipaexport

Export a set of IP addresses from the IP Address Association (IPA) data store to a SiLK IPset, Bag, or Prefix Map.

IP and Port Labeling Files

addrtype

The Address Type file provides a way to map an IPv4 address to an integer denoting the IP as internal, external, or non-routable.

ccfilter

The Country Code file provides a mapping from an IP address to a two-letter, lowercase abbreviation of the country where the IP address is located. The abbreviations used by the Country Code utility are those defined by ISO 3166-1.

pmapfilter

Prefix Map files provide a way to map either IP addresses or protocol/port pairs to string labels based on a user-defined map file. The map file is created by rwpmapbuild.

Run-Time Plug-Ins

To use most of these plug-ins, the plug-in must be explicitly loaded into an application by using the application's --plugin switch and giving the plug-in's library name or path as the argument. For a plug-in NAME, the library is typically named NAME.so.

app-mismatch

The application-mismatch plug-in helps to find services running on unusual or non-typical ports by causing rwfilter to only pass a flow record when the record's application field is non-zero and its value is different than that in the source port and destination port fields.

conficker-c

The conficker-c plug-in was written in March 2009 to detect traffic that matches the signature of the .C variant of the Conficker worm.

cutmatch

The cutmatch plug-in creates a field in rwcut that provides a more user-friendly representation of the match parameter value that rwmatch writes into a SiLK Flow record's next hop IP field.

flowkey

The Flowkey plug-in adds a switch and a field that computes a 32-bit hash for a flow record using the same algorithm as yaf uses for its flow key utility getFlowKeyHash. Since SiLK 3.15.0.

flowrate

The Flowrate plug-in adds switches and fields to compute packets/second, bytes/second, bytes/packet, payload-bytes, and payload-bytes/second.

int-ext-fields

The internal/external plug-in makes available fields containing internal and external IPs and ports (int-ip, ext-ip, int-port, and ext-port). It can be used to print, sort by, or group by the internal or external IP or port, which is useful when a single flow file contains flows in multiple directions.

ipafilter

The IPA (IP Association) plug-in works with rwfilter to partition flows based on data in an IPA data store. rwfilter will automatically load this plug-in if it is available. The plug-in requires that SiLK be compiled with IPA support.

silk-plugin

This page provides information on building your own SiLK plug-ins using C.

Packet and IPFIX Processing

These tools operate on packet capture files, IPFIX files, or files of NetFlow v5 data.

rwp2yaf2silk

Convert a packet capture (pcap) file---such as a file produced by tcpdump---to a single file of SiLK Flow records. rwp2yaf2silk assumes that the yaf and rwipfix2silk commands are available on your system as it is a simple Perl wrapper around those commands.

rwipfix2silk

Convert a stream of IPFIX (Internet Protocol Flow Information eXport) records to the SiLK Flow record format.

rwsilk2ipfix

Convert a stream of SiLK Flow records to an IPFIX (Internet Protocol Flow Information eXport) format.

rwpcut

Read a packet capture file and print its contents in a textual form similar to that produced by rwcut.

rwpdedupe

Detect and eliminate duplicate records from multiple packet capture input files. See also rwdedupe.

rwpmatch

Filter a packet capture file by writing only packets whose five-tuple and timestamp match corresponding records in a SiLK Flow file.

rwptoflow

Read a packet capture file and generate a SiLK Flow record for every packet.

rwpdu2silk

Create a stream of SiLK Flow records from a file containing NetFlow v5 PDU records.

Scan Detection

rwscan

Attempt to detect scanning activity from SiLK Flow records. rwscan can produce files that can be loaded into a database and queried with rwscanquery.

rwscanquery

Query the scan database which has been populated from database load files generated by rwscan.

Flow File Utilities

rwappend

Append the SiLK Flow records contained in the second through final file name arguments to the records contained in the first file name argument.

rwcat

Read SiLK Flow records from the files named on the command line, or from the standard input when no files are provided, and write the SiLK records to the specified output file or to the standard output if it is not connected to a terminal.

rwcombine

Read SiLK Flow records from files named on the command line or from the standard input. For records where the attributes field contains the "flow timed-out" flag, attempt to find the record with the corresponding "continuation" flag set and combine those records into a single flow. Write the results to the named output file or to the standard output. Since SiLK 3.9.0.

rwcompare

Determine whether two SiLK Flow files contain the same flow records.

rwdedupe

Read SiLK Flow records from files named on the command line or from the standard input and write the records to the named output path or to the standard output, removing any duplicate flow records. Note that rwdedupe will reorder the records as part of its processing.

rwnetmask

Read SiLK Flow records, zero the least significant bits of the source-, destination-, and/or next-hop-IP address(es), and write the resulting records to the named output path or to the standard output.

rwrandomizeip

Generate a new SiLK Flow file by substituting a pseudo-random IP address for the source and destination IP addresses in given input file.

rwsplit

Read SiLK Flow records and generate a set of sub-files from the input. The sub-files can be limited by flow-, byte-, or packet-counts, or by unique IP count. In addition, the sub-file may contain all the flows or only a sample of them.

rwswapbytes

Generate a new SiLK Flow file by changing the byte order of the records in a given input SiLK Flow file.

Utilities

rwfileinfo

Print information (type, version, etc.) about a SiLK Flow, IPset, Bag, or Prefix Map file.

rwsiteinfo

Print information about the sensors, classes, and types specified in the silk.conf file.

rwtuc

Generate SiLK Flow records from textual input; the input should be in a form similar to what rwcut generates.

rwfglob

Print to the standard output the list of files that rwfilter would normally process for a given set of file selection switches.

num2dot

Read delimited text from the standard input, convert integer values in the specified column(s) to dotted-decimal IP address, and print the result to the standard output.

rwresolve

Read delimited text from the standard input, attempt to resolve the IP addresses in the specified column(s) to host names, and print the result to the standard output.

rwrecgenerator

Generate SiLK Flow records using a pseudo-random number generator; these records can be used to test SiLK applications.

rwgeoip2ccmap

Create the country code mapping file required by the ccfilter utility from the MaxMind GeoIP database.

rwidsquery

Invoke rwfilter to find flow records matching Snort signatures.

silk_config

Print information about how SiLK was compiled; this information can be used to compile and link other files and programs against the SiLK header files and libraries.

Deprecated tools

mapsid

This command is deprecated; use rwsiteinfo instead. Map between sensor names and sensor IDs using the values specified in the silk.conf file.

rwguess

This command is deprecated; use a combination of rwpdu2silk, rwstats, and rwuniq instead. Read a file containing NetFlow v5 PDU records and print the SNMP interfaces that are used most often and the number of records seen for each interface.

rwip2cc

This command is deprecated; use rwpmaplookup instead. Map a (textual) list of IP addresses to their country code.

SiLK Packing System

The SiLK Packing System is comprised of daemon applications that collect flow data (IPFIX flows from yaf or NetFlow v5 or v9 PDUs from a router) and convert them into a more space efficient format, storing the packed records into service-specific binary flat files for use by the analysis suite. Files are organized in a time-based directory hierarchy with files covering each hour at the leaves.

flowcap

Listen to flow generators (devices which produce network flow data) and store the data in temporary files prior to transferring the files to a remote machine for processing by rwflowpack.

rwflowpack

Read flow data either directly from a flow generator or from files generated by flowcap, convert the data to the SiLK Flow record format, categorize the flow records according to rules loaded from a packing-logic plug-in, and write the records either to hourly flat-files organized in a time-based directory structure or to files for transfer to a remote machine for processing by rwflowappend.

rwflowappend

Watch a directory for files containing small numbers of SiLK flow records and append those records to hourly files organized in a time-based directory tree.

rwsender

Watch an incoming directory for files, move the files into a processing directory, and transfer the files to one or more rwreceiver processes. Either rwsender or rwreceiver may act as the server (i.e., listen for incoming network connections) with the other acting as the client.

rwreceiver

Accept files transferred from one or more rwsender processes and store them in a destination directory. Either rwsender or rwreceiver may act as the server with the other acting as the client.

rwpollexec

Monitor a directory for incoming files and run a user-specified command on each file.

rwpackchecker

Read SiLK Flow records and check for unusual patterns that may indicate data file corruption.

packlogic-twoway

One of the plug-ins available that describe a set of rules (the packing-logic) that rwflowpack may use when categorizing flow records.

packlogic-generic

One of the plug-ins available that describe a set of rules (the packing-logic) that rwflowpack may use when categorizing flow records.

Overview and Configuration

The following manual pages provide general information.

silk.conf

Configuration file naming the Classes, Types, and Sensors available at your installation.

sensor.conf

Configuration file for sensors and probes used by rwflowpack and flowcap.

silk

An overview of SiLK and a list of environment variables that affect SiLK.

Analysis Handbooks and References

an introduction to methods of analyzing network traffic, illustrated by commands from the SiLK tool suite, with the focus on learning to identify traffic features important to the security of information on the network
a quick reference guide to the analysis tools
Updated December 2019 for SiLK 3.18 and later
document containing the PySiLK and SiLK Python manual pages provided as a single reference for manipulating SiLK Flow data from within Python
every SiLK manual page presented in a single document

Installation Information

Quick instructions for building SiLK and YAF on a system that uses YUM/RPM. The instructions use CentOS 7; a RedHat or Fedora system is similar. It also includes instructions on creating your own RPMs of some of the NetSA tools.
Quick instructions for building SiLK YAF on a system that uses APT/DEB. The instructions use Debian 9.8.0; an Ubuntu system is similar.
very detailed instructions for configuring, building, and installing SiLK at a site