Use of the SiLK system and related source code is subject to the terms of the following licenses:

Contents

Introduction

The SiLK Reference Guide contains the manual page for each analysis tool, utility, plug-in, file format, and collection facility in the SiLK Collection and Analysis Suite.

This document is meant for reference only. The SiLK Analysis Handbook provides both a tutorial for learning about the tools and examples of how they can be used in analyzing flow data. See the SiLK Installation Handbook for instructions on installing SiLK at your site.

This reference guide is broken into sections like the traditional UNIX manual: end-user analysis tools and utilities are described in Section 1; the plug-ins that augment the behavior of some tools are presented in Section 3; Section 5 contains information about file formats; miscellaneous information is in Section 7; and commands for the installer and administor of SiLK appear in Section 8.

 1
SiLK Analysis Tools and Utilities

This section provides the manual page for each analysis tool and utility that the users of SiLK may employ in their day-to-day work.

mapsid

Map sensor name to sensor number or vice versa

SYNOPSIS

DESCRIPTION

mapsid is a utility that maps sensor names to sensor numbers or vice versa depending on the input arguments. When no arguments are given, the mapping of all sensor numbers to names is printed. When a numeric argument is given, the number to name mapping is printed for the specified argument. When a name is given, its numeric id is printed. For convenience when typing in sensor names, the case is irrelevant.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as –arg=param or –arg param, though the first form is required for options that take optional parameters.

–print-classes

For each sensor, print the classes for which the sensor collects data.

–site-config-file=FILENAME

Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the –version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application’s directory.

EXAMPLES

Name to number mapping:

Number to name mapping:

Print all mappings:

ENVIRONMENT

SILK_CONFIG_FILE

This environment variable is used as the value for the –site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR

When the –site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, mapsid looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH

This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, mapsid checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.

SEE ALSO

rwfilter(1), rwcut(1)

num2dot

Convert an integer IP to dotted-decimal notation

SYNOPSIS

DESCRIPTION

num2dot is a filter to speedup sorting of IP numbers and yet result in both a natural order (i.e., 29.23.1.1 will appear before 192.168.1.1) and readable output (i.e., dotted decimal rather than an integer representation of the IP number).

It is designed specifically to deal with the output of rwcut(1). Its job is to read stdin and convert specified fields (default field 1) separated by a delimiter (default ’∣’) from an integer number into a dotted decimal IP address. Up to three IP fields can be specified via the –ip-fields=FIELDS option. The –delimiter option can be used to specify an alternate delimiter.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as –arg=param or –arg param, though the first form is required for options that take optional parameters.

–ip-fields=FIELDS

Column number of the input that should be considered IP numbers. Column numbers start from 1. If not specified, the default is 1.

–delimiter=C

The character that separates the columns of the input. Default is ’∣’.

EXAMPLE

In addition to the default fields of 1-12 produced by rwcut, you also want to prefix each row with an integer form of the destination IP and the start time to make processing by another tool (e.g., a spreadsheet) easier. However, within the default rwcut output fields of 1-12, you want to see dotted-decimal IP addresses.

The first six columns produced by rwcut will be dIP, sTime, sIP, dIP, sPort, dPort. The –integer-ip switch makes the first, third, and fourth columns be integers, but you only want the first column to be an integer representation. The pipe through num2dot will convert the third and fourth columns to dotted-decimal IP numbers.

SEE ALSO

rwcut(1)

BUGS

num2dot has no support for IPv6 addresses.

rwaddrcount

Count activity by IP address

SYNOPSIS

DESCRIPTION

rwaddrcount reads SiLK Flow records from files named on the command line or from the standard input, sums the byte-, packet-, and record-counts by individual source or destination IP address and maintains the time window during which that IP address was active. At the end of the count operation, the results per IP address are displayed when the –print-recs switch is given. rwaddrcount includes facilities for displaying only those IP address whose byte-, packet- or flow-counts are between specified minima and maxima.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as –arg=param or –arg param, though the first form is required for options that take optional parameters.

For the application to operate, one of the three –print options must be chosen.

–print-recs

Print out count records: IP address, number of bytes, number of packets, number of filter records, earliest start time and latest end time.

–print-ips

Print out IP addresses exclusively

–print-stat

Print the following statistics for all SiLK flows that were read and for those meeting the minima and maxima criteria: byte, packet, and flow record counts and the number of unique IP addresses.

–use-dest

Count by destination IP address in the filter record rather than source IP.

–byte-min=BYTEMIN

Filtering criterion; for the final output (stats or printing), only include count records where the total number of bytes exceeds BYTEMIN

–packet-min=PACKMIN

Filtering criterion; for the final output (stats or printing), only include count records where the total number of packets exceeds PACKMIN

–rec-min=RECMIN

Filtering criterion; for the final output (stats or printing), only include count records where the total number of filter records contributing to that count record exceeds RECMIN.

–byte-max=BYTEMAX

Filtering criterion; for the final output (stats or printing), only include count records where the total number of bytes is less than BYTEMAX.

–packet-max=PACKMAX

Filtering criterion; for the final output (stats or printing), only include count records where the total number of packets is less than PACKMAX.

–rec-max=RECMAX

Filtering criterion; for the final output (stats or printing), only include count records which at most RECMAX filter records contributed to.

–set-file=PATHNAME

Write the IPs into the rwset(1)-style binary IP-set file named PATHNAME. Use rwsetcat(1) to see the contents of this file.

–integer-ips

For the –print-recs and –print-ips output formats, print the IPs as integers. By default, IP addresses are printed as dotted decimal.

–zero-pad-ips

For the –print-recs and –print-ips output formats, print IP addresses as dotted decimal, but use three digits per octet by adding zero-padding, e.g, 000.000.000.000.

–sort-ips

For the –print-recs and –print-ips output formats, the results are presented sorted by IP address.

–no-titles

Turn off column titles. By default, titles are printed.

–no-columns

Disable fixed-width columnar output.

–column-separator=C

Use specified character between columns and after the final column. When this switch is not specified, the default of ’∣’ is used.

–no-final-delimiter

Do not print the column separator after the final column. Normally a delimiter is printed.

–delimited

–delimited=C

Run as if –no-columns –no-final-delimiter –column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default ’∣’.

–print-filenames

Print to the standard error the names of input files as they are opened.

–copy-input=PATH

Copy all binary input to the specified file or named pipe. PATH can be stdout to print flows to the standard output as long as the –output-path switch has been used to redirect rwaddrcount’s ASCII output.

–output-path=PATH

Determine where the output of rwaddrcount (ASCII text) is written. If this option is not given, output is written to the standard output.

–pager=PAGER_PROG

When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the value of the pager is determined to be the empty string, no paging will be performed and all output will be printed to the terminal.

–site-config-file=FILENAME

Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the –version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application’s directory.

–legacy-timestamps

–legacy-timestamps=NUM

Specify the format for human readable timestamps, either the default (new) style, YYYY/MM/DDThh:mm:ss , or the legacy style, MM/DD/YYYY hh:mm:ss . When this switch is not present, the timestamps will be in the default format. When this switch is present and no argument is given, timestamps are in the legacy format. When an argument is supplied, timestamps will be in the new format if the argument begins with 0, and in the old format if the argument begins with 1. Any other argument to the switch is an error.

EXAMPLES

To print out a set of IP’s with exactly one tcp record during the time period, use:

In general, to print out record information, use rwaddrcount with –print-rec

ENVIRONMENT

SILK_PAGER

When set to a non-empty string, rwcut automatically invokes this program to display its output a screen at a time. If set to an empty string, rwcut does not automatically page its output.

PAGER

When set and SILK_PAGER is not set, rwcut automatically invokes this program to display its output a screen at a time.

SILK_CONFIG_FILE

This environment variable is used as the value for the –site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR

When the –site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwaddrcount looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH

This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwaddrcount checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.

SEE ALSO

rwfilter(1), rwset(1), rwsetcat(1), rwstats(1), rwtotal(1), rwuniq(1)

NOTES

When used in an IPv6 environment, rwaddrcount will attempt to convert any IPv6 addresses to IPv4. Records that can be converted will be processed, all other records will be silently ignored.

rwaddrcount uses a fairly large hashtable to store data, but it is likely that as the amount of data expands, the application will take more time to process data.

Similar binning of records are produced by rwstats(1), rwtotal(1), and rwuniq(1).

To generate a list of IP addresses without the volume information, use rwset(1).

rwappend

Append SiLK Flow file(s) to an existing SiLK Flow file

SYNOPSIS

DESCRIPTION

rwappend reads SiLK Flow records from the specified SOURCE_FILEs and appends them to the TARGET_FILE. If stdin is used as the name of one of the SOURCE_FILEs, SiLK flow records will be read from the standard input.

When the TARGET_FILE does not exist and the –create switch is not provided, rwappend will exit with an error. When –create is specified and TARGET_FILE does not exist, rwappend will create the TARGET_FILE using the same format, version, and byte-order as the specified TEMPLATE_FILE. If no TEMPLATE_FILE is given, the TARGET_FILE is created in the default format and version (the same format that rwcat(1) would produce).

The TARGET_FILE must not be compressed (i.e., in may not end in the .gz extension).

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as –arg=param or –arg param, though the first form is required for options that take optional parameters.

–create

–create=TEMPLATE_FILE

Create the TARGET_FILE if it does not exist. The file will have the same format, version, and byte-order as the TEMPLATE_FILE if it is provided; otherwise the defaults are used. The TEMPLATE_FILE will NOT be appended to TARGET_FILE unless it also appears in as the name of a SOURCE_FILE.

–print-statistics

Print the number of records read from each SOURCE_FILE and the total number of records appened to the TARGET_FILE.

–site-config-file=FILENAME

Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the –version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application’s directory.

EXAMPLES

Standard usage where results.dat exists:

To append files sample*.dat to results.dat, or to create results.dat using the same format as the first file argument (note that sample1.dat must be repeated):

If results.dat does not exist, the following two commands are equivalent:

ENVIRONMENT

SILK_CONFIG_FILE

This environment variable is used as the value for the –site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR

When the –site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwappend looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH

This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwappend checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.

SEE ALSO

rwcat(1)

BUGS

When used in an IPv6 environment, rwappend will convert IP addresses into the form used by the TARGET_FILE. Any records containing IP addresses that cannot be converted will be silently ignored.

rwappend makes some attempts to avoid appending a file to itself (which would eventually exhaust the disk space) by comparing the names of files it is given; it should be smarter about this.

rwbag

Build a binary Bag from SiLK Flow records.

SYNOPSIS

DESCRIPTION

rwbag reads SiLK Flow records and builds a Bag. Source IP address, destination IP address, next hop IP address, source port, destination port, protocol, input interface index, output interface index, or sensor ID may be used as the unique key by which to count volumes. Flows, packets, or bytes may be used as the counter. rwbag attempts to read raw flow records from the standard input or from any INPUTFILE arguments. INPUTFILE may also explicitly be the keyword stdin. If the raw flow records do not contain the proper key and counter fields, rwbag prints an error to stderr and exits abnormally.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as –arg=param or –arg param, though the first form is required for options that take optional parameters.

At least one of the following output flags must be defined. For each, OUTPUTFILE is the name of a non-existent file, a named pipe, or the keyword stdout to write the binary Bag to the standard output. Only one switch may use the standard output as its output stream.

–sip-flows=OUTPUTFILE

Count number of flows by unique source IP.

–sip-packets=OUTPUTFILE

Count number of packets by unique source IP.

–sip-bytes=OUTPUTFILE

Count number of bytes by unique source IP.

–dip-flows=OUTPUTFILE

Count number of flows by unique destination IP.

–dip-packets=OUTPUTFILE

Count number of packets by unique destination IP.

–dip-bytes=OUTPUTFILE

Count number of bytes by unique destination IP.

–sport-flows=OUTPUTFILE

Count number of flows by unique source port.

–sport-packets=OUTPUTFILE

Count number of packets by unique source port.

–sport-bytes=OUTPUTFILE

Count number of bytes by unique source port.

–dport-flows=OUTPUTFILE

Count number of flows by unique destination port.

–dport-packets=OUTPUTFILE

Count number of packets by unique destination port.

–dport-bytes=OUTPUTFILE

Count number of bytes by unique destination port.

–proto-flows=OUTPUTFILE

Count number of flows by unique protocol.

–proto-packets=OUTPUTFILE

Count number of packets by unique protocol.

–proto-bytes=OUTPUTFILE

Count number of bytes by unique protocol.

–sensor-flows=OUTPUTFILE

Count number of flows by unique sensor ID.

–sensor-packets=OUTPUTFILE

Count number of packets by unique sensor ID.

–sensor-bytes=OUTPUTFILE

Count number of bytes by unique sensor ID.

–input-flows=OUTPUTFILE

Count number of flows by unique input interface index.

–input-packets=OUTPUTFILE

Count number of packets by unique input interface index.

–input-bytes=OUTPUTFILE

Count number of bytes by unique input interface index.

–output-flows=OUTPUTFILE

Count number of flows by unique output interface index.

–output-packets=OUTPUTFILE

Count number of packets by unique output interface index.

–output-bytes=OUTPUTFILE

Count number of bytes by unique output interface index.

–nhip-flows=OUTPUTFILE

Count number of flows by unique next hop IP.

–nhip-packets=OUTPUTFILE

Count number of packets by unique next hop IP.

–nhip-bytes=OUTPUTFILE

Count number of bytes by unique next hop IP.

–note-add=TEXT

Add the specified TEXT to the header of every output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

–note-file-add=FILENAME

Open FILENAME and add the contents of that file to the header of every output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

–compression-method=COMP_METHOD

Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the –help and –version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

none

Do not compress the output using an external library

zlib

Use the zlib(3) library for compressing the output

lzo1x

Use the lzo1x algorithm from the LZO real time compression library for compression

best

Use whichever available method gives the best compression in general, though not necessarily the best for this particular output.

–print-filenames

Prints to the standard error the names of input files as they are opened.

–copy-input=PATH

Copy all binary input to the specified file or named pipe. PATH can be stdout to print flows to the standard output as long as the –output-path switch has been used to redirect rwuniq’s ASCII output.

–site-config-file=FILENAME

Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the –version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application’s directory.

–legacy-help

Print the usage information for rwbag and include the names of the deprecated options in the output.

The following options are deprecated.

–sf-file=OUTPUTFILE

Deprecated alias for –sip-flows.

–sp-file=OUTPUTFILE

Deprecated alias for –sip-packets.

–sb-file=OUTPUTFILE

Deprecated alias for –sip-bytes.

–df-file=OUTPUTFILE

Deprecated alias for –dip-flows.

–dp-file=OUTPUTFILE

Deprecated alias for –dip-packets.

–db-file=OUTPUTFILE

Deprecated alias for –dip-bytes.

–port-sf-file=OUTPUTFILE

Deprecated alias for –sport-flows.

–port-sp-file=OUTPUTFILE

Deprecated alias for –sport-packets.

–port-sb-file=OUTPUTFILE

Deprecated alias for –sport-bytes.

–port-df-file=OUTPUTFILE

Deprecated alias for –dport-flows.

–port-dp-file=OUTPUTFILE

Deprecated alias for –dport-packets.

–port-db-file=OUTPUTFILE

Deprecated alias for –dport-bytes.

–proto-f-file=OUTPUTFILE

Deprecated alias for –proto-flows.

–proto-p-file=OUTPUTFILE

Deprecated alias for –proto-packets.

–proto-b-file=OUTPUTFILE

Deprecated alias for –proto-bytes.

EXAMPLES

To build both source IP and destination IP Bags of flows:

ENVIRONMENT

SILK_CONFIG_FILE

This environment variable is used as the value for the –site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR

When the –site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwbag looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH

This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwbag checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.

SEE ALSO

rwbagbuild(1), rwbagcat(1), rwbagtool(1), rwfileinfo(1), rwfilter(1)

BUGS

Currently there is no support for Bag files keyed by an IPv6 address.

When used in an IPv6 environment, rwbag will process every record when creating Bags that are not keyed by the IP address. For Bags keyed by the IP address, rwbag will attempt to convert any IPv6 addresses to IPv4. Records that can be converted will be processed, all other records will be silently ignored for the IP-keyed Bags, but will be used for any non-IP-keyed Bags.

rwbagbuild

Create a binary Bag from non-flow data.

SYNOPSIS

DESCRIPTION

rwbagbuild builds a binary Bag file from an IPset file or from textual input.

When creating a Bag from an IPset, the value associated with each IP address is the value given by the –default-count switch, or 1 if the switch isn’t provided.

The textual input read from the argument to the –bag-input switch is processed a line at a time. Comments begin with a ’#’-character and continue to the end of the line; they are stripped from each line. Any line that is blank or contains only whitespace is ignored. All other lines must contain a valid key or key-count pair; whitespace around the key and count is ignored.

If the delimiter character (specified by the –delimiter switch and having pipe (’∣’) as its default) is not present, the line must contain only an IP address or an integer key. If the delimiter is present, the line must contain an IP address or integer key before the delimiter and an integer count after the delimiter. These lines may have a second delimiter after the integer count; the delimiter and any text to the right of it are ignored.

When the –default-count switch is specified, its value will used as the count for each key, and the count value parsed from each line, if any, is ignored. Otherwise, the parsed count is used, or 1 is used as the count if no delimiter was present.

For each key-count pair, the key will be inserted into Bag with its count or, if the key is already present in the Bag, its total count will be incremented by the count from this line.

The IP address or integer key must be expresed in one of these formats:

• Dotted decimal—all 4 octets are required:

  10.1.2.4

• An unsigned 32-bit integer:

  167838212

• Either of the above with a CIDR designation—for dotted decimal all four octets are still required:

  10.1.2.4/31     167838212/31

• SiLK wildcard notation: Four octets separated by periods where each octet may be a single number, a range of numbers, e.g., 1-10, a comma separated list of numbers and ranges, or the character ’x’ used to represent all values in an octet, that is 0-255:

  10.x.1-2.4,5

If an IP address or count cannot be parsed, or if a line contains a delimiter character but no count, rwbagbuild prints an error and exits.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as –arg=param or –arg param, though the first form is required for options that take optional parameters.

The following two switches control the type of input; one and only one must be provided:

–set-input=SETFILE

Create a Bag from an IPset. SETFILE is a filename, a named pipe, or the keyword stdin. Counts have a volume of 1 unless overridden with –default-count.

–bag-input=TEXTFILE

Create a Bag from a delimited text file. TEXTFILE is a filename, a named pipe, or the keyword stdin. See the DESCRIPTION section for the syntax of the TEXTFILE.

–delimiter=C

The delimiter to expect between each key-count pair of the TEXTFILE read by the –bag-input switch. The delimiter is ignored if the –set-input switch is specified. Since ’#’ is used to denote comments and newline is used to used to denote records, neither is a valid delimiter character.

–default-count=DEFAULTCOUNT

Override the counts of all values in the input bag or set with the value of DEFAULTCOUNT. DEFAULTCOUNT must be a positive integer.

–note-add=TEXT

Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

–note-file-add=FILENAME

Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

–compression-method=COMP_METHOD

Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the –help and –version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

none

Do not compress the output using an external library

zlib

Use the zlib(3) library for compressing the output

lzo1x

Use the lzo1x algorithm from the LZO real time compression library for compression

best

Use whichever available method gives the best compression in general, though not necessarily the best for this particular output.

–output-path=OUTPUTFILE

Redirect output to OUTPUTFILE. OUTPUTFILE is a filename, named pipe, or the keyword stdout.

EXAMPLES

Assume the file mybag.txt contains the following (ignore leading whitespace and every line ends with a newline):

To build a bag with it:

Or given the IP set myset.set, create a bag where every entry in the set has a count of 3:

SEE ALSO

rwbag(1), rwbagcat(1), rwbagtool(1), rwfileinfo(1), rwset(1)

rwbagcat

Output a binary Bag as text.

SYNOPSIS

DESCRIPTION

rwbagcat reads a binary Bag, converts it to text, and outputs it to the standard output or the specified file. It can also print various statistics and summary information about the Bag.

rwbagcat reads the BAGFILEs specified on the command line; if no BAGFILE arguments are given, rwbagcat attempts to read the Bag from the standard input. BAGFILE may also explicitly be the keyword stdin to allow rwbagcat to combine files and piped input. If any input does not contain a Bag, rwbagcat prints an error to stderr and exits abnormally.

When multiple BAGFILEs are specified, each is handled individually; to process the combination of the BAGFILEs, invoke rwbagcat on the output from rwbagtool(1).

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as –arg=param or –arg param, though the first form is required for options that take optional parameters.

–network-structure

–network-structure={T ∣ A ∣ B ∣ C ∣ X ∣ H}...[S]

Print the network structure around the keys.

T

total sum of all keys

A

class A (/8) network header and subtotal

B

class B (/16) network header and subtotal

C

class C (/24) network header and subtotal

X

(/27) network header and subtotal

H

individual IP addresses and counter

S

count of /8,/16,/24,/27s in each smaller subnet

If no paramter is specified to the –network-structure switch, the default is TS. Any combination of T,A,B,C,X,H may be specified; the value S by itself is illegal.

–bin-ips

–bin-ips=SCALE

Invert the bag and count the total number of unique IP addresses for a given value of the volume bin. For example, turn a Bag {sip:flow} into {flow:count(sip)}. SCALE is a string containing the value linear, binary, or decimal.

• The default behavior is linear: Each distinct counter gets its own bin. Any counter in the input Bag file that is larger than the maximum possible key will be attributed to the maximum key; to prevent this, specify --maxcounter=4294967295.

• binary creates a bag of {log2(flow):count(sip)}. Bin n contains counts in the range [ 2^n, 2^(n+1) ).

• decimal creates one hundred bins for each counter in the range [1,100), and one hundred bins for each counter in the range [100,1000), each counter in the range [1000,10000), etc. Counters are logarithmically distributed among the bins.

–stats

–stats=OUTFILE

Print out breakdown of the network hosts seen, and print out general statistics about the keys and counters.

• count of unique keys

• sum of all the counters

• minimum key

• maximum key

• minimum counter

• maximum counter

• mean of counters

• variance of counters

• standard deviation of counters

• skew of counters

• kurtosis of counters

OUTFILE is a filename, named pipe, or one of the keywords stdout or stderr. Defaults to printing on stderr unless output is being paged, in which case output is to stdout.

–tree-stats

–tree-stats=OUTFILE

Print out metadata about how the bag is performing:

• count of nodes allocated

• total bytes allocated for nodes

• count of leaves allocated

• total bytes allocated for leaves

• count of keys entered

• density of data

OUTFILE is a filename, named pipe, or one of the keywords stdout or stderr. Defaults to printing on stdout.

–mincounter=VALUE

Only output records whose minimum counter value is VALUE or higher. The valid range of VALUE is 1 to 18446744073709551615. The default is to print all records with non-zero counter; use –zero-counts to show records whose counter is 0.

–maxcounter=VALUE

Only output records whose maximum counter value is VALUE or lower. The valid range of VALUE is 1 to 18446744073709551615, with the default being the maximum counter value.

–minkey=VALUE

Only output records whose minimum key value is VALUE or higher. The valid range is of VALUE 0 to 4294967295, or 0.0.0.0 to 255.255.255.255. Default is 0 (for port or protocol) or 0.0.0.0 (for IP address). Accepts dotted decimal or integer notation.

–maxkey=VALUE

Only output records whose maximum key value is VALUE or lower. The valid range of VALUE is 0 to 4294967295, or 0.0.0.0 to 255.255.255.255. Default is all ports or protocols, or the maximum IP address 255.255.255.255. Accepts dotted decimal or integer notation.

–zero-counts

Print keys whose counter is zero. Normally, keys with a counter of zero are suppressed since all keys have a default counter of zero. In order to use this flag, –minkey and –maxkey must be specified. When this switch is specified, any counter limit explicitly set by the –maxcounter switch will still be applied.

–output-path=OUTPUTFILE

Redirect output of the –network-structure or –bin-ips options to OUTPUTFILE. OUTPUTFILE is a filename, named pipe, or the keyword stdout.

–zero-pad-ips

Pad IP address octets with zeros so that every octet is three characters wide.

–integer-keys

Print the keys as integers. This flag should be used if the bag is a port or protocol bag.

–no-columns

Disable fixed-width columnar output.

–column-separator=C

Use specified character between columns and after the final column. When this switch is not specified, the default of ’∣’ is used.

–no-final-delimiter

Do not print the column separator after the final column. Normally a delimiter is printed. When the network summary is requested (–network-structure=S), the separator is always printed before the summary column and never that column.

–delimited

–delimited=C

Run as if –no-columns –no-final-delimiter –column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default ’∣’.

–pager=PAGER_PROG

When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the value of the pager is determined to be the empty string, no paging will be performed and all output will be printed to the terminal.

EXAMPLES

To print the bag:

To print it with full network: