NAME

rwipfix2silk - Convert IPFIX records to SiLK Flow records

SYNOPSIS

  rwipfix2silk [--silk-output=PATH] [--print-statistics]
        [--interface-values={snmp | vlan}]
        [--log-destination={stdout | stderr | none | PATH}]
        [--log-flags=FLAGS] [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD]
        {[--xargs] | [--xargs=FILENAME] | [IPFIXFILE [IPFIXFILE...]]}

  rwipfix2silk --help

  rwipfix2silk --version

DESCRIPTION

rwipfix2silk reads IPFIX (Internet Protocol Flow Information eXport) records from files or from the standard input, converts the records to the SiLK Flow format, and writes the SiLK records to the path specified by --silk-output or to the standard output when stdout is not the terminal and --silk-output is not provided.

rwipfix2silk reads IPFIX records from the files named on the command line or from the standard input when no file names are specified and --xargs is not present. To read the standard input in addition to the named files, use - or stdin as a file name. When the --xargs switch is provided, rwipfix2silk reads the names of the files to process from the named text file or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--silk-output=PATH

Write the binary SiLK Flow records to PATH, where PATH is a filename, a named pipe, the keyword stderr to write the output to the standard error, or the keyword stdout or - to write the output to the standard output. If PATH names an existing file, rwipfix2silk exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this switch is not given, the output is written to the standard output. Attempting to write the binary output to a terminal causes rwipfix2silk to exit with an error.

Print, to the standard error, the number of records that were written to the SiLK output file. See also --log-destination.

--interface-values={snmp | vlan}

Specify which IPFIX fields should be stored in the input and output fields of the generated SiLK Flow records. If this switch is not specified, the default is snmp. The choices are:

snmp

Store the indexes of the network interface cards where the flows entered and left the router. That is, store the ingressInterface in input and the egressInterface in output.

vlan

Store the VLAN identifiers for the source and destination networks. That is, store vlanId in input and postVlanId in output. If only one VLAN ID is available, input is set to that value and output is set to 0.

--log-destination={none | stdout | stderr | PATH}

Write more detailed information to the specified destination. The default destination is none which suppresses messages. Use stdout or stderr to send messages to the standard output or standard error, respectively. Any other value is treated as a file name in which to write the messages. When an existing file is specified, rwipfix2silk appends any messages to the file. Information that is written includes the following:

  • For each input stream, the number of forward and reverse IPFIX records read and number of records ignored.

  • Messages about invalid records.

  • When the SILK_IPFIX_PRINT_TEMPLATES environment variable is set to 1, the IPFIX templates that were read.

  • Additional messages enabled by the --log-flags switch.

--log-flags=FLAGS

Write additional messages regarding the IPFIX data to the --log-destination, where FLAGS is a comma-separated list of names specifying the type messages to write. When this switch is not specified, the default value for FLAGS is none. This switch takes the same values as the log-flags setting in the sensor.conf(5) file. This manual page documents the values that are relevant for IPFIX data. Since SiLK 3.10.2.

all

Log everything.

default

Enable the default set of log-flags used by sensor.conf: sampling. Despite the name, this is not the default setting for this switch; none is.

none

Log nothing. It is an error to combine this log-flag name with any other. This is the default setting for --log-flags.

record-timestamps

Log the timestamps that appear on each record. This produces a lot of output, and it is primarily used for debugging.

sampling

Write messages constructed by parsing the IPFIX Options Templates that specify the sampling algorithm (when samplingAlgorithm and samplingInterval IEs are present) or flow sampler mode (when flowSamplerMode and flowSamplerRandomInterval IEs are present).

--note-add=TEXT

Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME

Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD

Specify the compression library to use when writing output files. If this switch is not given, the value in the SILK_COMPRESSION_METHOD environment variable is used if the value names an available compression method. When no compression method is specified, output to the standard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

none

Do not compress the output using an external library.

zlib

Use the zlib(3) library for compressing the output, and always compress the output regardless of the destination. Using zlib produces the smallest output files at the cost of speed.

lzo1x

Use the lzo1x algorithm from the LZO real time compression library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead.

snappy

Use the snappy library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead. Since SiLK 3.13.0.

best

Use lzo1x if available, otherwise use snappy if available, otherwise use zlib if available. Only compress the output when writing to a file.

--xargs
--xargs=FILENAME

Read the names of the input files from FILENAME or from the standard input if FILENAME is not provided. The input is expected to have one filename per line. rwipfix2silk opens each named file in turn and reads records from it as if the filenames had been listed on the command line.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash (\) is used to indicate a wrapped line.

To convert a packet capture (pcap(3)) file, packets.pcap, such as that produced by tcpdump(1), to the SiLK format, use the yaf(1) tool (see http://tools.netsa.cert.org/yaf/) to convert the capture data to IPFIX and rwipfix2silk to convert the IPFIX data to the SiLK format, storing the records in silk.rw:

 $ yaf --silk --in packets.pcap --out -             \
   | rwipfix2silk --silk-output=silk.rw

Note that you can produce the same result using the rwp2yaf2silk(1) wrapper script:

 $ rwp2yaf2silk --in packets.pcap --out silk.rw

You can use rwsilk2ipfix(1) to convert the SiLK file back to an IPFIX format, storing the result in ipfix.dat:

 $ rwsilk2ipfix --silk-output=silk.rw ipfix.dat

If you want to create flow records that contain a single packet (similar to the output of rwptoflow(1)), specify --idle-timeout=0 on the yaf command line:

 $ yaf --silk --in packets.pcap --out - --idle-timeout=0    \
   | rwipfix2silk --silk-output=silk.rw

To have yaf decode VLAN identifiers for 802.1Q packets and to have rwipfix2silk store the VLAN IDs in the input and output fields of the SiLK Flow records, use:

 $ yaf --silk --in packets.pcap --out -                             \
   | rwipfix2silk --silk-output=silk.rw --interface-values=vlan

Note: yaf releases prior to 1.3 would only export the VLAN identifiers when the --mac switch was provided on the command line.

ENVIRONMENT

SILK_IPFIX_PRINT_TEMPLATES

When set to 1, rwipfix2silk writes messages to the log file describing each IPFIX template it reads. (Use --log-destination to change the destination from its default of none.) The first message includes the domain, the template identifier, the number of information elements in the template, and the name of this environment variable. Next, a message is printed for each information element in the template where the message contains the domain id, the template id, and the element's position in the template, length in octets, numeric information element identifier, and name. For elements defined by a private enterprise, the IE number has two parts: the private enterprise number and the information element number, separated by a slash (/). (Requires libfixbuf 1.4.0 or later.) Since SiLK 3.8.2.

SILK_LIBFIXBUF_SUPPRESS_WARNINGS

When set to 1, rwipfix2silk disables all warning messages generated by libfixbuf. These warning messages include out-of-sequence packets, data records not having a corresponding template, record count discrepancies, and issues decoding list elements. Since SiLK 3.10.0.

SILK_CLOBBER

The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SILK_COMPRESSION_METHOD

This environment variable is used as the value for --compression-method when that switch is not provided. Since SiLK 3.13.0.

SEE ALSO

rwsilk2ipfix(1), rwfileinfo(1), rwp2yaf2silk(1), rwptoflow(1), sensor.conf(5), silk(7), yaf(1), tcpdump(1), pcap(3), zlib(3)