NAME

rwptoflow - Generate SiLK Flow records from packet data

SYNOPSIS

  rwptoflow [--dynamic-library=PLUG_IN_PATH]
        [--active-time=YYYY/MM/DD:hh:dd:mm:ss.uuuuuu-YYYY/MM/DD:hh:dd:mm:ss.uuuuuu]
        [--flow-output=FLOW_PATH] [--packet-pass-output=PCKTS_PASS]
        [--packet-reject-output=PCKTS_REJECT]
        [--reject-all-fragments] [--reject-nonzero-fragments]
        [--reject-incomplete] [--set-sensorid=SCALAR]
        [--set-inputindex=SCALAR] [--set-outputindex=SCALAR]
        [--set-nexthopip=IP_ADDRESS] [--print-statistics]
        [--compression-method=COMP_METHOD] TCPDUMP_INPUT

DESCRIPTION

rwptoflow attempts to generate a SiLK Flow record for every IP packet in the pcap(3) (tcpdump(1)) capture file TCPDUMP_INPUT. TCPDUMP_INPUT must contain data captured from an Ethernet datalink.

rwptoflow will read from its standard input if TCPDUMP_INPUT is specified as stdin. The SiLK Flow records are written to the specified flow-output file or to the standard output. The application will fail when attempting to read or write binary data from or to a terminal.

Packets outside of a user-specified active-time window can be ignored. Additional filtering on the TCPDUMP_INPUT can be performed by using tcpdump with an expression filter and piping tcpdump's output into rwptoflow.

In addition to generating flow records, rwptoflow can write pcap files containing the packets that it used to generate each flow, and/or the packets that were rejected. Note that packets falling outside the active-time window are ignored and are not written to the packet-reject-output.

Statistics of the number of packets read, rejected, and written can be printed.

rwptoflow will reject any packet that is not an IPv4 Ethernet packet and any packet that is too short to contain the Ethernet and IP headers. At the user's request, packets may be rejected when

they are fragmented---either the initial (zero-offset) fragment or a subsequent fragment
they have a non-zero fragment offset
they are not fragmented or they are the zero-fragment but the capture file does not contain enough information about the packet to set the ICMP type and code, the UDP source and destination ports, or the TCP source and destination ports and flags

Since the input packet formats do not contain some fields normally found in NetFlow data, rwptoflow provides a way to set those flow values in all packets. For example, it is possible to set the sensor-id manually for a tcpdump source, so that flow data can be filtered or sorted by that value later.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--dynamic-library=PLUG_IN_PATH

Use the specified dynamic library to ignore or reject packets or to modify the flow record that is generated from the packet. See the PLUG-IN SUPPORT section below for details.

--active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]

--active-time=YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]-YYYY/MM/DD[:hh[:dd[:mm[:ss[.uuuuuu]]]]]

Ignore all packets whose time falls outside the specified range. The times must be specified to at least day precision. The start time is required; when the end-time is not present, it is treated as infinite. The end-time will be rounded-up to instant before the next time unit; i.e., an end-time of 2006/08/31:15 is treated as 2006/08/31:15:59:59.999999.

--flow-output=FLOW_PATH

Write the generated SiLK Flow records to the specified file at FLOW_PATH. When this switch is not provided, the flows are written to the standard output.

--packet-pass-output=PCKTS_PASS

For each generated SiLK Flow record, write the packet that generated the flow to the pcap file specified by PCKTS_PASS. Use stdout to write the packets to the standard output.

--packet-reject-output=PCKTS_REJECT

Write each packet that occurs within the active-time window but for which a SiLK Flow record was not generated to the pcap file specified by PCKTS_REJECT. Use stdout to write the packets to the standard output.

The packets that get written to this file may include packets that were shorter than that required to get the IP header, non-IPv4 packets, and packets that get treated as reject packets by the following switches.

--reject-all-fragments

Do not generate a SiLK Flow record for the packet when the packet is fragmented. This includes the initial (zero-offset) fragment and all subsequent fragments. If --packet-reject-output is specified, the packet will be written to that file.

--reject-nonzero-fragments

Do not generate a SiLK Flow record for the packet when the packet is fragmented unless this is the initial fragment. That is, reject all packets that have a non-zero fragmentation offset. Normally flow records are generated for these packets, but the ports and TCP flag information is set to zero. If --packet-reject-output is specified, the packet will be written to that file.

--reject-incomplete

Do not generate a SiLK Flow record for the packet when the packet's fragmentation-offset is zero yet the packet does not contain enough information to completely specify an ICMP, UDP, or TCP record (that is, the packet is too short to set the ICMP type and code, the UDP or TCP source or destination port, or the TCP flags). Normally, flow records are generated for these packets but the ports and TCP flag information is set to zero. This switch has no effect on packets where the protocol is not 1,6, or 17.

This switch does not imply --reject-nonzero-fragments; to indicate that all generated flow records must have valid port and TCP flag information, specify --reject-nonzero-fragments --reject-incomplete.

--set-sensorid=SCALAR

Set the sensor ID for all flows to SCALAR. SCALAR should be an integer value between 0 and 65534, inclusive. When not specified, the sensor ID is set to 65535.

--set-inputindex=SCALAR

Set the input SNMP index value for all flows to SCALAR. SCALAR should be an integer value between 0 and 65535, inclusive. When not specified, the SNMP input is set to 0.

--set-outputindex=SCALAR

Set the output SNMP index value for all flows to SCALAR. SCALAR should be an integer value between 0 and 65535, inclusive. When not specified, the SNMP output is set to 0.

--set-nexthopip=IP_ADDRESS

Set the next-hop IP address for all flows to IP_ADDRESS; IP_ADDRESS may be in dotted-decimal notation or an integer. When not specified, the next-hop IP is set to 0.0.0.0.

--print-statistics

Print a summary of the packets that were processed. This summary includes

the total number of packets read
the number that fell outside the time-window
the number that were too short to get the IP header
the number that were not IPv4
the number that were discarded by a plug-in
the total number of fragmented packets
the number of fragments where the offset was zero
the number of zero-offset packets that were incomplete
the number of flows written to the output

--compression-method=COMP_METHOD

Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

none

Do not compress the output using an external library

zlib

Use the zlib(3) library for compressing the output

lzo1x

Use the lzo1x algorithm from the LZO real time compression library for compression

best

Use whichever available method gives the best compression in general, though not necessarily the best for this particular output.

PLUG-IN SUPPORT

rwptoflow allows the user to provide additional logic to ignore or reject packets, or to modify the flow record that is generated from the packet. To do this, the user creates a plug-in that gets loaded at run-time by giving rwptoflow the --dynamic-library switch with the path to the plug-in as the parameter to the switch.

A plug-in is a shared object file (a.k.a. dynamic library) that is compiled from C source code. The plugin should have four subroutines defined:

setup()

is called when the object is first loaded. This is the place to initialize global variables to their default values. If the plug-in provides switches of its own, they must be registered in this subroutine.

initialize()

gets called after all options have been processed but before any packets are read from the input. If this subroutine does not return 0, the application will quit.

ptoflow()

will be called for every packet that rwptoflow is able to convert into a flow record just before the flow record is written. This subroutine will not see packets that are short or that are not IPv4; it will also not see fragmented packets if --reject-all-fragments is specified.

The ptoflow() function is called with two parameters:

a pointer to the rwRec object that rwptoflow created from the packet. The subroutine may modify the record as it sees fit.
a void pointer that the function may cast to a pointer to the C structure:

  typedef struct _sk_pktsrc_t {
      /* the source of the packets */
      pcap_t                     *pcap_src;
      /* the pcap header as returned from pcap_next() */
      const struct pcap_pkthdr   *pcap_hdr;
      /* the packet as returned from pcap_next() */
      const u_char               *pcap_data;
  } sk_pktsrc_t;

This structure gives the user access to all the information about the packet.

The following return values from ptoflow() determines whether rwptoflow writes the flow and the packet:

 0 

Write the flow record to the flow-output and the packet to the PCKTS_PASS unless another plug-in instructs otherwise.

 1 

Write the flow record to the flow-output and the packet to the PCKTS_PASS immediately; do not call the ptoflow() routine on any other plug-in.

 2 

Treat the packet as a reject: Do not write the flow record; write the packet to the PCKTS_REJECT immediately; do not call the ptoflow() routine on any other plug-in.

 3 

Ignore the packet immediately: Do not write the flow record nor the packet; do not call the ptoflow() routine on any other plug-in.

If ptoflow() returns any other value, the rwptoflow application will terminate with an error.

teardown()

is called as the application exits. The user can use this routine to print results and to free() any data structures that were used.

rwptoflow uses the following rules to find the plug-in: When PLUG_IN_PATH contains a slash (/), rwptoflow assumes the path to PLUG_IN_PATH is correct. Otherwise, rwptoflow will attempt to find the file in $SILK_PATH/lib/silk, $SILK_PATH/share/lib, $SILK_PATH/lib, and in these directories parallel to the application's directory: lib/silk, share/lib, and lib. If rwptoflow does not find the file, it assumes the plug-in is in the current directory. To force rwptoflow to look in the current directory first, specify --dynamic-library=./PLUG_IN_PATH.

When the SILK_DYNLIB_DEBUG environment variable is non-empty, rwptoflow prints status messages to the standard error as it tries to open each of its plug-ins.

EXAMPLES

Given the pcap capture file data.pcap, convert it to a SiLK flow file:

  rwptoflow data.pcap --packet-pass=good.pcap --flow-out=data.rw

Filter the SiLK flows---passing those records whose source IPs are found in the IPset file sip.set:

  rwfilter --sipset=sip.set --pass=filtered.rw  data.rw

Match the original pcap file against the filtered SiLK file, in effect generating a pcap file which has been filtered by sip.set:

  rwpmatch --flow-file=filtered.rw good.pcap > filtered.pcap

ENVIRONMENT

SILK_DYNLIB_DEBUG

When set to 1, rwsort print status messages to the standard error as it tries to open each of its plug-ins.

SEE ALSO

tcpdump(1), pcap(3), rwpmatch(1), rwfilter(1), rwpdedupe(1), mergecap(1)

NOTES

SiLK supports millisecond timestamps. When reading packets whose timestamps have finer precision, the times are truncated at the millisecond position.

The mergecap(1) or rwpdedupe(1) programs can be used to join multiple tcpdump capture files in order to convert into a single flow file.