Monitoring for Large-Scale Networks

SiLK

rwstats

NAME

rwstats - Print top-N or bottom-N lists or summarize data by protocol

SYNOPSIS

rwstats --fields=KEY [--values=VALUES]
      {--count=N | --threshold=N | --percentage=N}
      [{--top | --bottom}] [--presorted-input] [--no-percents]
      [--ipv6-policy={ignore,asv4,mix,force,only}]
      [{--bin-time=SECONDS | --bin-time}]
      [--timestamp-format=FORMAT] [--epoch-time]
      [--ip-format=FORMAT] [--integer-ips] [--zero-pad-ips]
      [--integer-sensors] [--integer-tcp-flags]
      [--no-titles] [--no-columns] [--column-separator=CHAR]
      [--no-final-delimiter] [{--delimited | --delimited=CHAR}]
      [--print-filenames] [--copy-input=PATH] [--output-path=PATH]
      [--pager=PAGER_PROG] [--temp-directory=DIR_PATH]
      [{--legacy-timestamps | --legacy-timestamps={1,0}}]
      [--site-config-file=FILENAME]
      [--plugin=PLUGIN [--plugin=PLUGIN ...]]
      [--python-file=PATH [--python-file=PATH ...]]
      [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]]
      [--pmap-column-width=NUM]
      {[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]}

rwstats {--overall-stats | --detail-proto-stats=PROTO[,PROTO]}
      [--no-titles] [--no-columns] [--column-separator=CHAR]
      [--no-final-delimiter] [{--delimited | --delimited=CHAR}]
      [--print-filenames] [--copy-input=PATH] [--output-path=PATH]
      [--pager=PAGER_PROG]
      {[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]}

rwstats [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]]
      [--plugin=PLUGIN ...] [--python-file=PATH ...] --help

rwstats [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]]
      [--plugin=PLUGIN ...] [--python-file=PATH ...] --help-fields

rwstats --legacy-help

rwstats --version

DESCRIPTION

rwstats has two modes of operation: it can compute a Top-N or Bottom-N list, or it can summarize data for a list of protocols.

In either mode, rwstats reads SiLK Flow records from the files named on the command line or from the standard input when no file names are specified and --xargs is not present. To read the standard input in addition to the named files, use - or stdin as a file name. If an input file name ends in .gz, the file is uncompressed as it is read. When the --xargs switch is provided, rwstats reads the names of the files to process from the named text file or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line.

Top-N Description

rwstats, similar to rwuniq(1), reads SiLK Flow records and groups them by a key composed of user-specified attributes of the flows. For each group (or bin), a collection of aggregate values is computed; these values are typically related to the volume of the bin, such as the sum of the bytes fields for all records that match the key. The first aggregate value is called the primary aggregate value.

Where rwstats and rwuniq differ is that rwstats sorts by the primary aggregate value. Specifically, once all the SiLK Flow records are read, rwstats sorts the bins by the primary aggregate value in either decreasing order (for a top-N list) or increasing order (for a bottom-N list). The ordering of bins that have the same primary aggregate value is arbitrary. The bins are printed as text, and the number of bins to print may be specified as a fixed value (e.g., print 10 bins), as a threshold (print bins whose byte count is greater than 400), or as a percentage of the total volume across all bins (print bins that contain at least 10% of all the packets).

The user must provide the --fields switch to select the flow attribute(s) (or field(s)) that comprise the key for each bin. The available fields are similar to those supported by rwcut(1); see the description of the --fields switch in the "OPTIONS" section below for the details or run rwstats with the --help-fields switch. The list of fields may be extended by loading PySiLK files (see silkpython(3)) or plug-ins (silk-plugin(3)). The fields are printed in the order in which they occur in the --fields switch. The size of the key is limited to 256 octets. A larger key more quickly uses the available the memory and results in slower performance.

The aggregate value(s) to compute for each bin are also chosen by the user. As with the key fields, the user may extend the list of aggregate fields by using PySiLK or plug-ins. The preferred way to specify the aggregate fields is to use the --values switch; the aggregate fields are printed in the order they occur in the --values switch. If the user does not select any aggregate value(s), rwstats defaults to computing the number of flow records for each bin. As with the key fields, requesting more aggregate values slows performance.

In addition to computing the primary aggregate value for the flows in each bin, rwstats computes that aggregate value across all flow records. When printing the results, the output for each bin includes the ratio of the bin's aggregate value to the total aggregate value (displayed as a percentage). In addition, a cumulative percentage column is printed. When the primary aggregate value is a distinct count, the cumulative percentage may be greater than 100. The percentage columns contain a question mark when the primary aggregate value comes from a plug-in since rwstats does not know whether summing the aggregate values is reasonable. The display of the percentage columns may be suppressed by specifying --no-percents.

rwstats attempts to keep all key and aggregate value data in the computer's memory. If rwstats runs out of memory, the current key and aggregate value data is written to a temporary file. Once all input has been processed, the data from the temporary files is merged to produce the final output. By default, these temporary files are stored in the /tmp directory. Because these files can be large, it is strongly recommended that /tmp not be used as the temporary directory. To modify the temporary directory used by rwstats, provide the --temp-directory switch, set the SILK_TMPDIR environment variable, or set the TMPDIR environment variable.

rwstats may also run out of memory if the requested Top-N is too large.

The --presorted-input switch may allow rwstats to process data more efficiently by causing rwstats to assume the input has been previously sorted with the rwsort(1) command. With this switch, rwstats does not need large amounts of memory during the binning stage because it does not bin each flow; instead, it keeps a running summation for the bin. When the key changes, the bin's primary aggregate value is compared with those of the current Top-N (or Bottom-N) to see if the new bin is a closer to the top (or bottom). For the output to be meaningful, rwsort and rwstats must be invoked with the same --fields value. When multiple input files are specified and --presorted-input is given, rwstats merge-sorts the flow records from the input files. rwstats usually runs faster if you do not include the --presorted-input switch when counting distinct IP addresses, even when reading sorted input. Finally, you may get unusual results with --presorted-input when the --fields switch contains multiple time-related key fields (sTime, duration, eTime), or when the time-related key is not the final key listed in --fields; see the "NOTES" section for details.

Protocol Statistics Description

Alternatively, rwstats can provide statistics for each of bytes, packets, and bytes-per-packet giving minima, maxima, quartile, and interval flow-counts across all flows or across a list of protocols specified by the user.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

Top-N Invocation

To compute a Top-N or Bottom-N list, the key field(s) must be specified. Normally the --fields switch is used to specify the key field(s), but for backward compatibility older switches may be specified (see the "Legacy Switches" section below).

--fields=KEY

KEY contains the list of flow attributes (a.k.a. fields or columns) that make up the key into which flows are binned. The columns are displayed in the order the fields are specified. Each field may be specified once only. KEY is a comma separated list of field-names, field-integers, and ranges of field-integers; a range is specified by separating the start and end of the range with a hyphen (-). Field-names are case insensitive. Example:

--fields=stime,10,1-5

There is no default value for the --fields switch.

The complete list of built-in fields that the SiLK tool suite supports follows, though note that not all fields are present in all SiLK file formats; when a field is not present, its value is 0.

sIP,1: source IP address
dIP,2: destination IP address
sPort,3: source port for TCP and UDP, or equivalent
dPort,4: destination port for TCP and UDP, or equivalent. See note at iType.
protocol,5: IP protocol
packets,pkts,6: packet count
bytes,7: byte count
flags,8: bit-wise OR of TCP flags over all packets
sTime,9: starting time of the flow, in seconds resolution by default or millisecond resolution when --bin-time is given and includes fractional seconds. When all time-related fields (sTime, duration, eTime) are in use, rwstats ignores the final time field when binning the records.
duration,10: duration of the flow, in seconds resolution by default. If --bin-time is given, duration is set to the difference of the binned sTime and eTime fields at the same resolution. When all time-related fields (sTime, duration, eTime) are in use, rwstats ignores the final time field when binning the records.
eTime,11: end time of the flow in seconds resolution by default or millisecond resolution when --bin-time is given and includes fractional seconds. When all time-related fields (sTime, duration, eTime) are in use, rwstats ignores the final time field when binning the records.
sensor,12: name or ID of the sensor where the flow was collected
class,20: class assigned to the flow by rwflowpack(8). Binning by class and/or type equates to binning by the integer value used internally to represent the class/type pair. When --fields contains class but not type, rwstats's output may have multiple rows with the same value(s) for the key field(s).
type,21: type assigned to the flow by rwflowpack(8). See note on previous entry.
iType: the ICMP type value for ICMP or ICMPv6 flows and empty (numerically zero) for non-ICMP flows. Internally, SiLK stores the ICMP type and code in the dPort field. To avoid getting very odd results, either do not use the dPort field when your key includes ICMP field(s) or be certain to include the protocol field as part of your key. This field was added in SiLK 3.8.1.
iCode: the ICMP code value for ICMP or ICMPv6 flows and empty for non-ICMP flows. See note at iType.
icmpTypeCode,25: equivalent to iType,iCode when used in --fields. This field may not be mixed with iType or iCode, and this field is deprecated as of SiLK 3.8.1. As of SiLK 3.8.1, icmpTypeCode may no longer be used as the argument to the Distinct: value field; the dPort field provides an equivalent result as long as the input is limited to ICMP flow records.

Many SiLK file formats do not store the following fields and their values are always 0; they are listed here for completeness:

in,13: router SNMP input interface or vlanId if packing tools were configured to capture it (see sensor.conf(5))
out,14: router SNMP output interface or postVlanId
nhIP,15: router next hop IP

SiLK can store flows generated by enhanced collection software that provides more information than NetFlow v5. These flows may support some or all of these additional fields; for flows without this additional information, the field's value is always 0.

initialFlags,26

TCP flags on first packet in the flow

sessionFlags,27

bit-wise OR of TCP flags over all packets except the first in the flow

attributes,28

flow attributes set by the flow generator:

S: all the packets in this flow record are exactly the same size
F: flow generator saw additional packets in this flow following a packet with a FIN flag (excluding ACK packets)
T: flow generator prematurely created a record for a long-running connection due to a timeout. (When the flow generator yaf(1) is run with the --silk switch, it will prematurely create a flow and mark it with T if the byte count of the flow cannot be stored in a 32-bit value.)
C: flow generator created this flow as a continuation of long-running connection, where the previous flow for this connection met a timeout (or a byte threshold in the case of yaf).

Consider a long-running ssh session that exceeds the flow generator's active timeout. (This is the active timeout since the flow generator creates a flow for a connection that still has activity). The flow generator will create multiple flow records for this ssh session, each spanning some portion of the total session. The first flow record will be marked with a T indicating that it hit the timeout. The second through next-to-last records will be marked with TC indicating that this flow both timed out and is a continuation of a flow that timed out. The final flow will be marked with a C, indicating that it was created as a continuation of an active flow.

application,29

guess as to the content of the flow. Some software that generates flow records from packet data, such as yaf, will inspect the contents of the packets that make up a flow and use traffic signatures to label the content of the flow. SiLK calls this label the application; yaf refers to it as the appLabel. The application is the port number that is traditionally used for that type of traffic (see the /etc/services file on most UNIX systems). For example, traffic that the flow generator recognizes as FTP will have a value of 21, even if that traffic is being routed through the standard HTTP/web port (80).

The following fields provide a way to label the IPs or ports on a record. These fields require external files to provide the mapping from the IP or port to the label:

sType,16: for the source IP address, the value 0 if the address is non-routable, 1 if it is internal, or 2 if it is routable and external. Uses the mapping file specified by the SILK_ADDRESS_TYPES environment variable, or the address_types.pmap mapping file, as described in addrtype(3).
dType,17: as sType for the destination IP address
scc,18: for the source IP address, a two-letter country code abbreviation denoting the country where that IP address is located. Uses the mapping file specified by the SILK_COUNTRY_CODES environment variable, or the country_codes.pmap mapping file, as described in ccfilter(3). The abbreviations are those defined by ISO 3166-1 (see for example https://www.iso.org/iso-3166-country-codes.html or https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) or the following special codes: -- N/A (e.g. private and experimental reserved addresses); a1 anonymous proxy; a2 satellite provider; o1 other
dcc,19: as scc for the destination IP
src-map-name: label contained in the prefix map file associated with map-name. If the prefix map is for IP addresses, the label is that associated with the source IP address. If the prefix map is for protocol/port pairs, the label is that associated with the protocol and source port. See also the description of the --pmap-file switch below and the pmapfilter(3) manual page.
dst-map-name: as src-map-name for the destination IP address or the protocol and destination port.
sval: as src-map-name when no map-name is associated with the prefix map file
dval: as dst-map-name when no map-name is associated with the prefix map file

Finally, the list of built-in fields may be augmented by the run-time loading of PySiLK code or plug-ins written in C (also called shared object files or dynamic libraries), as described by the --python-file and --plugin switches.

--values=VALUES

When computing a Top-N or Bottom-N, all flows that have the same key field(s) are binned together. For each bin, one or more aggregate values are computed as specified by VALUES, a comma separated list of names. Names are case insensitive. The first entry in VALUES is the primary value, and it is used as the basis to compute the Top-N or Bottom-N. If the --values switch is not specified (and no legacy switch that sets values is specified), rwstats counts the number of flow records for each bin. The aggregate fields are printed in the order they occur in VALUES. The names of the built-in value fields follow. This list can be augmented through the use of PySiLK and plug-ins.

Records: Count the number of flow records that mapped to each bin.
Packets: Sum the number of packets across all records that mapped to each bin.
Bytes: Sum the number of bytes across all records that mapped to each bin.
Duration: Sum the durations of all records that mapped to each bin, in floating-point seconds with milliseconds resolution. Since SiLK 3.23.0.
sIP-Distinct: Count the number of distinct source IP addresses that were seen for each bin.
dIP-Distinct: Count the number of distinct destination IP addresses that were seen for each bin.
Distinct:KEY_FIELD: Count the number of distinct values for KEY_FIELD, where KEY_FIELD is any field that can be used as an argument to --fields except for icmpTypeCode. For example, Distinct:sPort counts the number of distinct source ports for each bin. When this aggregate value field is used, the specified KEY_FIELD may not be present in the argument to --fields (since the distinct count in that case is one).

--plugin=PLUGIN

Augment the list of key fields and/or aggregate value fields by using run-time loading of the plug-in (shared object) whose path is PLUGIN. The switch may be repeated to load multiple plug-ins. The creation of plug-ins is described in the silk-plugin(3) manual page. When PLUGIN does not contain a slash (/), rwstats attempts to find a file named PLUGIN in the directories listed in the "FILES" section. If rwstats finds the file, it uses that path. If PLUGIN contains a slash or if rwstats does not find the file, rwstats relies on your operating system's dlopen(3) call to find the file. When the SILK_PLUGIN_DEBUG environment variable is non-empty, rwstats prints status messages to the standard error as it attempts to find and open each of its plug-ins.

--pmap-file=PATH

--pmap-file=MAPNAME:PATH

Load the prefix map file located at PATH and create fields named src-map-name and dst-map-name where map-name is either the MAPNAME part of the argument or the map-name specified when the file was created (see rwpmapbuild(1)). If no map-name is available, rwstats names the fields sval and dval. Specify PATH as - or stdin to read from the standard input. The switch may be repeated to load multiple prefix map files, but each prefix map must use a unique map-name. The --pmap-file switch(es) must precede the --fields switch. See also pmapfilter(3).

--pmap-column-width=NUM

When printing a label associated with a prefix map, this switch gives the maximum number of characters to use when displaying the textual value of the field.

--python-file=PATH

When the SiLK Python plug-in is used, rwstats reads the Python code from the file PATH to define additional fields that can be used as part of the key or as an aggregate value. This file should call register_field() for each field it wishes to define. For details and examples, see the silkpython(3) and pysilk(3) manual pages.

To determine the value of N for a Top-N (or Bottom-N) list, one of the following switches must be specified. The primary value may limit which switch may be specified.

--count=COUNT: Print the COUNT bins with the largest (or smallest) primary values. When COUNT is 0, all bins are printed. If COUNT is 0 and rwstats runs out of memory while attempting to sort all bins, rwstats prints the Top-N or Bottom-N bins using the amount of memory it was able to allocate. rwstats did not accept a value of 0 for COUNT prior to SiLK 3.12.0.
--threshold=THESHOLD: Print the bins where the primary value is greater-than (or less-than) the value THESHOLD. Using this switch when the primary value comes from a plug-in causes rwstats to exit with an error. If rwstats runs out of memory while locating all bins that meet the threshold, rwstats prints the Top-N or Bottom-N bins using the amount of memory it was able to allocate. rwstats did not accept a value of 0 for THESHOLD prior to SiLK 3.12.0.
--percentage=PERCENT: Print the bins where the primary value is greater-than (or less-than) PERCENT percent of the sum of the primary values across all bins. PERCENT may be a floating point value between 0.0 and 100.0 inclusive. To use this switch, the --presorted-input switch must not be present and the primary value must be Bytes, Packets, Records, or a distinct count. If rwstats runs out of memory while locating all bins that meet the percentage, rwstats prints the Top-N or Bottom-N using the amount of memory it was able to allocate. The value of PERCENT was required to be an integer prior to SiLK 3.12.0. Support for computing the percentages of distinct counts was added in SiLK 3.16.0.

To determine whether to compute the Top-N or the Bottom-N, specify one of the following switches. If neither switch is given, --top is assumed:

--top: Sort the bins in order of decreasing primary aggregate value. This is the default behavior.
--bottom: Sort the bins in order of increasing primary aggregate value.

Protocol Statistics Invocation

The following switches compute and print, for each of bytes, packets, and bytes per packet, the minimum value, the maximum value, quartiles, and a count of the number of flows that fall into each of one of ten intervals statistics. These switches may not be combined with the switches that produce Top-N or Bottom-N lists.

--overall-stats: Print intervals and quartiles across all flows that were read by rwstats.
--detail-proto-stats=PROTO[,PROTO...]: Print intervals and quartiles for each individual protocol listed as an argument. The argument should be a comma separated list of protocols or ranges of protocols: 1-6,17. Specifying this option implies --overall-stats.

Miscellaneous Switches

The following switches are available when rwstats is running in either mode, though many only applicable to the Top-N mode.

--presorted-input

Cause rwstats to assume that it is reading sorted input; i.e., that rwstats's input file(s) were generated by rwsort(1) using the exact same value for the --fields switch. When no distinct counts are being computed, rwstats can process its input without needing to write temporary files. When multiple input files are specified, rwstats merge-sorts the flow records from the input files. When using --presorted-input and computing a Top-N or Bottom-N, the --percentage limit cannot be used. See the "NOTES" section for issues that may occur when using --presorted-input.

--no-percents

For the Top-N invocation, do not print the percent-of-total and cumulative-percentage columns. These columns contain a question mark when the primary aggregate value comes from a plug-in, and this switch allows you to suppress them.

--ipv6-policy=POLICY

Determine how IPv4 and IPv6 flows are handled when SiLK has been compiled with IPv6 support. When the switch is not provided, the SILK_IPV6_POLICY environment variable is checked for a policy. If it is also unset or contains an invalid policy, the POLICY is mix. When SiLK has not been compiled with IPv6 support, IPv6 flows are always ignored, regardless of the value passed to this switch or in the SILK_IPV6_POLICY variable. The supported values for POLICY are:

ignore: Ignore any flow record marked as IPv6, regardless of the IP addresses it contains.
asv4: Convert IPv6 flow records that contain addresses in the ::ffff:0:0/96 netblock (that is, IPv4-mapped IPv6 addresses) to IPv4 and ignore all other IPv6 flow records.
mix: Process the input as a mixture of IPv4 and IPv6 flow records. When an IP address is used as part of the key or value, this policy is equivalent to force.
force: Convert IPv4 flow records to IPv6, mapping the IPv4 addresses into the ::ffff:0:0/96 netblock.
only: Process only flow records that are marked as IPv6 and ignore IPv4 flow records in the input.

--bin-time=SECONDS

--bin-time

Adjust the times in the key fields sTime and eTime to appear on SECONDS-second boundaries (the floor of the time is used), and adjust duration to be the difference of those values. As of SiLK 3.17.0, SECONDS may be a floating-point value of 0.001 or greater. When the fractional part of SECONDS is non-zero, rwstats uses millisecond resolution for the time-related key fields. If this switch is not used, times appear on 1-second boundaries with fractional seconds truncated. When the switch is used but no argument is given, rwstats uses 60-second time bins.

--timestamp-format=FORMAT

Specify the format and/or timezone to use when printing timestamps. When this switch is not specified, the SILK_TIMESTAMP_FORMAT environment variable is checked for a default format and/or timezone. If it is empty or contains invalid values, timestamps are printed in the default format, and the timezone is UTC unless SiLK was compiled with local timezone support. FORMAT is a comma-separated list of a format and/or a timezone. The format is one of:

default: Print the timestamps as YYYY/MM/DDThh:mm:ss.
iso: Print the timestamps as YYYY-MM-DD hh:mm:ss.
m/d/y: Print the timestamps as MM/DD/YYYY hh:mm:ss.
epoch: Print the timestamps as the number of seconds since 00:00:00 UTC on 1970-01-01.

When a timezone is specified, it is used regardless of the default timezone support compiled into SiLK. The timezone is one of:

utc: Use Coordinated Universal Time to print timestamps.
local: Use the TZ environment variable or the local timezone.

--epoch-time

Print timestamps as epoch time (number of seconds since midnight GMT on 1970-01-01). This switch is equivalent to --timestamp-format=epoch, it is deprecated as of SiLK 3.0.0, and it will be removed in the SiLK 4.0 release.

--ip-format=FORMAT

Specify how IP addresses are printed, where FORMAT is a comma-separated list of the arguments described below. When this switch is not specified, the SILK_IP_FORMAT environment variable is checked for a value and that format is used if it is valid. The default FORMAT is canonical. Since SiLK 3.7.0.

canonical: Print IP addresses in the canonical format. If the key only contains IPv4 addresses, use dot-separated decimal (192.0.2.1). Otherwise, use colon-separated hexadecimal (2001:db8::1) or a mixed IPv4-IPv6 representation for IPv4-mapped IPv6 addresses (the ::ffff:0:0/96 netblock, e.g., ::ffff:192.0.2.1) and IPv4-compatible IPv6 addresses (the ::/96 netblock other than ::/127, e.g., ::192.0.2.1).
no-mixed: Print IP addresses in the canonical format (192.0.2.1 or 2001:db8::1) but do not used the mixed IPv4-IPv6 representations. For example, use ::ffff:c000:201 instead of ::ffff:192.0.2.1. Since SiLK 3.17.0.
decimal: Print IP addresses as integers in decimal format. For example, print 192.0.2.1 and 2001:db8::1 as 3221225985 and 42540766411282592856903984951653826561, respectively.
hexadecimal: Print IP addresses as integers in hexadecimal format. For example, print 192.0.2.1 and 2001:db8::1 as c00000201 and 20010db8000000000000000000000001, respectively.
zero-padded: Make all IP address strings contain the same number of characters by padding numbers with leading zeros. For example, print 192.0.2.1 and 2001:db8::1 as 192.000.002.001 and 2001:0db8:0000:0000:0000:0000:0000:0001, respectively. For IPv6 addresses, this setting implies no-mixed, so that ::ffff:192.0.2.1 is printed as 0000:0000:0000:0000:0000:ffff:c000:0201. As of SiLK 3.17.0, may be combined with any of the above, including decimal and hexadecimal.

The following arguments modify certain IP addresses prior to printing. These arguments may be combined with the above formats.

map-v4: Change IPv4 addresses to IPv4-mapped IPv6 addresses (addresses in the ::ffff:0:0/96 netblock) prior to formatting. Since SiLK 3.17.0.
unmap-v6: When the key contains IPv6 addresses, change any IPv4-mapped IPv6 addresses (addresses in the ::ffff:0:0/96 netblock) to IPv4 addresses prior to formatting. Since SiLK 3.17.0.

The following argument is also available:

force-ipv6: Set FORMAT to map-v4,no-mixed.

--integer-ips

Print IP addresses as integers. This switch is equivalent to --ip-format=decimal, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--zero-pad-ips

Print IP addresses as fully-expanded, zero-padded values in their canonical form. This switch is equivalent to --ip-format=zero-padded, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--integer-sensors

Print the integer ID of the sensor rather than its name.

--integer-tcp-flags

Print the TCP flag fields (flags, initialFlags, sessionFlags) as an integer value. Typically, the characters F,S,R,P,A,U,E,C are used to represent the TCP flags.

--no-titles

Disable section and column titles. By default, titles are printed.

--no-columns

Disable fixed-width columnar output.

--column-separator=C

Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.

--no-final-delimiter

Do not print the column separator after the final column. Normally a delimiter is printed.

--delimited

--delimited=C

Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

--print-filenames

Print to the standard error the names of input files as they are opened.

--copy-input=PATH

Copy all binary SiLK Flow records read as input to the specified file or named pipe. PATH may be stdout or - to write flows to the standard output as long as the --output-path switch is specified to redirect rwstats' textual output to a different location.

--output-path=PATH

Write the textual output to PATH, where PATH is a filename, a named pipe, the keyword stderr to write the output to the standard error, or the keyword stdout or - to write the output to the standard output (and bypass the paging program). If PATH names an existing file, rwstats exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this switch is not given, the output is either sent to the pager or written to the standard output.

--pager=PAGER_PROG

When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the --output-path switch is given or if the value of the pager is determined to be the empty string, no paging is performed and all output is written to the terminal.

--temp-directory=DIR_PATH

Specify the name of the directory in which to store data files temporarily when the memory is not large enough to store all the bins and their aggregate values. This switch overrides the directory specified in the SILK_TMPDIR environment variable, which overrides the directory specified in the TMPDIR variable, which overrides the default, /tmp.

--site-config-file=FILENAME

Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwstats searches for the site configuration file in the locations specified in the "FILES" section.

--legacy-timestamps

--legacy-timestamps=NUM

When NUM is not specified or is 1, this switch is equivalent to --timestamp-format=m/d/y. Otherwise, the switch has no effect. This switch is deprecated as of SiLK 3.0.0, and it will be removed in the SiLK 4.0 release.

--xargs

--xargs=FILENAME

Read the names of the input files from FILENAME or from the standard input if FILENAME is not provided. The input is expected to have one filename per line. rwstats opens each named file in turn and reads records from it as if the filenames had been listed on the command line.

--help

Print the available options and exit. Specifying switches that add new fields, values, or additional switches before --help allows the output to include descriptions of those fields or switches.

--help-fields

Print the description and alias(es) of each field and value and exit. Specifying switches that add new fields before --help-fields allows the output to include descriptions of those fields.

--legacy-help

Print help, including legacy switches. See the "Legacy Switches" section below for these switches.

--version

Print the version number and information about how SiLK was configured, then exit the application.

Legacy Switches

Use of the following switches has been discouraged since SiLK 2.0.0. As of SiLK 3.8.1, the switches are deprecated and they will be removed in SiLK 4.0. For each switch, use the replacement indicated.

--sip: Use: --fields=sip
--sip=CIDR: Use the most significant CIDR bits of the source address as the key. Using this switch with IPv6 data causes an error. The user should use rwnetmask(1) to mask the data prior to processing it with rwstats.
--dip: Use: --fields=dip
--dip=CIDR: Use the most significant CIDR bits of the destination address as the key. Using this switch with IPv6 data causes an error. The user should use rwnetmask to mask the data prior to processing it with rwstats.
--sport: Use: --fields=sport
--dport: Use: --fields=dport
--protocol: Use: --fields=protocol
--icmp: Use: --fields=iType,iCode
--flows: Use: --values=records
--packets: Use: --values=packets
--bytes: Use: --values=bytes

EXAMPLES

In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash (\) is used to indicate a wrapped line.

Top-N Examples

Print the top talkers (based on number of flow records, limit to the top four):

$ rwstats --fields=sip --count=4 data.rw
INPUT: 549092 Records for 12990 Bins and 549092 Total Records
OUTPUT: Top 4 Bins by Records
        sIP|     Records|  %Records|   cumul_%|
   10.1.1.1|       36604|  6.666278|  6.666278|
   10.1.1.2|       13897|  2.530906|  9.197184|
   10.1.1.3|       12739|  2.320012| 11.517196|
   10.1.1.4|       11807|  2.150277| 13.667473|

Print the seven hosts that received the most packets:

$ rwstats --fields=dip --values=packets --count=7 data.rw
INPUT: 549092 Records for 44654 Bins and 6620587 Total Packets
OUTPUT: Top 7 Bins by Packets
        dIP|     Packets|  %Packets|   cumul_%|
   10.1.1.1|      217574|  3.286325|  3.286325|
   10.1.1.2|      138177|  2.087081|  5.373407|
   10.1.1.3|      121892|  1.841106|  7.214512|
   10.1.1.4|       97073|  1.466230|  8.680742|
   10.1.1.5|       82284|  1.242851|  9.923593|
   10.1.1.6|       80051|  1.209123| 11.132715|
   10.1.1.7|       73602|  1.111714| 12.244430|

Print the IP pairs that shared 100,000,000 bytes or more:

$ rwstats --fields=sip,dip --values=byte --threshold=100000000 data.rw
INPUT: 549092 Records for 107136 Bins and 3410300252 Total Bytes
OUTPUT: Top 5 Bins by Bytes (threshold 100000000)
        sIP|            dIP|       Bytes|    %Bytes|   cumul_%|
   10.1.1.1|       10.1.1.2|   307478707|  9.016177|  9.016177|
   10.1.1.3|       10.1.1.4|   172164463|  5.048367| 14.064544|
   10.1.1.5|       10.1.1.6|   142059589|  4.165604| 18.230147|
   10.1.1.7|       10.1.1.8|   119388394|  3.500818| 21.730965|
   10.1.1.9|      10.1.1.10|   108268824|  3.174759| 24.905725|

Print the ports that were the source of at least 5% of all records:

$ rwstats --fields=sport --percentage=5 data.rw
INPUT: 549092 Records for 56799 Bins and 549092 Total Records
OUTPUT: Top 3 Bins by Records (5% == 27454)
  sPort|     Records|  %Records|   cumul_%|
     80|       86677| 15.785515| 15.785515|
     53|       64681| 11.779629| 27.565144|
      0|       47760|  8.697996| 36.263140|

Print the destination ports that saw the least number of records (limit to the bottom eight):

$ rwstats --fields=dport --bottom --count=8 data.rw
INPUT: 549092 Records for 44772 Bins and 549092 Total Records
OUTPUT: Bottom 8 Bins by Records
  dPort|     Records|  %Records|   cumul_%|
  19417|           1|  0.000182|  0.000182|
  12110|           1|  0.000182|  0.000364|
  34777|           1|  0.000182|  0.000546|
   8999|           1|  0.000182|  0.000728|
  36404|           1|  0.000182|  0.000911|
  16682|           1|  0.000182|  0.001093|
  27420|           1|  0.000182|  0.001275|
  14162|           1|  0.000182|  0.001457|

Print the source-destination port pairs that shared more than 500,000 packets (there were none):

$ rwstats --fields=sport,dport --values=packets \
       --top --threshold=500000 data.rw
INPUT: 366309 Records for 130307 Bins and 5597540 Total Packets
OUTPUT: No bins above threshold of 500000

Print the source-destination port pairs that shared more than 50,000 packets:

$ rwstats --fields=sport,dport --values=packets \
       --top --threshold=50000 data.rw
INPUT: 366309 Records for 130307 Bins and 5597540 Total Packets
OUTPUT: Top 3 Bins by Packets (threshold 50000)
  sPort|dPort|      Packets|  %Packets|   cumul_%|
   6699| 3607|       138177|  2.468531|  2.468531|
     80| 1179|        59774|  1.067862|  3.536393|
     80| 9659|        50319|  0.898949|  4.435342|

Print the protocols from least to most active (based on number of records):

$ rwstats --fields=protocol --bottom --count=10 data.rw
INPUT: 545262 Records for 3 Bins and 545262 Total Records
OUTPUT: Bottom 10 Bins by Records
   protocol|     Records|  %Records|   cumul_%|
          1|       46319|  8.494815|  8.494815|
         17|      132634| 24.324820| 32.819635|
          6|      366309| 67.180365|100.000000|

Print the packet and byte counts for the pair of /16s that shared the most packets (use rwnetmask(1) on the input to rwstats; limit result to top ten):

$ rwnetmask --4sip-prefix=16 --4dip-prefix=16 data.rw   \
  | rwstats --fields=sip,dip --values=packets,bytes     \
       --count=10 --no-percent
INPUT: 250928 Records for 230 Bins and 72279154 Total Packets
OUTPUT: Top 10 Bins by Packets
           sIP|            dIP|  Packets|        Bytes|
    10.255.0.0|    192.168.0.0|  2711524|   2207297227|
    10.253.0.0|    192.168.0.0|  2690120|   2288595669|
    10.254.0.0|    192.168.0.0|  2593074|   2141263178|
    10.252.0.0|    192.168.0.0|  2553388|   2117294828|
    10.250.0.0|    192.168.0.0|  2312661|   1982654956|
    10.251.0.0|    192.168.0.0|  2218194|   1785263601|
    10.249.0.0|    192.168.0.0|  2196041|   1934938137|
    10.248.0.0|    192.168.0.0|  2160037|   1804446929|
    10.247.0.0|    192.168.0.0|  2000379|   1579214987|
    10.246.0.0|    192.168.0.0|  1878143|   1578321728|

Print the number of distinct destination hosts seen for every destination port, limiting the result to the ports that saw at least 3% of the hosts. The percentage for each bin is relative to the number of distinct destination IP addresses seen in the input.

$ rwstats --fields=dport --values=distinct:dip --percent=3 data.rw
INPUT: 243127 Records for 4738 Bins and 122064 Total dIP-Distinct
OUTPUT: Top 5 bins by dIP-Distinct (3.0000% == 3661)
dPort|dIP-Distin|%dIP-Disti|   cumul_%|
   80|     26940| 22.070389|   22.0704|
   25|     15538| 12.729388|   34.7998|
  443|      7907|  6.477749|   41.2775|
   22|      7733|  6.335201|   47.6127|
 8080|      3942|  3.229453|   50.8422|

Print the number of distinct destination ports seen for each protocol. When the primary aggregate value is counting the number of distinct values, the cumulative percentage may be larger than 100%.

$ rwstats --fields=proto --values=distinct:dport --count=0 data.rw
INPUT: 243127 Records for 2 Bins and 5335 Total dPort-Distinct
OUTPUT: Top 2 Bins by dPort-Distinct
pro|dPort-Dist|%dPort-Dis|   cumul_%|
  6|      4672| 87.572634|   87.5726|
 17|      4669| 87.516401|  175.0890|

The following example uses PySiLK to create an aggregate value field that computes the average byte count for each bin. The code for this field is shown in the silkpython(3) manual page. Note that the percentage columns are empty.

$ rwstats --python-file=avg-bytes.py --fields=sport      \
       --values=avg-bytes,bytes,flow --count=6 data.rw
INPUT: 243127 Records for 4738 Bins
OUTPUT: Top 6 Bins by avg-bytes
sPort|    avg-bytes|         Bytes|   Records|%avg-bytes|   cumul_%|
   22|   1010658.57|   28292376134|     27994|         ?|         ?|
 8080|    739703.65|    2918870591|      3946|         ?|         ?|
   80|    732930.03|   19821359790|     27044|         ?|         ?|
  443|    731919.66|    5794607921|      7917|         ?|         ?|
25605|     86376.00|         86376|         1|         ?|         ?|
25349|     83556.00|        167112|         2|         ?|         ?|

The --threshold switch is not supported when the primary aggregate value is from PySiLK or a plug-in.

$ rwstats --python-file=avg-bytes.py --fields=sport             \
       --values=avg-bytes,bytes,flow --threshold=90000 data.rw
rwstats: Only the --count limit is supported when the primary \
       values field is from a plug-in
rwstats: Cannot add value field 'avg-bytes' from plugin

When using rwstats on input that contains both incoming and outgoing flow records, consider using the int-ext-fields(3) plug-in which defines four additional fields representing the external IP address, the external port, the internal IP address, and the internal port. The plug-in requires the user to specify which class/type pairs are incoming and which are outgoing. See its manual page for additional information. As an example, here we run rwstats on a file containing incoming and outgoing web traffic.

$ rwstats --fields=sip,sport,dip,dport --values=bytes \
       --count=6 data.rw
INPUT: 155140 Records for 155140 Bins and 59036553615 Total Bytes
OUTPUT: Top 6 Bins by Bytes
            sIP|sPort|            dIP|dPort|   Bytes|    %Bytes|   cumul_%|
  10.242.96.200|   80|192.168.234.203|29868| 2681287|  0.004542|  0.004542|
192.168.211.200|   80|  10.253.27.160|25453| 2675740|  0.004532|  0.009074|
192.168.233.168|   80|  10.247.60.163|29777| 2672196|  0.004526|  0.013600|
192.168.229.229|  443|  10.250.19.210|27512| 2666647|  0.004517|  0.018117|
 192.168.255.24| 8080|  10.240.75.236|29826| 2659828|  0.004505|  0.022623|
192.168.241.247|   80|  10.216.173.77|26654| 2658141|  0.004503|  0.027125|

Here the int-ext-fields plug-in is used:

$ export INCOMING_FLOWTYPES=all/in,all/inweb
$ export OUTGOING_FLOWTYPES=all/out,all/outweb
$ rwstats --plugin=int-ext-fields.so                          \
       --fields=ext-ip,ext-port,int-ip,int-port --value=bytes \
       --count=6 data.rw
INPUT: 155140 Records for 77570 Bins and 59036553615 Total Bytes
OUTPUT: Top 6 Bins by Bytes
         ext-ip|ext-p|         int-ip|int-p|   Bytes|    %Bytes|   cumul_%|
  10.253.27.160|25453|192.168.211.200|   80| 2736332|  0.004635|  0.004635|
  10.242.96.200|   80|192.168.234.203|29868| 2722619|  0.004612|  0.009247|
  10.247.60.163|29777|192.168.233.168|   80| 2716749|  0.004602|  0.013849|
  10.250.19.210|27512|192.168.229.229|  443| 2714974|  0.004599|  0.018447|
  10.254.241.55|24206| 192.168.207.45|   80| 2713597|  0.004596|  0.023044|
 10.226.206.118|29557|192.168.247.227| 8080| 2707265|  0.004586|  0.027630|

Protocol Statistics Example

Print the interval breakdowns for flow records, packets, and bytes across all protocols, and for protocols 6 (TCP) and 17 (UDP):

$ rwstats --detail-proto-stats=6,17 data.rw
FLOW STATISTICS--ALL PROTOCOLS:  549092 records
*BYTES min 28; max 88906238
  quartiles LQ 122.06478 Med 420.30930 UQ 876.21920 UQ-LQ 754.15442
   interval_max|count<=max|%_of_input|   cumul_%|
             40|     35107|  6.393646|  6.393646|
             60|     35008|  6.375616| 12.769263|
            100|     49500|  9.014883| 21.784145|
            150|     40014|  7.287303| 29.071449|
            256|     65444| 11.918586| 40.990034|
           1000|    224016| 40.797535| 81.787569|
          10000|     75708| 13.787853| 95.575423|
         100000|     21981|  4.003154| 99.578577|
        1000000|      1901|  0.346208| 99.924785|
     4294967295|       413|  0.075215|100.000000|
*PACKETS min 1; max 70023
  quartiles LQ 1.76962 Med 3.68119 UQ 7.61567 UQ-LQ 5.84605
   interval_max|count<=max|%_of_input|   cumul_%|
              3|    232716| 42.381969| 42.381969|
              4|     61407| 11.183372| 53.565341|
             10|    195310| 35.569631| 89.134972|
             20|     33310|  6.066379| 95.201351|
             50|     17686|  3.220954| 98.422304|
            100|      4854|  0.884005| 99.306309|
            500|      2760|  0.502648| 99.808957|
           1000|       373|  0.067930| 99.876888|
          10000|       637|  0.116010| 99.992897|
     4294967295|        39|  0.007103|100.000000|
*BYTES/PACKET min 28; max 1500
  quartiles LQ 57.98319 Med 90.71150 UQ 164.77250 UQ-LQ 106.78932
   interval_max|count<=max|%_of_input|   cumul_%|
             40|     42568|  7.752435|  7.752435|
             44|     15173|  2.763289| 10.515724|
             60|     91003| 16.573361| 27.089085|
            100|    163850| 29.840173| 56.929258|
            200|    153190| 27.898786| 84.828043|
            400|     39761|  7.241227| 92.069271|
            600|     12810|  2.332942| 94.402213|
            800|      7954|  1.448573| 95.850786|
           1500|     22783|  4.149214|100.000000|
     4294967295|         0|  0.000000|100.000000|

FLOW STATISTICS--PROTOCOL 6:  366309/549092 records
*BYTES min 40; max 88906238
  quartiles LQ 310.47331 Med 656.53661 UQ 1089.75344 UQ-LQ 779.28013
   interval_max|count<=max|%_of_proto|   cumul_%|
             40|     29774|  8.128110|  8.128110|
             60|     11453|  3.126595| 11.254706|
            100|      6915|  1.887751| 13.142456|
            150|     16369|  4.468632| 17.611088|
            256|     12651|  3.453642| 21.064730|
           1000|    196881| 53.747246| 74.811976|
          10000|     68989| 18.833553| 93.645529|
         100000|     21099|  5.759891| 99.405420|
        1000000|      1784|  0.487021| 99.892441|
     4294967295|       394|  0.107559|100.000000|
*PACKETS min 1; max 70023
  quartiles LQ 3.39682 Med 5.85903 UQ 8.80427 UQ-LQ 5.40745
   interval_max|count<=max|%_of_proto|   cumul_%|
              3|     69358| 18.934288| 18.934288|
              4|     55993| 15.285729| 34.220016|
             10|    186559| 50.929407| 85.149423|
             20|     30947|  8.448332| 93.597755|
             50|     16186|  4.418674| 98.016429|
            100|      4204|  1.147665| 99.164094|
            500|      2178|  0.594580| 99.758674|
           1000|       315|  0.085993| 99.844667|
          10000|       537|  0.146598| 99.991264|
     4294967295|        32|  0.008736|100.000000|
*BYTES/PACKET min 40; max 1500
  quartiles LQ 60.19817 Med 96.78616 UQ 175.08044 UQ-LQ 114.88228
   interval_max|count<=max|%_of_proto|   cumul_%|
             40|     36559|  9.980372|  9.980372|
             44|     14929|  4.075521| 14.055893|
             60|     39593| 10.808634| 24.864527|
            100|    100117| 27.331297| 52.195824|
            200|    111258| 30.372718| 82.568542|
            400|     26020|  7.103293| 89.671834|
            600|      8600|  2.347745| 92.019579|
            800|      7726|  2.109148| 94.128727|
           1500|     21507|  5.871273|100.000000|
     4294967295|         0|  0.000000|100.000000|

FLOW STATISTICS--PROTOCOL 17:  132634/549092 records
*BYTES min 32; max 2115559
  quartiles LQ 66.53665 Med 150.61551 UQ 242.44095 UQ-LQ 175.90430
   interval_max|count<=max|%_of_proto|   cumul_%|
             20|         0|  0.000000|  0.000000|
             40|      5195|  3.916794|  3.916794|
             80|     42150| 31.779182| 35.695975|
            130|     11528|  8.691587| 44.387563|
            256|     45497| 34.302667| 78.690230|
           1000|     23401| 17.643289| 96.333519|
          10000|      4447|  3.352836| 99.686355|
         100000|       389|  0.293288| 99.979643|
        1000000|        23|  0.017341| 99.996984|
     4294967295|         4|  0.003016|100.000000|
*PACKETS min 1; max 8839
  quartiles LQ 0.84383 Med 1.68768 UQ 2.53149 UQ-LQ 1.68766
   interval_max|count<=max|%_of_proto|   cumul_%|
              3|    117884| 88.879171| 88.879171|
              4|      4452|  3.356605| 92.235777|
             10|      6678|  5.034908| 97.270685|
             20|      1766|  1.331484| 98.602168|
             50|      1055|  0.795422| 99.397590|
            100|       368|  0.277455| 99.675046|
            500|       353|  0.266146| 99.941192|
           1000|        33|  0.024880| 99.966072|
          10000|        45|  0.033928|100.000000|
     4294967295|         0|  0.000000|100.000000|
*BYTES/PACKET min 32; max 1415
  quartiles LQ 63.23827 Med 91.27180 UQ 158.10219 UQ-LQ 94.86392
   interval_max|count<=max|%_of_proto|   cumul_%|
             20|         0|  0.000000|  0.000000|
             24|         0|  0.000000|  0.000000|
             40|      5671|  4.275676|  4.275676|
            100|     70970| 53.508150| 57.783826|
            200|     39298| 29.628904| 87.412730|
            400|     12175|  9.179396| 96.592126|
            600|      4130|  3.113832| 99.705958|
            800|       160|  0.120633| 99.826590|
           1500|       230|  0.173410|100.000000|
     4294967295|         0|  0.000000|100.000000|

The silkpython(3) manual page provides examples that use PySiLK to create arbitrary fields to use as part of the key for rwstats.

ENVIRONMENT

SILK_IPV6_POLICY: This environment variable is used as the value for --ipv6-policy when that switch is not provided.
SILK_IP_FORMAT: This environment variable is used as the value for --ip-format when that switch is not provided. Since SiLK 3.11.0.
SILK_TIMESTAMP_FORMAT: This environment variable is used as the value for --timestamp-format when that switch is not provided. Since SiLK 3.11.0.
SILK_PAGER: When set to a non-empty string, rwstats automatically invokes this program to display its output a screen at a time. If set to an empty string, rwstats does not automatically page its output.
PAGER: When set and SILK_PAGER is not set, rwstats automatically invokes this program to display its output a screen at a time.
SILK_TMPDIR: When set and --temp-directory is not specified, rwstats writes the temporary files it creates to this directory. SILK_TMPDIR overrides the value of TMPDIR.
TMPDIR: When set and SILK_TMPDIR is not set, rwstats writes the temporary files it creates to this directory.
PYTHONPATH: This environment variable is used by Python to locate modules. When --python-file is specified, rwstats must load the Python files that comprise the PySiLK package, such as silk/__init__.py. If this silk/ directory is located outside Python's normal search path (for example, in the SiLK installation tree), it may be necessary to set or modify the PYTHONPATH environment variable to include the parent directory of silk/ so that Python can find the PySiLK module.
SILK_PYTHON_TRACEBACK: When set, Python plug-ins output traceback information on Python errors to the standard error.
SILK_COUNTRY_CODES: This environment variable allows the user to specify the country code mapping file that rwstats uses when computing the scc and dcc fields. The value may be a complete path or a file relative to the SILK_PATH. See the "FILES" section for standard locations of this file.
SILK_ADDRESS_TYPES: This environment variable allows the user to specify the address type mapping file that rwstats uses when computing the sType and dType fields. The value may be a complete path or a file relative to the SILK_PATH. See the "FILES" section for standard locations of this file.
SILK_CLOBBER: The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.
SILK_CONFIG_FILE: This environment variable is used as the value for the --site-config-file when that switch is not provided.
SILK_DATA_ROOTDIR: This environment variable specifies the root directory of data repository. As described in the "FILES" section, rwstats may use this environment variable when searching for the SiLK site configuration file.
SILK_PATH: This environment variable gives the root of the install tree. When searching for configuration files and plug-ins, rwstats may use this environment variable. See the "FILES" section for details.
TZ: When the argument to the --timestamp-format switch includes local or when a SiLK installation is built to use the local timezone, the value of the TZ environment variable determines the timezone in which rwstats displays timestamps. (If both of those are false, the TZ environment variable is ignored.) If the TZ environment variable is not set, the machine's default timezone is used. Setting TZ to the empty string or 0 causes timestamps to be displayed in UTC. For system information on the TZ variable, see tzset(3) or environ(7). (To determine if SiLK was built with support for the local timezone, check the Timezone support value in the output of rwstats --version.)
SILK_PLUGIN_DEBUG: When set to 1, rwstats prints status messages to the standard error as it attempts to find and open each of its plug-ins. In addition, when an attempt to register a field fails, rwstats prints a message specifying the additional function(s) that must be defined to register the field in rwstats. Be aware that the output can be rather verbose.
SILK_TEMPFILE_DEBUG: When set to 1, rwstats prints debugging messages to the standard error as it creates, re-opens, and removes temporary files.
SILK_UNIQUE_DEBUG: When set to 1, the binning engine used by rwstats prints debugging messages to the standard error.

FILES

${SILK_ADDRESS_TYPES}
${SILK_PATH}/share/silk/address_types.pmap
${SILK_PATH}/share/address_types.pmap
/usr/share/silk/address_types.pmap
/usr/share/address_types.pmap: Possible locations for the address types mapping file required by the sType and dType fields.
${SILK_CONFIG_FILE}
${SILK_DATA_ROOTDIR}/silk.conf
/data/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/share/silk/silk.conf
/usr/share/silk.conf: Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided.
${SILK_COUNTRY_CODES}
${SILK_PATH}/share/silk/country_codes.pmap
${SILK_PATH}/share/country_codes.pmap
/usr/share/silk/country_codes.pmap
/usr/share/country_codes.pmap: Possible locations for the country code mapping file required by the scc and dcc fields.
${SILK_PATH}/lib64/silk/
${SILK_PATH}/lib64/
${SILK_PATH}/lib/silk/
${SILK_PATH}/lib/
/usr/lib64/silk/
/usr/lib64/
/usr/lib/silk/
/usr/lib/: Directories that rwstats checks when attempting to load a plug-in.
${SILK_TMPDIR}/
${TMPDIR}/
/tmp/: Directory in which to create temporary files.

NOTES

rwstats functionally replaces the combination the following, where N is one more than the number of fields passed to rwuniq(1):

rwuniq --fields=... | sort -r -t '|' -k N | head -10

When the --bin-time switch is given and the three time-related key fields (starting-time (sTime), ending-time (eTime), and duration (duration)) are present, the duration field's value is modified to be the difference between the ending and starting times.

The time-related key fields use seconds resolution unless --bin-time is given and includes non-zero fractional seconds. The --bin-time switch began accepting a floating-point value in SiLK 3.17.0.

When the three time-related key fields (sTime, duration, eTime) are all in use, rwstats ignores the final time field when binning the records, but the field does appear in the output. Due to truncation of fractional values, rwstats may generate different numbers of bins depending on the order in which those three values appear in the --fields switch.

When computing distinct counts over a field, the field may not be part of the key; that is, you may not have --fields=sip --values=sip-distinct. The distinct count in that case is always 1.

Using the --presorted-input switch sometimes introduces more issues than it solves, and --presorted-input is less necessary now that rwstats can use temporary files while processing input.

When computing distinct IP counts, rwstats typically runs faster if you do not use the --presorted-input switch, even if the data was previously sorted.

When using the --presorted-input switch, it is highly recommended that you use no more than one time-related key field (sTime, duration, eTime) in the --fields switch and that the time-related key appear last in --fields. The issue is caused by rwsort considering the complete timestamp when sorting while rwstats typically uses less precision.

rwstats's strength is its ability to build arbitrary keys and aggregate fields. For maps of a single key to a single value, see also rwbag(1). To create a binary file that contains multiple keys and values, use rwaggbag(1).