NAME
rwstats - Print interval counts or top-N or bottom-N lists
SYNOPSIS
rwstats [--overall-stats] [--detail-proto-stats=PROTO[,PROTO]]
[{--sip | --sip=CIDR}] [{--dip | --dip=CIDR}]
[--sport] [--dport] [--protocol] [--icmp]
[--flows] [--packets] [--bytes]
[--count=N] [--threshold=N] [--percentage=N]
[--top] [--bottom]
[--integer-ips] [--no-titles] [--no-columns]
[--column-separator=CHAR] [--no-final-delimiter]
[{--delimited | --delimited=CHAR}] [--print-filenames]
[--copy-input=PATH] [--output-path=PATH] [--pager=PAGER_PROG]
[--site-config-file=FILENAME] [--legacy-help] [FILES...]
DESCRIPTION
rwstats reads SiLK Flow records from files named on the command line or from the standard input, bins those records by the user-specified specified key, computes the volume (record count or sum of packets or bytes) per bin, sorts the bins by volume, and prints the bins with the greatest or least volume. Alternatively, rwstats can provide statistics for each of bytes, packets, and bytes-per-packet giving minima, maxima, quartile, and interval flow-counts across all flows or across a list of protocols specified by the user.
The number of bins printed can be specified as a fixed value (e.g., print 10 bins), as a threshold (print bins whose byte count is less than 400), or as a percentage of the total volume across all bins (print bins who that contain at least 10% of all the packets).
RWSTATS WILL POTENTIALLY RUN OUT OF MEMORY. See the BUGS section for details.
OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.
The following switches will compute and print, for each of bytes, packets, and bytes per packet, the minimum value, the maximum value, quartiles, and a count of the number of flows that fall into each of one of ten intervals statistics. These switches cannot be combined with the switches that produce Top-N or Bottom-N lists.
- --overall-stats
- Print intervals and quartiles across all flows that were read by rwstats.
- --detail-proto-stats=PROTO[,PROTO...]
-
Print intervals and quartiles for each individual protocol listed as
an argument. The argument should be a comma separated list of
protocols or ranges of protocols:
1-6,17. Specifying this option implies --overall-stats.
To compute a Top-N or Bottom-N list, one or two key fields must be specified. Currently, rwstats only supports pairing --sip with --dip or pairing --sport with --dport.
- --sip
- --sip=CIDR
- Use the source address as the key. When CIDR is given, only the most significant CIDR bits of the source address are considered; i.e., the least significant 32-CIDR bits of the source address are set to 0. May be used with --dip to key by IP-pairs.
- --dip
- --dip=CIDR
- Use the destination address as the key. When CIDR is given, only the most significant CIDR bits of the destination address are considered; i.e., the least significant 32-CIDR bits of the destination address are set to 0. May be used with --sip to key by IP-pairs.
- --sport
- Use the source port as the key. May be used with --dport to key by port-pairs.
- --dport
- Use the destination port as the key. May be used with --sport to key by port-pairs.
- --protocol
- Use the protocol as the key.
- --icmp
- Use the ICMP type and code as the key. This switch assumes that all the records fed to rwstats are ICMP. It is not possible to key solely on the ICMP type or the ICMP code.
When computing a Top-N or Bottom-N, all flows that have the same key
field(s) will be binned together. The value of each bin may be
based on the number of flows or on the sum of packets or bytes in the
flows. If none of the following value field switches is specified,
--flows will be used.
- --flows
- Use the flow count as the value.
- --packets
- Use the packet count as the value.
- --bytes
- Use the byte count as the value.
To determine the value of N for a Top-N (or Bottom-N) list, one of the following switches must be specified:
- --count=N
- Print the specified number of key-value pairs with the largest (or smallest) values.
- --threshold=N
- Print key-value pairs where the value is greater-than (or less-than) this threshold.
- --percentage=N
- Print key-value pairs where the value is greater-than (or less-than) this percentage of the total value.
To determine whether to compute the Top-N or the Bottom-N, specify one of the following switches. If neither switch is given, --top is assumed:
Miscellaneous switches:
- --integer-ips
- Print IP addresses as integers. If not specified, IP addresses are printed in dotted decimal form.
- --no-titles
- Disable section and column titles. By default, titles are printed.
- --no-columns
- Disable fixed-width columnar output.
- --column-separator=C
- Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.
- --no-final-delimiter
- Do not print the column separator after the final column. Normally a delimiter is printed.
- --delimited
- --delimited=C
- Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.
- --print-filenames
- Print to the standard error the names of input files as they are opened.
- --copy-input=PATH
-
Copy all binary input to the specified file or named pipe. PATH
can be
stdoutto print flows to the standard output as long as the --output-path switch has been used to redirect rwaddrcount's ASCII output. - --output-path=PATH
- Determine where the output of rwaddrcount (ASCII text) is written. If this option is not given, output is written to the standard output.
- --pager=PAGER_PROG
- When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the value of the pager is determined to be the empty string, no paging will be performed and all output will be printed to the terminal.
- --site-config-file=FILENAME
- Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application's directory.
- --legacy-help
- Print help, including legacy switches. See the LEGACY SWITCHES section below for these switches.
LEGACY SWITCHES
Use of the following switches is discouraged; instead, use the replacement switches as indicated.
- --sip-topn=N
-
Use:
--sip [--top] [--flows] --count=N - --sip-top-threshold=N
-
Use:
--sip [--top] [--flows] --threshold=N - --sip-top-pct=N
-
Use:
--sip [--top] [--flows] --percentage=N - --sip-btmn=N
-
Use:
--sip --bottom [--flows] --count=N - --sip-btm-threshold=N
-
Use:
--sip --bottom [--flows] --threshold=N - --sip-btm-pct=N
-
Use:
--sip --bottom [--flows] --percentage=N - --dip-topn=N
-
Use:
--dip [--top] [--flows] --count=N - --dip-top-threshold=N
-
Use:
--dip [--top] [--flows] --threshold=N - --dip-top-pct=N
-
Use:
--dip [--top] [--flows] --percentage=N - --dip-btmn=N
-
Use:
--dip --bottom [--flows] --count=N - --dip-btm-threshold=N
-
Use:
--dip --bottom [--flows] --threshold=N - --dip-btm-pct=N
-
Use:
--dip --bottom [--flows] --percentage=N - --pair-topn=N
-
Use:
--sip --dip [--top] [--flows] --count=N - --pair-top-threshold=N
-
Use:
--sip --dip [--top] [--flows] --threshold=N - --pair-top-pct=N
-
Use:
--sip --dip [--top] [--flows] --percentage=N - --pair-btmn=N
-
Use:
--sip --dip --bottom [--flows] --count=N - --pair-btm-threshold=N
-
Use:
--sip --dip --bottom [--flows] --threshold=N - --pair-btm-pct=N
-
Use:
--sip --dip --bottom [--flows] --percentage=N - --sport-topn=N
-
Use:
--sport [--top] [--flows] --count=N - --sport-top-threshold=N
-
Use:
--sport [--top] [--flows] --threshold=N - --sport-top-pct=N
-
Use:
--sport [--top] [--flows] --percentage=N - --sport-btmn=N
-
Use:
--sport --bottom [--flows] --count=N - --sport-btm-threshold=N
-
Use:
--sport --bottom [--flows] --threshold=N - --sport-btm-pct=N
-
Use:
--sport --bottom [--flows] --percentage=N - --dport-topn=N
-
Use:
--dport [--top] [--flows] --count=N - --dport-top-threshold=N
-
Use:
--dport [--top] [--flows] --threshold=N - --dport-top-pct=N
-
Use:
--dport [--top] [--flows] --percentage=N - --dport-btmn=N
-
Use:
--dport --bottom [--flows] --count=N - --dport-btm-threshold=N
-
Use:
--dport --bottom [--flows] --threshold=N - --dport-btm-pct=N
-
Use:
--dport --bottom [--flows] --percentage=N - --portpair-topn=N
-
Use:
--sport --dport [--top] [--flows] --count=N - --portpair-top-threshold=N
-
Use:
--sport --dport [--top] [--flows] --threshold=N - --portpair-top-pct=N
-
Use:
--sport --dport [--top] [--flows] --percentage=N - --portpair-btmn=N
-
Use:
--sport --dport --bottom [--flows] --count=N - --portpair-btm-threshold=N
-
Use:
--sport --dport --bottom [--flows] --threshold=N - --portpair-btm-pct=N
-
Use:
--sport --dport --bottom [--flows] --percentage=N - --proto-topn=N
-
Use:
--protocol [--top] [--flows] --count=N - --proto-top-threshold=N
-
Use:
--protocol [--top] [--flows] --threshold=N - --proto-top-pct=N
-
Use:
--protocol [--top] [--flows] --percentage=N - --proto-btmn=N
-
Use:
--protocol --bottom [--flows] --count=N - --proto-btm-threshold=N
-
Use:
--protocol --bottom [--flows] --threshold=N - --proto-btm-pct=N
-
Use:
--protocol --bottom [--flows] --percentage=N - --cidr-src=N
-
Use:
--sip=N - --cidr-dest=N
-
Use:
--dip=N
EXAMPLES
$ rwfilter ...| rwstats --sip --count=4
INPUT SIZE: 549092 records for 12990 unique keys
SOURCE IP Key: Top 4 flow counts
sIP| Records|%_of_total| cumul_%|
10.1.1.1| 36604| 6.666278| 6.666278|
10.1.1.2| 13897| 2.530906| 9.197184|
10.1.1.3| 12739| 2.320012| 11.517196|
10.1.1.4| 11807| 2.150277| 13.667473|
$ rwfilter ...| rwstats --dip --packet --count=7
INPUT SIZE: 549092 records for 44654 unique keys
DESTINATION IP Key: Top 7 packet counts
dIP| Packets|%_of_total| cumul_%|
10.1.1.1| 217574| 3.286325| 3.286325|
10.1.1.2| 138177| 2.087081| 5.373407|
10.1.1.3| 121892| 1.841106| 7.214512|
10.1.1.4| 97073| 1.466230| 8.680742|
10.1.1.5| 82284| 1.242851| 9.923593|
10.1.1.6| 80051| 1.209123| 11.132715|
10.1.1.7| 73602| 1.111714| 12.244430|
$ rwfilter ...| rwstats --sip --dip --byte --threshold=100000000
INPUT SIZE: 549092 records for 107136 unique keys
SIP/DIP PAIR Key: Top 5 byte counts (threshold 100000000)
sIP| dIP| Bytes|%_of_total| cumul_%|
10.1.1.1| 10.1.1.2| 307478707| 9.016177| 9.016177|
10.1.1.3| 10.1.1.4| 172164463| 5.048367| 14.064544|
10.1.1.5| 10.1.1.6| 142059589| 4.165604| 18.230147|
10.1.1.7| 10.1.1.8| 119388394| 3.500818| 21.730965|
10.1.1.9| 10.1.1.10| 108268824| 3.174759| 24.905725|
$ rwfilter ...| rwstats --sport --sport --percentage=5
INPUT SIZE: 549092 records for 56799 unique keys
SOURCE PORT Key: Top 3 flow counts (5% == 27454)
sPort| Records|%_of_total| cumul_%|
80| 86677| 15.785515| 15.785515|
53| 64681| 11.779629| 27.565144|
0| 47760| 8.697996| 36.263140|
$ rwfilter ...| rwstats --dport --bottom --count=8
INPUT SIZE: 549092 records for 44772 unique keys
DESTINATION PORT Key: Bottom 8 flow counts
dPort| Records|%_of_total| cumul_%|
19417| 1| 0.000182| 0.000182|
12110| 1| 0.000182| 0.000364|
34777| 1| 0.000182| 0.000546|
8999| 1| 0.000182| 0.000728|
36404| 1| 0.000182| 0.000911|
16682| 1| 0.000182| 0.001093|
27420| 1| 0.000182| 0.001275|
14162| 1| 0.000182| 0.001457|
$ rwfilter ... | \
rwstats --sport --dport --packet --top --threshold=500000
INPUT SIZE: 366309 records for 130307 unique keys
SPORT/DPORT PAIR Key: No packet counts above threshold of 500000
$ rwfilter ... | \
rwstats --sport --dport --packet --top --threshold=50000
INPUT SIZE: 366309 records for 130307 unique keys
SPORT/DPORT PAIR Key: Top 3 packet counts (threshold 50000)
sPort| dPort| Packets|%_of_total| cumul_%|
6699| 3607| 138177| 2.468531| 2.468531|
80| 1179| 59774| 1.067862| 3.536393|
80| 9659| 50319| 0.898949| 4.435342|
$ rwfilter ...| rwstats --protocol --bottom --count=10
INPUT SIZE: 545262 records for 3 unique keys
PROTOCOL Key: Bottom 10 flow counts
protocol| Records|%_of_total| cumul_%|
1| 46319| 8.494815| 8.494815|
17| 132634| 24.324820| 32.819635|
6| 366309| 67.180365|100.000000|
$ rwfilter ...| rwstats --detail-proto-stats=6,17
FLOW STATISTICS--ALL PROTOCOLS: 549092 records
*BYTES min 28; max 88906238
quartiles LQ 122.06478 Med 420.30930 UQ 876.21920 UQ-LQ 754.15442
interval_max|count<=max|%_of_input| cumul_%|
40| 35107| 6.393646| 6.393646|
60| 35008| 6.375616| 12.769263|
100| 49500| 9.014883| 21.784145|
150| 40014| 7.287303| 29.071449|
256| 65444| 11.918586| 40.990034|
1000| 224016| 40.797535| 81.787569|
10000| 75708| 13.787853| 95.575423|
100000| 21981| 4.003154| 99.578577|
1000000| 1901| 0.346208| 99.924785|
4294967295| 413| 0.075215|100.000000|
*PACKETS min 1; max 70023
quartiles LQ 1.76962 Med 3.68119 UQ 7.61567 UQ-LQ 5.84605
interval_max|count<=max|%_of_input| cumul_%|
3| 232716| 42.381969| 42.381969|
4| 61407| 11.183372| 53.565341|
10| 195310| 35.569631| 89.134972|
20| 33310| 6.066379| 95.201351|
50| 17686| 3.220954| 98.422304|
100| 4854| 0.884005| 99.306309|
500| 2760| 0.502648| 99.808957|
1000| 373| 0.067930| 99.876888|
10000| 637| 0.116010| 99.992897|
4294967295| 39| 0.007103|100.000000|
*BYTES/PACKET min 28; max 1500
quartiles LQ 57.98319 Med 90.71150 UQ 164.77250 UQ-LQ 106.78932
interval_max|count<=max|%_of_input| cumul_%|
40| 42568| 7.752435| 7.752435|
44| 15173| 2.763289| 10.515724|
60| 91003| 16.573361| 27.089085|
100| 163850| 29.840173| 56.929258|
200| 153190| 27.898786| 84.828043|
400| 39761| 7.241227| 92.069271|
600| 12810| 2.332942| 94.402213|
800| 7954| 1.448573| 95.850786|
1500| 22783| 4.149214|100.000000|
4294967295| 0| 0.000000|100.000000|
FLOW STATISTICS--PROTOCOL 6: 366309/549092 records
*BYTES min 40; max 88906238
quartiles LQ 310.47331 Med 656.53661 UQ 1089.75344 UQ-LQ 779.28013
interval_max|count<=max|%_of_proto| cumul_%|
40| 29774| 8.128110| 8.128110|
60| 11453| 3.126595| 11.254706|
100| 6915| 1.887751| 13.142456|
150| 16369| 4.468632| 17.611088|
256| 12651| 3.453642| 21.064730|
1000| 196881| 53.747246| 74.811976|
10000| 68989| 18.833553| 93.645529|
100000| 21099| 5.759891| 99.405420|
1000000| 1784| 0.487021| 99.892441|
4294967295| 394| 0.107559|100.000000|
*PACKETS min 1; max 70023
quartiles LQ 3.39682 Med 5.85903 UQ 8.80427 UQ-LQ 5.40745
interval_max|count<=max|%_of_proto| cumul_%|
3| 69358| 18.934288| 18.934288|
4| 55993| 15.285729| 34.220016|
10| 186559| 50.929407| 85.149423|
20| 30947| 8.448332| 93.597755|
50| 16186| 4.418674| 98.016429|
100| 4204| 1.147665| 99.164094|
500| 2178| 0.594580| 99.758674|
1000| 315| 0.085993| 99.844667|
10000| 537| 0.146598| 99.991264|
4294967295| 32| 0.008736|100.000000|
*BYTES/PACKET min 40; max 1500
quartiles LQ 60.19817 Med 96.78616 UQ 175.08044 UQ-LQ 114.88228
interval_max|count<=max|%_of_proto| cumul_%|
40| 36559| 9.980372| 9.980372|
44| 14929| 4.075521| 14.055893|
60| 39593| 10.808634| 24.864527|
100| 100117| 27.331297| 52.195824|
200| 111258| 30.372718| 82.568542|
400| 26020| 7.103293| 89.671834|
600| 8600| 2.347745| 92.019579|
800| 7726| 2.109148| 94.128727|
1500| 21507| 5.871273|100.000000|
4294967295| 0| 0.000000|100.000000|
FLOW STATISTICS--PROTOCOL 17: 132634/549092 records
*BYTES min 32; max 2115559
quartiles LQ 66.53665 Med 150.61551 UQ 242.44095 UQ-LQ 175.90430
interval_max|count<=max|%_of_proto| cumul_%|
20| 0| 0.000000| 0.000000|
40| 5195| 3.916794| 3.916794|
80| 42150| 31.779182| 35.695975|
130| 11528| 8.691587| 44.387563|
256| 45497| 34.302667| 78.690230|
1000| 23401| 17.643289| 96.333519|
10000| 4447| 3.352836| 99.686355|
100000| 389| 0.293288| 99.979643|
1000000| 23| 0.017341| 99.996984|
4294967295| 4| 0.003016|100.000000|
*PACKETS min 1; max 8839
quartiles LQ 0.84383 Med 1.68768 UQ 2.53149 UQ-LQ 1.68766
interval_max|count<=max|%_of_proto| cumul_%|
3| 117884| 88.879171| 88.879171|
4| 4452| 3.356605| 92.235777|
10| 6678| 5.034908| 97.270685|
20| 1766| 1.331484| 98.602168|
50| 1055| 0.795422| 99.397590|
100| 368| 0.277455| 99.675046|
500| 353| 0.266146| 99.941192|
1000| 33| 0.024880| 99.966072|
10000| 45| 0.033928|100.000000|
4294967295| 0| 0.000000|100.000000|
*BYTES/PACKET min 32; max 1415
quartiles LQ 63.23827 Med 91.27180 UQ 158.10219 UQ-LQ 94.86392
interval_max|count<=max|%_of_proto| cumul_%|
20| 0| 0.000000| 0.000000|
24| 0| 0.000000| 0.000000|
40| 5671| 4.275676| 4.275676|
100| 70970| 53.508150| 57.783826|
200| 39298| 29.628904| 87.412730|
400| 12175| 9.179396| 96.592126|
600| 4130| 3.113832| 99.705958|
800| 160| 0.120633| 99.826590|
1500| 230| 0.173410|100.000000|
4294967295| 0| 0.000000|100.000000|
ENVIRONMENT
- SILK_PAGER
- When set to a non-empty string, rwstats automatically invokes this program to display its output a screen at a time. If set to an empty string, rwstats does not automatically page its output.
- PAGER
- When set and SILK_PAGER is not set, rwstats automatically invokes this program to display its output a screen at a time.
- SILK_CONFIG_FILE
- This environment variable is used as the value for the --site-config-file when that switch is not provided.
- SILK_DATA_ROOTDIR
- When the --site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwstats looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.
- SILK_PATH
- This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwstats checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.
NOTES
When used in an IPv6 environment, rwstats will process every record as long as the IP address is not part of the key. When aggregating by an IP address or an IP-pair, rwstats will attempt to convert any IPv6 addresses to IPv4. Records that can be converted will be processed, all other records will be silently ignored.
The output of rwstats is similar to that of rwaddrcount(1), rwtotal(1), and rwuniq(1).
To compute Top-N lists for other key combinations or to see values for Records, Packets, and Bytes in a single view, consider using another SiLK tool and passing the output through sort and head. For example, to see the Top-10 lists for sip,sport combinations, counting by Bytes:
$ rwfilter ...| rwuniq --fields=sip,sport --all --no-titles \
| sort -r -t '|' -k 3 | head -10
rwstats uses an hash table internally when computing Top-N and Bottom-N lists. rwstats may run of memory when processing IP addresses, especially IP-pairs. If rwstats's hash table does run out of memory, rwstats will stop processing input, print a warning to the standard error, output the entries it has computed to that point, and exit with code 16.