NAME

rwbagcat - Output a binary Bag as text.

SYNOPSIS

  rwbagcat [--stats[=OUTFILE]] [--tree-stats[=OUTFILE]]
        [ --network-structure[=STRUCTURE] | --bin-ips[=SCALE] ]
        [--minkey=VALUE] [--maxkey=VALUE] [--mask-set=PATH]
        [--mincounter=VALUE] [--maxcounter=VALUE] [--zero-counts]
        [--integer-keys | --zero-pad-ips] [--output-path=OUTPUTFILE]
        [--no-columns] [--column-separator=C] [--no-final-delimiter]
        [{--delimited | --delimited=C}] [--pager=PAGER_PROG]
        [BAGFILE...]

  rwbagcat --help

  rwbagcat --version

DESCRIPTION

rwbagcat reads a binary Bag as created by rwbag(1) or rwbagbuild(1), converts it to text, and outputs it to the standard output or the specified file. It can also print various statistics and summary information about the Bag.

rwbagcat reads the BAGFILEs specified on the command line; if no BAGFILE arguments are given, rwbagcat attempts to read the Bag from the standard input. BAGFILE may also explicitly be the keyword stdin or a hyphen (-) to allow rwbagcat to combine files and piped input. If any input does not contain a Bag, rwbagcat prints an error to the standard error and exits abnormally.

When multiple BAGFILEs are specified, each is handled individually; to process the combination of the BAGFILEs, invoke rwbagcat on the output from rwbagtool(1).

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--network-structure

--network-structure=STRUCTURE

Print the sum of the counters for each CIDR block of the specified size listed in STRUCTURE. The switch can also, for each CIDR block, print the number of hosts and smaller CIDR blocks that are occupied. STRUCTURE has one of three forms: CIDR_LIST, CIDR_LIST/, or CIDR_LIST/SUMMARY_EXTRAS. CIDR_LIST and SUMMARY_EXTRAS are each a comma separated list of integers from 1 to 32 as well as the following letters:

• T: /0 network (the total network; ignored in SUMMARY_EXTRAS)

• A: /8 network (legacy class A)

• B: /16 nework (legacy class B)

• C: /24 network (legacy class C)

• X: /27 network

• H: /32 network (individual host IP addresses)

A comma is not required between adjacent letters. Any combination of integers and the symbols T,A,B,C,X,H may be specified in CIDR_LIST. In addition, if the argument contains the letter S or a slash (/), the output line for a CIDR block will also show the number of hosts and smaller CIDR blocks that are occupied. This list of smaller CIDR blocks to summarize is generated by forming the union of CIDR_LIST and SUMMARY_EXTRAS. By default, SUMMARY_EXTRAS is 8,16,24,27, and this default is used when the argument contains S but no slash. If the argument includes a slash and SUMMARY_EXTRAS is empty, the list of smaller subnets is set exactly to CIDR_LIST. If an argument is provided, the CIDR_LIST must contain at least one element. If no argument is specified to the switch, the default is TS/ABCX. An argument that contains nothing but S and/or slash is illegal. This option disables printing of the individual IPs; specify the H argument to the switch to print the IP addresses and their counters.

--bin-ips

--bin-ips=SCALE

Invert the bag and count the total number of unique IP addresses for a given value of the volume bin. For example, turn a Bag {sip:flow} into {flow:count(sip)}. SCALE is a string containing the value linear, binary, or decimal.

• The default behavior is linear: Each distinct counter gets its own bin. Any counter in the input Bag file that is larger than the maximum possible key will be attributed to the maximum key; to prevent this, specify --maxcounter=4294967295.

• binary creates a bag of {log2(flow):count(sip)}. Bin n contains counts in the range [ 2^n, 2^(n+1) ).

• decimal creates one hundred bins for each counter in the range [1,100), and one hundred bins for each counter in the range [100,1000), each counter in the range [1000,10000), etc. Counters are logarithmically distributed among the bins.

--stats

--stats=OUTFILE

Print out breakdown of the network hosts seen, and print out general statistics about the keys and counters.

• count of unique keys

• sum of all the counters

• minimum key

• maximum key

• minimum counter

• maximum counter

• mean of counters

• variance of counters

• standard deviation of counters

• skew of counters

• kurtosis of counters

OUTFILE is a filename, named pipe, or one of the keywords stdout or stderr. Defaults to printing on stderr unless output is being paged, in which case output is to stdout.

--tree-stats

--tree-stats=OUTFILE

Print out metadata about how the bag is performing:

• count of nodes allocated

• total bytes allocated for nodes

• count of leaves allocated

• total bytes allocated for leaves

• count of keys entered

• density of data

OUTFILE is a filename, named pipe, or one of the keywords stdout or stderr. Defaults to printing on stdout.

--minkey=VALUE

Only output records whose minimum key value is VALUE or higher. The valid range is of VALUE 0 to 4294967295, or 0.0.0.0 to 255.255.255.255. Default is 0 (for port or protocol) or 0.0.0.0 (for IP address). Accepts dotted decimal or integer notation.

--maxkey=VALUE

Only output records whose maximum key value is VALUE or lower. The valid range of VALUE is 0 to 4294967295, or 0.0.0.0 to 255.255.255.255. Default is all ports or protocols, or the maximum IP address 255.255.255.255. Accepts dotted decimal or integer notation.

--mask-set=PATH

Only output records whose key appears in the IPset read from the file PATH. When used with --minkey and/or --maxkey, the key must be in the IPset and within when the specified range.

--mincounter=VALUE

Only output records whose minimum counter value is VALUE or higher. The valid range of VALUE is 1 to 18446744073709551615. The default is to print all records with non-zero counter; use --zero-counts to show records whose counter is 0.

--maxcounter=VALUE

Only output records whose maximum counter value is VALUE or lower. The valid range of VALUE is 1 to 18446744073709551615, with the default being the maximum counter value.

--zero-counts

Print keys whose counter is zero. Normally, keys with a counter of zero are suppressed since all keys have a default counter of zero. In order to use this flag, either --mask-set or both --minkey and --maxkey must be specified. When this switch is specified, any counter limit explicitly set by the --maxcounter switch will still be applied.

--output-path=OUTPUTFILE

Redirect output of the --network-structure or --bin-ips options to OUTPUTFILE. OUTPUTFILE is a filename, named pipe, or the keyword stdout.

--zero-pad-ips

Pad IP address octets with zeros so that every octet is three characters wide.

--integer-keys

Print the keys as integers. This flag should be used if the bag is a port or protocol bag.

--no-columns

Disable fixed-width columnar output.

--column-separator=C

Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.

--no-final-delimiter

Do not print the column separator after the final column. Normally a delimiter is printed. When the network summary is requested (--network-structure=S), the separator is always printed before the summary column and never that column.

--delimited

--delimited=C

Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

--pager=PAGER_PROG

When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the value of the pager is determined to be the empty string, no paging will be performed and all output will be printed to the terminal.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

To print the bag:

  $ rwbagcat mybag.bag 
       172.23.1.1|              5|
       172.23.1.2|            231|
       172.23.1.3|              9|
       172.23.1.4|             19|
    192.168.0.100|              1|
    192.168.0.101|              1|
    192.168.0.160|             15|
   192.168.20.161|              1|
   192.168.20.162|              5|
   192.168.20.163|              5|

To print it with full network:

  $ rwbagcat --network-structure=TABCHX mybag.bag
            172.23.1.1      |              5|
            172.23.1.2      |            231|
            172.23.1.3      |              9|
            172.23.1.4      |             19|
          172.23.1.0/27     |            264|
        172.23.1.0/24       |            264|
      172.23.0.0/16         |            264|
    172.0.0.0/8             |            264|
            192.168.0.100   |              1|
            192.168.0.101   |              1|
          192.168.0.96/27   |              2|
            192.168.0.160   |             15|
          192.168.0.160/27  |             15|
        192.168.0.0/24      |             17|
            192.168.20.161  |              1|
            192.168.20.162  |              5|
            192.168.20.163  |              5|
          192.168.20.160/27 |             11|
        192.168.20.0/24     |             11|
      192.168.0.0/16        |             28|
    192.0.0.0/8             |             28|
  TOTAL                     |            292|

Or an abbreviated network structure by class A and C only, including summary information:

  $ rwbagcat --network-structure=ACS mybag.bag 
      172.23.1.0/24     |            264| 4 hosts in 1 /27
  172.0.0.0/8           |            264| 4 hosts in 1 /16, 1 /24, and 1 /27
      192.168.0.0/24    |             17| 3 hosts in 2 /27s
      192.168.20.0/24   |             11| 3 hosts in 1 /27
  192.0.0.0/8           |             28| 6 hosts in 1 /16, 2 /24s, and 3 /27s

To bin by number of unique IP addresses by volume:

  $ rwbagcat --bin-ips mybag.bag
                1|              3|
                5|              3|
                9|              1|
               15|              1|
               19|              1|
              231|              1|

This means there were 3 source hosts in the bag that had a single flow; 3 hosts that had 5 flows; and one host each that had 9, 15, 19, and 231 flows.

For a log2 breakdown of the counts:

  $ rwbagcat --bin-ips=binary mybag.bag
     2^0 to 2^1-1|              3|
     2^2 to 2^3-1|              3|
     2^3 to 2^4-1|              2|
     2^4 to 2^5-1|              1|
     2^7 to 2^8-1|              1|

Statistics:

  $ rwbagcat --stats mybag.bag

  Statistics
                keys:  10
     sum of counters:  292
         minimum key:  172.23.1.1
         maximum key:  192.168.20.163
       minimum count:  1
       maximum count:  231
                mean:  29.2
            variance:  5064
  standard deviation:  71.16
                skew:  2.246
            kurtosis:  8.1

  $ rwbagcat --tree-stats mybag.bag 
     nodes allocated:  5 (10240 bytes)
    leaves allocated:  4 (1024 bytes)
       keys inserted:  10 (10 unique)
     counter density:  7.81%

ENVIRONMENT

SILK_PAGER

When set to a non-empty string, rwbagcat automatically invokes this program to display its output a screen at a time. If set to an empty string, rwbagcat does not automatically page its output.

PAGER

When set and SILK_PAGER is not set, rwbagcat automatically invokes this program to display its output a screen at a time.

SEE ALSO

rwbag(1), rwbagbuild(1), rwbagtool(1)