NAME

rwbagtool - Perform high-level operations on binary Bag files

SYNOPSIS

  rwbagtool [BAGFILE[,BAGFILE...]] 
        { --add | --intersect=SETFILE | 
          --complement-intersect=SETFILE |
          --divide | --subtract | --coverset | --invert }
        [--mincounter=VALUE] [--maxcounter=VALUE]
        [--minkey=VALUE] [--maxkey=VALUE] [--output-path=OUTPUTFILE]
        [--note-strip] [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD]

DESCRIPTION

rwbagtool performs various operations on Bags. It can add Bags together, subtract a subset of data from a Bag, perform key intersection of a Bag with an IP set, extract the key list of a Bag as an IP set, or filter Bag records based on their counter value.

BAGFILE is a filename, named pipe, or the keyword stdin. If no Bag file names are given on the command line, rwbagtool attempts to read a Bag from the standard input. If BAGFILE does not contain a Bag, rwbagtool prints an error to stderr and exits abnormally.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--add

Add all Bag files given on the command line. If any file cannot be parsed as a Bag, rwbagtool prints an error to stderr and exits abnormally. Key values are unioned and counter values are summed. If a key does not exist, it has a counter of zero.

--intersect=SETFILE

Add Bag files as described for --add, then mask the keys in the resulting Bag using the set in SETFILE. SETFILE is a filename, named pipe, or the keyword stdin containing an IP set. If SETFILE does not contain an IP set, rwbagtool prints an error to stderr and exits abnormally. Only keys which match an entry in SETFILE are output.

--complement-intersect=SETFILE

As --intersect, but only keys which do not match an entry in SETFILE are output.

--coverset

Add Bag files as described for --add, then output an IPset which contains the keys contained in the resulting Bag.

--invert

Add Bag files as described for --add, then generate a new Bag whose keys are the counters in the intermediate Bag and whose counter is the number of times the counter was seen. For example, this turns the Bag {sip:flow} into the Bag {flow:count(sip)}. Any counter in the intermediate Bag that is larger than the maximum possible key will be attributed to the maximum key; to prevent this, specify --maxcounter=4294967295.

--subtract

Subtract from the first Bag file all subsequent Bag files. If any inputs cannot be parsed as a Bag, rwbagtool prints an error to stderr and exits abnormally. If any counter subtraction results in a negative number, rwbagtool prints an error to stderr and exits abnormally.

--divide

Divide the first Bag file by all subsequent Bag files. For example,

  ( ( BAGFILE1 / BAGFILE2 ) / BAGFILE3 ) ...

All Bags must have the same keys (i.e., all Bags must have the same keys with non-zero counters), or else rwbagtool prints an error and exits abnormally. Likewise, Bags do not support fractional values. To work around this limitation, all counters in the first Bag must be at least as large as the counter for the matching key in subsequent Bags.

Fractional remainders after division are truncated.

--mincounter=VALUE

Cause the output to contain only those records whose counter value is VALUE or higher. Default is 1.

--maxcounter=VALUE

Cause the output to contain only those records whose counter value is VALUE or lower. Default is the maximum counter value.

--minkey=VALUE

Cause the output to contain only those records whose key value is VALUE or higher. Default is 0.0.0.0. Accepts input as an integer or as an IP address in dotted decimal notation.

--maxkey=VALUE

Cause the output to contain only those records whose key value is VALUE or higher. Default is 255.255.255.255. Accepts input as an integer or as an IP address in dotted decimal notation.

--output-path=OUTPUTFILE

Redirect output to OUTPUTFILE. OUTPUTFILE is a filename, named pipe, or the keyword stdout.

--note-strip

Do not copy the notes (annotations) from the input files to the output file. Normally notes from the input files are copied to the output.

--note-add=TEXT

Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME

Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD

Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

none

Do not compress the output using an external library

zlib

Use the zlib(3) library for compressing the output

lzo1x

Use the lzo1x algorithm from the LZO real time compression library for compression

best

Use whichever available method gives the best compression in general, though not necessarily the best for this particular output.

EXAMPLES

To add bag1.bag and bag2.bag together:

  rwbagtool --add bag1.bag bag2.bag > newbag.bag

To only store entries with a counter of 3:

  rwbagtool bag1.bag --mincounter=3 --maxcounter=3 > count3.bag

To build the coverset for that Bag:

  rwbagtool count3.bag --coverset > count3.set

SEE ALSO

rwbag(1), rwbagbuild(1), rwbagcat(1), rwfileinfo(1), rwset(1)