NAME

rwbagtool - Perform high-level operations on binary Bag files

SYNOPSIS

  rwbagtool [BAGFILE[,BAGFILE...]] 
        { --add | --subtract | --minimize | --maximize | --divide
          | --scalar-multiply=VALUE
          | --compare={lt | le | eq | ge | gt} }
        [--intersect=SETFILE | --complement-intersect=SETFILE]
        [--mincounter=VALUE] [--maxcounter=VALUE]
        [--minkey=VALUE] [--maxkey=VALUE]
        [--invert] [--coverset] [--output-path=OUTPUTFILE]
        [--note-strip] [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD]

  rwbagtool --help

  rwbagtool --version

DESCRIPTION

rwbagtool performs various operations on Bags. It can add Bags together, subtract a subset of data from a Bag, perform key intersection of a Bag with an IP set, extract the key list of a Bag as an IP set, or filter Bag records based on their counter value.

BAGFILE is a the name of a file or a named pipe, or the names stdin or - to have rwbagtool read from the standard input. If no Bag file names are given on the command line, rwbagtool attempts to read a Bag from the standard input. If BAGFILE does not contain a Bag, rwbagtool prints an error to stderr and exits abnormally.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

Operation switches

The first set of options are mutually exclusive; only one may be specified. If none are specified, the counters in the Bag files are summed.

--add: Sum the counters for each key for all Bag files given on the command line. If a key does not exist, it has a counter of zero. If no other operation is specified, the add operation is the default.
--subtract: Subtract from the first Bag file all subsequent Bag files. If a key does not appear in the first Bag file, rwbagtool assumes it has a value of 0. If any counter subtraction results in a negative number, the key will not appear in the resulting Bag file.
--minimize: Cause the output to contain the minimum counter seen for each key. Keys that do not appear in all input Bags will not appear in the output.
--maximize: Cause the output to contain the maximum counter seen for each key. The output will contain each key that appears in any input Bag.
--divide: Divide the first Bag file by the second Bag file. It is an error if more than two Bag files are specified. Every key in the first Bag file must appear in the second file; the second Bag may have keys that do not appear in the first, and those keys will not appear in the output. Since Bags do not support floating point numbers, the result of the division is rounded to the nearest integer (values ending in .5 are rounded up). If the result of the division is less than 0.5, the key will not appear in the output.
--scalar-multiply=VALUE: Multiply each counter in the Bag file by the scalar VALUE, where VALUE is an integer in the range 1 to 18446744073709551615. This switch accepts a single Bag as input.
--compare=OPERATION: Compare the key/counter pairs in exactly two Bag files. It is an error if more than two Bag files are specified. The keys in the output Bag will only be those whose counter in the first Bag is OPERATION the counter in the second Bag. The counters for all keys in the output will be 1. Any key that does not appear in both input Bag files will not appear in the result. The possible OPERATION values are the strings:

Masking/Limiting switches

The result of the above operation is an intermediate Bag file. The following switches are applied next to remove entries from the intermediate Bag:

--intersect=SETFILE: Mask the keys in the intermediate Bag using the set in SETFILE. SETFILE is the name of a file or a named pipe containing an IPset, or the name stdin or - to have rwbagtool read the IPset from the standard input. If SETFILE does not contain an IPset, rwbagtool prints an error to stderr and exits abnormally. Only key/counter pairs where the key matches an entry in SETFILE are written to the output.
--complement-intersect=SETFILE: As --intersect, but only writes key/counter pairs for keys which do not match an entry in SETFILE.
--mincounter=VALUE: Cause the output to contain only those records whose counter value is VALUE or higher. The allowable range is 1 to the maximum counter value; the default is 1.
--maxcounter=VALUE: Cause the output to contain only those records whose counter value is VALUE or lower. The allowable range is 1 to the maximum counter value; the default is the maximum counter value.
--minkey=VALUE: Cause the output to contain only those records whose key value is VALUE or higher. Default is 0 (or 0.0.0.0). Accepts input as an integer or as an IP address in dotted decimal notation.
--maxkey=VALUE: Cause the output to contain only those records whose key value is VALUE or higher. Default is 4294967295 (or 255.255.255.255). Accepts input as an integer or as an IP address in dotted decimal notation.

Output switches

The following switches control the output.

--invert: Generate a new Bag whose keys are the counters in the intermediate Bag and whose counter is the number of times the counter was seen. For example, this turns the Bag {sip:flow} into the Bag {flow:count(sip)}. Any counter in the intermediate Bag that is larger than the maximum possible key will be attributed to the maximum key; to prevent this, specify --maxcounter=4294967295.
--coverset: Instead of creating a Bag file as the output, write an IPset which contains the keys contained in the intermediate Bag.
--output-path=OUTPUTFILE: Redirect output to OUTPUTFILE. OUTPUTFILE is the name of a file or a named pipe, or the name stdout or - to write the result to the standard output.
--note-strip: Do not copy the notes (annotations) from the input files to the output file. Normally notes from the input files are copied to the output.
--note-add=TEXT: Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.
--note-file-add=FILENAME: Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.
--compression-method=COMP_METHOD: Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:
--help: Print the available options and exit.
--version: Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

The examples assume the following contents for the files:

 Bag1.bag    Bag2.bag    Bag3.bag    Bag4.bag    Mask.set
  3|  10|     1|   1|     2|   8|     1|   1|          2
  4|   7|     4|   2|     4|  10|     4|   3|          4
  6|  14|     7|  32|     6|  14|     6|   4|          6
  7|  23|     8|   2|     7|  12|     7|   4|          8
  8|   2|                 9|   8|     8|   6|

Adding Bag files

 $ rwbagtool --add Bag1.bag Bag2.bag > Bag-sum.bag
 $ rwbagcat --integer-keys Bag-sum.bag
  1|   1|
  3|  10|
  4|   9|
  6|  14|
  7|  55|
  8|   4|

 $ rwbagtool --add Bag1.bag Bag2.bag Bag3.bag > Bag-sum2.bag
 $ rwbagcat --integer-keys Bag-sum2.bag
  1|   1|
  2|   8|
  3|  10|
  4|  19|
  6|  28|
  7|  67|
  8|   4|
  9|   8|

Subtracting Bag Files

 $ rwbagtool --sub Bag1.bag Bag2.bag > Bag-diff.bag
 $ rwbagcat --integer-keys Bag-diff.bag
  3|  10|
  4|   5|
  6|  14|

 $ rwbagtool --sub Bag2.bag Bag1.bag > Bag-diff2.bag
 $ rwbagcat --integer-keys Bag-diff2.bag
  1|   1|
  7|   9|

Getting the Minimum Value

 $ rwbagtool --minimize Bag1.bag Bag2.bag Bag3.bag > Bag-min.bag
 $ rwbagcat --integer-keys Bag-min.bag
  4|   2|
  7|  12|

Getting the Maximum Value

 $ rwbagtool --maximize Bag1.bag Bag2.bag Bag3.bag > Bag-max.bag
 $ rwbagcat --integer-keys Bag-max.bag
  1|   1|
  2|   8|
  3|  10|
  4|  10|
  6|  14|
  7|  32|
  8|   2|
  9|   8|

Dividing Bag Files

 $ rwbagtool --divide Bag2.bag Bag4.bag > Big-div1.bag
 $ rwbagcat --integer-keys Big-div1.bag
   1|   1|
   4|   1|
   7|   8|
 $ rwbagtool --divide Bag4.bag Bag2.bag > Big-div2.bag
 rwbagtool: Error dividing bags; key 6 not in divisor bag

Scalar Multiplication

 $ rwbagtool --scalar-multiply=7 Bag1.bag > Bag-multiply.bag
 $ rwbagcat --integer-keys Bag-multiply.bag
  3|  70|
  4|  49|
  6|  98|
  7| 161|
  8|  14|

Comparing Bag Files

 $ rwbagtool --compare=lt Bag1.bag Bag2.bag > Bag-lt.bag
 $ rwbagcat --integer-keys Bag-lt.bag
  7|   1|

 $ rwbagtool --compare=le Bag1.bag Bag2.bag > Bag-le.bag
 $ rwbagcat --integer-keys Bag-le.bag
  7|   1|
  8|   1|

 $ rwbagtool --compare=eq Bag1.bag Bag2.bag > Bag-eq.bag
 $ rwbagcat --integer-keys Bag-eq.bag
  8|   1|

 $ rwbagtool --compare=ge Bag1.bag Bag2.bag > Bag-ge.bag
 $ rwbagcat --integer-keys Bag-ge.bag
  4|   1|
  8|   1|

 $ rwbagtool --compare=gt Bag1.bag Bag2.bag > Bag-gt.bag
 $ rwbagcat --integer-keys Bag-gt.bag
  4|   1|

Making a Cover Set

 $ rwbagtool --coverset Bag1.bag Bag2.bag Bag3.bag > Cover.set
 $ rwsetcat --integer-keys Cover.set
  1
  2
  3
  4
  6
  7
  8
  9

Inverting a Bag

 $ rwbagtool --invert Bag1.bag > Bag-inv1.bag
 $ rwbagcat --integer-keys Bag-inv1.bag
  2|   1|
  7|   1|
 10|   1|
 14|   1|
 23|   1|

 $ rwbagtool --invert Bag2.bag > Bag-inv2.bag
 $ rwbagcat --integer-keys Bag-inv2.bag
  1|   1|
  2|   2|
 32|   1|

 $ rwbagtool --invert Bag3.bag > Bag-inv3.bag
 $ rwbagcat --integer-keys Bag-inv3.bag
  8|   2|
 10|   1|
 12|   1|
 14|   1|

Masking Bag Files

 $ rwbagtool --intersect=Mask.set Bag1.bag > Bag-mask.bag
 $ rwbagcat --integer-keys Bag-mask.bag
  4|   7|
  6|  14|
  8|   2|

 $ rwbagtool --complement-intersect=Mask.set Bag1.bag > Bag-mask2.bag
 $ rwbagcat --integer-keys Bag-mask2.bag
  3|  10|
  7|  23|

Restricting the Output

 $ rwbagtool --add --maxkey=5 Bag1.bag Bag2.bag > Bag-res1.bag
 $ rwbagcat --integer-keys Bag-res1.bag
  1|   1|
  3|  10|
  4|   9|

 $ rwbagtool --minkkey=3 --maxkey=6 Bag1.bag > Bag-res2.bag
 $ rwbagcat --integer-keys Bag-res2.bag
  3|  10|
  4|   9|
  6|  14|

 $ rwbagtool --mincounter=20 Bag1.bag Bag2.bag > Bag-res3.bag
 $ rwbagcat --integer-keys Bag-res3.bag
  7|  55|

 $ rwbagtool --sub --maxcounter=9 Bag1.bag Bag2.bag > Bag-res4.bag
 $ rwbagcat --integer-keys Bag-res4.bag
  4|   5|