NAME

rwsetcat - Print the IP addresses in a binary IPset file

SYNOPSIS

  rwsetcat [--count-ips] [--print-statistics] [--print-ips]
        [--cidr-blocks | --cidr-blocks=0 | --cidr-blocks=1]
        [--network-structure | --network-structure=STRUCTURE]
        [--ip-ranges]
        [--ip-format=FORMAT] [--integer-ips] [--zero-pad-ips]
        [--no-columns] [--column-separator=C] [--no-final-delimiter]
        [{--delimited | --delimited=C}]
        [--print-filenames | --print-filenames=0 | --print-filenames=1]
        [--output-path=PATH] [--pager=PAGER_PROG] [SET_FILE...]

  rwsetcat --help

  rwsetcat --version

DESCRIPTION

When run with no switches, rwsetcat reads each IPset file given on the command line and prints its constituent IP addresses to the standard output. When the input IPset contains IPv4 data, rwsetcat prints one IP address per line; when the IPset contains IPv6 data, rwsetcat prints the IPs as CIDR blocks. If no file names are listed on the command line, rwsetcat will attempt to read an IPset from the standard input.

rwsetcat can produce additional information about IPset files, such as the number of IPs they contain, the number of IPs at the /8, /16, /24, and /27 levels, and the minimum and maximum IPs.

To create an IPset file from SiLK Flow records, use rwset(1). rwsetbuild(1) creates an IPset from textual input. The --coverset switch on rwbagtool(1) creates an IPset from a binary SiLK Bag. To determine whether an IPset file contains an IP address, use rwsetmember(1).

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--count-ips

Print a count of the number of IP addresses in the IPset file. This switch disables the printing of the IP addresses in the IPset file. See --print-ips for more information. When --count-ips is specified and more than one IPset file is provided, rwsetcat prepends the name of the input file and a colon to the IP address count. See the description of the --print-filenames switch for more information.

Print a summary of the IPset. The summary includes the minimum IP address, the maximum IP address, the number of IP addresses in the IPset, and the number of IPs in each netblock predefined netblocks. For an IPset containing only IPv4 addresses, the netblocks are /8, /16, /24, and /27, and the output includes what percentage of IPv4 address space is covered. For an IPv6 IPset, the netblock are /8, /16, /24, /32, /40, /48, /56, /64, /72, /80, /88, /96, /112, and /120.

This switch disables the printing of the IP addresses in the IPset. See --print-ips for more information. When --print-statistics is specified and more than one IPset file is provided, rwsetcat prints the name of the input file, a colon, and a newline prior to printing the statistics. See the description of the --print-filenames switch for more information.

Force printing of the IP addresses, even when the --count-ips or --print-statistics option is provided.

--cidr-blocks
--cidr-blocks=0
--cidr-blocks=1

When an argument is not provided to the switch or when the argument is 1, group sequential IPs into the largest possible CIDR block and print CIDR blocks in the IPset file, If the argument is 0, print the individual IPs in the IPset file. By default, rwsetcat prints individual IPs for IPv4 IPsets and CIDR blocks for IPv6 IPsets. See also the --ip-ranges switch. This switch cannot be combined with the --network-structure switch.

--network-structure
--network-structure=STRUCTURE

For each numeric value in STRUCTURE, group the IPs in the IPset into a netblock of that size and print the number of hosts and, optionally, print the number of smaller, occupied netblocks that each larger netblock contains. When STRUCTURE begins with v6:, the IPs in the IPset are treated as IPv6 addresses, and any IPv4 addresses are mapped into the ::ffff:0:0/96 netblock. Otherwise, the IPs are treated as IPv4 addresses, and any IPv6 address outside the ::ffff:0:0/96 netblock is ignored. Aside from the initial v6: (or v4:, for consistency), STRUCTURE has one of following forms:

  1. NETBLOCK_LIST/SUMMARY_LIST. Group IPs into the sizes specified in either NETBLOCK_LIST or SUMMARY_LIST. rwsetcat prints a row for each occupied netblock specified in NETBLOCK_LIST, where the row lists the base IP of the netblock, the number of hosts, and the number of smaller, occupied netblocks having a size that appears in either NETBLOCK_LIST or SUMMARY_LIST. (The values in SUMMARY_LIST are only summarized; they are not printed.)

  2. NETBLOCK_LIST/. Similar to the first form, except all occupied netblocks are printed, and there are no netblocks that are only summarized.

  3. NETBLOCK_LISTS. When the character S appears anywhere in the NETBLOCK_LIST, rwsetcat provides a default value for the SUMMARY_LIST. That default is 8,16,24,27 for IPv4, and 48,64 for IPv6. rwsetcat ignores S if / is present.

  4. NETBLOCK_LIST. When neither S nor / appear in STRUCTURE, the output does not include the number of smaller, occupied netblocks.

  5. Empty. When STRUCTURE is empty or only contains v6: or v4:, the NETBLOCK_LIST prints a single row for the total network (the /0 netblock) giving the number of hosts and the number of smaller, occupied netblocks using the same default list specified in form 3.

NETBLOCK_LIST and SUMMARY_LIST contain a comma separated list of numbers between 0 (the total network) and the size for an individual host (32 for IPv4 or 128 for IPv6). The characters T and H may be used as aliases for 0 and the host netblock, respectively. In addition, when parsing the lists as IPv4 netblocks, the characters A, B, C, and X are supported as aliases for 8, 16, 24, and 27, respectively. A comma is not required between adjacent letters. The --network-structure switch disables printing of the IPs in the IPset file; specify the H argument to the switch to print each individual IP address.

--ip-ranges

Cause the output to contain three pipe-delimited (|) columns: the first is the number of IPs in the contiguous range, the second is the start of the range, and the final is the end of the range. This prints the IPset in the fewest number of lines.

--ip-format=FORMAT

Specify how IP addresses are printed. When this switch is not specified, the SILK_IP_FORMAT environment variable is checked for a format. If it is empty or contains an invalid format, IPs are printed in the canonical format. The FORMAT is one of:

canonical

Print IP addresses in their canonical form: dotted quad for IPv4 (127.0.0.1) and hexadectet for IPv6 (2001:db8::1). Note that IPv6 addresses in ::ffff:0:0/96 and some IPv6 addresses in ::/96 will be printed as a mixture of IPv6 and IPv4.

zero-padded

Print IP addresses in their canonical form, but add zeros to the output so it fully fills the width of column. The addresses 127.0.0.1 and 2001:db8::1 are printed as 127.000.000.001 and 2001:0db8:0000:0000:0000:0000:0000:0001, respectively.

decimal

Print IP addresses as integers in decimal format. The addresses 127.0.0.1 and 2001:db8::1 are printed as 2130706433 and 42540766411282592856903984951653826561, respectively.

hexadecimal

Print IP addresses as integers in hexadecimal format. The addresses 127.0.0.1 and 2001:db8::1 are printed as 7f000001 and 20010db8000000000000000000000001, respectively.

force-ipv6

Print all IP addresses in the canonical form for IPv6 without using any IPv4 notation. Any IPv4 address is mapped into the ::ffff:0:0/96 netblock. The addresses 127.0.0.1 and 2001:db8::1 are printed as ::ffff:7f00:1 and 2001:db8::1, respectively.

--integer-ips

Print IP addresses as integers. This switch is equivalent to --ip-format=decimal, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--zero-pad-ips

Print IP addresses as fully-expanded, zero-padded values in their canonical form. This switch is equivalent to --ip-format=zero-padded, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--no-columns

Disable fixed-width columnar output when printing the output from the --network-structure or --ip-ranges switch.

--column-separator=C

Use specified character between columns produced by the --network-structure and --ip-ranges switches. This character is also used after the final column when --ip-ranges is specified. When this switch is not specified, the default of '|' is used.

--no-final-delimiter

Do not print the column separator after the final column in the output produced by --ip-ranges. Normally a delimiter is printed.

--delimited
--delimited=C

Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

If an argument is not provided to the switch or if the argument is 1, print the name of the IPset file prior to printing information about the IPset file regardless of the number of IPset files specified on the command line or the type of information to be printed. If the switch is provided and its argument is 0, suppress printing the name of the IPset file regardless of the number of IPset files or type of information. When the switch is not provided, rwsetcat's behavior depends on the type of information to be printed and on the number of input IPset files: If multiple IPset files are provided and --count-ips or --print-statistics is given, rwsetcat prints the name of a file, a colon (:), a newline (unless --count-ips was specified), and the requested information; otherwise, rwsetcat does not print the file name.

--output-path=PATH

Write the textual output to PATH, where PATH is a filename, a named pipe, the keyword stderr to write the output to the standard error, or the keyword stdout or - to write the output to the standard output (and bypass the paging program). If PATH names an existing file, rwsetcat exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this switch is not given, the output is either sent to the pager or written to the standard output. Since SiLK 3.15.0.

--pager=PAGER_PROG

When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the --output-path switch is given or if the value of the pager is determined to be the empty string, no paging is performed and all output is written to the terminal.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

In the following examples, the dollar sign ($) represents the shell prompt. Some input lines are split over multiple lines in order to improve readability, and a backslash (\) is used to indicate such lines.

The file sample.set contains an IPset of IPv4 addresses, and the file set1-v6.set contains an IPset of IPv6 addresses.

Producing simple output with an IPv4 IPset

By default, rwsetcat prints the contents of an IPset.

 $ rwsetcat sample.set
 10.1.2.250
 10.1.2.251
 10.1.2.252
 10.1.2.253
 10.1.2.254
 10.1.2.255
 10.1.3.0
 10.1.3.1
 10.1.3.2
 10.1.3.3
 10.1.3.4

Use the --cidr-blocks switch to print the contents in CIDR notation.

 $ rwsetcat --cidr-blocks sample.set
 10.1.2.250/31
 10.1.2.252/30
 10.1.3.0/30
 10.1.3.4

Add the --ip-format switch to change how the IPs are presented. For text-based sorting, use the --ip-format=zero-padded switch to force three digits per octet.

 $ rwsetcat --ip-format=zero-padded --cidr-blocks sample.set
 010.001.002.250/31
 010.001.002.252/30
 010.001.003.000/30
 010.001.003.004

For numerical sorting, print the IPs as integers.

 $ rwsetcat --ip-format=decimal sample.set
 167838458
 167838459
 167838460
 167838461
 167838462
 167838463
 167838464
 167838465
 167838466
 167838467
 167838468

Getting simple output for an IPv6 IPset

When printing an IPset containing IPv6 addresses, addresses are grouped into CIDR blocks by default.

 $ rwsetcat set1-v6.set
 2001:db8:0:5::/68
 2001:db8:0:5:f000::/68
 2001:db8:0:c::/67
 2001:db8:0:c:4000::/66
 2001:db8:0:f:8000::/65
 2001:db8:0:11::/64
 2001:db8:0:12::/63
 2001:db8:0:14::/62
 2001:db8:0:18::/61
 2001:db8:0:20::/60
 2001:db8:0:40::/59

Specify an argument of 0 to the --cidr-blocks switch to see the individual IPs.

 $ rwsetcat --cidr-blocks=0 set1-v6.set | head -4
 2001:db8:0:5::
 2001:db8:0:5::1
 2001:db8:0:5::2
 2001:db8:0:5::3

Finding the number of IPs in an IPset

The --count-ips switch prints the number IPs in the IPset.

 $ rwsetcat --count-ips sample.set
 11

 $ rwsetcat --count-ips set1-v6.set
 1180591620717411303424

The number of IPs may also be produced using the --network-structure switch as described below.

Viewing IP ranges

To see contiguous IPs printed as ranges, use the --ip-ranges switch. The output has three columns that contain the length of the range, its starting IP, and its ending IP.

 $ rwsetcat --ip-ranges sample.set
         11|     10.1.2.250|       10.1.3.4|

Add the --ip-format=decimal switch to see contiguous IPs printed as ranges of integers.

 $ rwsetcat --ip-ranges --ip-format=decimal sample.set
         11| 167838458| 167838468|

Use the --delimited switch to produce the same output as a list of comma separated values.

 $ rwsetcat --ip-ranges --ip-format=decimal --delimited=, sample.set
 11,167838458,167838468

The UNIX cut(1) tool can be used to remove the number of IPs in the range, so that the output only contains the starting and ending IPs.

 $ rwsetcat --ip-ranges --ip-format=decimal --delimited=, sample.set \
     | cut -d"," -f2,3
 167838458,167838468

 $ rwsetcat --ip-ranges set1-v6.set | cut -d'|' -f2,3
          2001:db8:0:5::|         2001:db8::5:fff:ffff:ffff:ffff
     2001:db8:0:5:f000::|        2001:db8::5:ffff:ffff:ffff:ffff
          2001:db8:0:c::|        2001:db8::c:1fff:ffff:ffff:ffff
     2001:db8:0:c:4000::|        2001:db8::c:7fff:ffff:ffff:ffff
     2001:db8:0:f:8000::|        2001:db8::f:ffff:ffff:ffff:ffff
         2001:db8:0:11::|       2001:db8::2f:ffff:ffff:ffff:ffff
         2001:db8:0:40::|       2001:db8::5f:ffff:ffff:ffff:ffff

Reading an IPset from the standard input

rwsetcat will read the IPset file from the standard input when no file name is given on the command line. An IP address converter is created by having the input to rwsetcat be the output from rwsetbuild(1).

 $ echo 10.10.10.10 | rwsetbuild | rwsetcat --ip-format=decimal
 168430090

To see the unique source and destination IP addresses in the SiLK Flow file data.rw, use rwset(1) to generate an IPset and send the output of rwset to the standard input of rwsetcat.

 $ rwset --any-file=stdout data.rw | rwsetcat | head -4
 10.4.52.235
 10.5.231.251
 10.9.77.117
 10.11.88.88

Getting multiple types of output

To see the contents of the IPset and also get a count of IPs, use multiple options.

 $ rwsetcat --count-ips --cidr-blocks sample.set
 11
 10.1.2.250/31
 10.1.2.252/30
 10.1.3.0/30
 10.1.3.4

Working with multiple IPset files

When multiple IPset files are specified on the command line, rwsetcat prints the contents of each file one after the other.

 $ rwsetcat --cidr-blocks=1  sample.set set1-v6.set
 10.1.2.250/31
 10.1.2.252/30
 10.1.3.0/30
 10.1.3.4
 2001:db8:0:5::/68
 2001:db8:0:5:f000::/68
 2001:db8:0:c::/67
 2001:db8:0:c:4000::/66
 2001:db8:0:f:8000::/65
 2001:db8:0:11::/64
 2001:db8:0:12::/63
 2001:db8:0:14::/62
 2001:db8:0:18::/61
 2001:db8:0:20::/60
 2001:db8:0:40::/59

To print the union of multiple the IPset files, use rwsettool(1) to join the files and have rwsetcat print the result.

 $ rwsettool --union set1-v6.set sample.set | rwsetcat --cidr-blocks=1
 ::ffff:10.1.2.250/127
 ::ffff:10.1.2.252/126
 ::ffff:10.1.3.0/126
 ::ffff:10.1.3.4
 2001:db8:0:5::/68
 2001:db8:0:5:f000::/68
 2001:db8:0:c::/67
 2001:db8:0:c:4000::/66
 2001:db8:0:f:8000::/65
 2001:db8:0:11::/64
 2001:db8:0:12::/63
 2001:db8:0:14::/62
 2001:db8:0:18::/61
 2001:db8:0:20::/60
 2001:db8:0:40::/59

When counting the IPs in multiple IPset files, rwsetcat prepends the file name and a colon to the count. (The - argument causes rwsetcat to read the standard input in addition to the named file.)

 $ cat set1-v6.set | rwsetcat --count-ips sample.set -
 sample.set:11
 -:1180591620717411303424

Provide an argument of 0 to --print-filenames to suppress printing of the input IPset file name.

 $ cat set1-v6.set \
     | rwsetcat --count-ips --print-filenames=0 sample.set -
 11
 1180591620717411303424

Use the --print-filenames switch to force rwsetcat to print the file name when only one IPset is given.

 $ rwsetcat --count-ips --print-filenames sample.set
 sample.set:11

The --print-filenames switch also causes rwsetcat to print the file name when it normally would not.

 $ rwsetcat --ip-ranges --ip-format=decimal --print-filenames sample.set
 sample.set:
         11| 167838458| 167838468|

Seeing which netblocks are occupied

The --network-structure switch counts and prints information about which netblocks are occupied. The default output when no argument is given to the switch is a single line.

 $ rwsetcat --network sample.set
 TOTAL| 11 hosts in 1 /8, 1 /16, 2 /24s, and 2 /27s

The default is equivalent to an argument of TS.

 $ rwsetcat --network=TS sample.set
 TOTAL| 11 hosts in 1 /8, 1 /16, 2 /24s, and 2 /27s

An argument of T suppresses the subnet counts, and the output is the number of IPs in the IPset.

 $ rwsetcat --network=T sample.set
 TOTAL| 11

The argument T is equivalent to the 0 netblock.

 $ rwsetcat --network=0 sample.set
 TOTAL| 11

The subnets represented by S are 8, 16, 24, and 27. A different set of subnets to summarize may be specified by giving those subnets after a slash:

 $ rwsetcat --network=T/12,18,30 sample.set
 TOTAL| 11 hosts in 1 /12, 1 /18, and 4 /30s

The presence of a slash causes rwsetcat to ignore S.

 $ rwsetcat --network=TS/12,18 sample.set
 TOTAL| 11 hosts in 1 /12 and 1 /18

Putting a number in front of the slash adds a row the output for each netblock of that size that is occupied.

 $ rwsetcat --network=30T/12,18 sample.set
   10.1.2.248/30     | 2 hosts
   10.1.2.252/30     | 4 hosts
   10.1.3.0/30       | 4 hosts
   10.1.3.4/30       | 1 host
 TOTAL               | 11 hosts in 1 /12, 1 /18, and 4 /30s

For each row, the number of smaller, occupied netblocks is printed.

 $ rwsetcat --network=12,18/30 sample.set
     10.1.0.0/18       | 11 hosts in 4 /30s
   10.0.0.0/12         | 11 hosts in 1 /18 and 4 /30s
 TOTAL                 | 11 hosts in 1 /12, 1 /18, and 4 /30s

Although no numbers are required to follow the slash, the argument must include the slash for rwsetcat to produce the counts for each subnet.

 $ rwsetcat --network=16,24/ sample.set
   10.1.2.0/24       | 6 hosts
   10.1.3.0/24       | 5 hosts
 10.1.0.0/16         | 11 hosts in 2 /24s

 $ rwsetcat --network=16,24 sample.set
   10.1.2.0/24       | 6
   10.1.3.0/24       | 5
 10.1.0.0/16         | 11

For historical reasons, A, B, C, and X are equivalent to the 8, 16, 24, and 27 netblocks.

 $ rwsetcat --network=B,C sample.set
   10.1.2.0/24       | 6
   10.1.3.0/24       | 5
 10.1.0.0/16         | 11

Adding an argument of H tells rwsetcat to print the hosts.

 $ rwsetcat --network=ABCXHST sample.set
           10.1.2.250      |
           10.1.2.251      |
           10.1.2.252      |
           10.1.2.253      |
           10.1.2.254      |
           10.1.2.255      |
         10.1.2.224/27     | 6 hosts
       10.1.2.0/24         | 6 hosts in 1 /27
           10.1.3.0        |
           10.1.3.1        |
           10.1.3.2        |
           10.1.3.3        |
           10.1.3.4        |
         10.1.3.0/27       | 5 hosts
       10.1.3.0/24         | 5 hosts in 1 /27
     10.1.0.0/16           | 11 hosts in 2 /24s and 2 /27s
   10.0.0.0/8              | 11 hosts in 1 /16, 2 /24s, and 2 /27s
 TOTAL                     | 11 hosts in 1 /8, 1 /16, 2 /24s, and 2 /27s

The --network-structure switch defaults to treating the input as an IPset containing only IPv4 addresses. The results when running it on the IPv6 IPset file set1-v6.set are odd.

 $ rwsetcat --network=TS set1-v6.set
 TOTAL| 0 hosts in 0 /8s, 0 /16s, 0 /24s, and 0 /27s

The v6: prefix is required for rwsetcat to treat the input as IPv6.

 $ rwsetcat --network=v6:TS set1-v6.set
 TOTAL| 1180591620717411303424 hosts in 1 /48 and 66 /64s

As shown in that example, when the v6: prefix is given, the S character represents the 48 and 64 netblocks. The characters A, B, C, and X are not allowed when treating the input as IPv6.

 $ rwsetcat --network=v6:A set1-v6.set
 rwsetcat: Invalid network-structure character 'A'

The H character still represents the hosts.

$ rwsetcat --network=v6:H set1-v6.set | head -4 2001:db8:0:5::| 2001:db8:0:5::1| 2001:db8:0:5::2| 2001:db8:0:5::3|

When processing an IPv4 IPset as though it is IPv6, the IPv4 hosts are mapped into the ::ffff:0:0/96 netblock. (This is similar to passing a value of force to the --ipv6-policy switch on tools such as rwcut(1).)

 $ rwsetcat --network=v6:96TS sample.set
   ::ffff:0.0.0.0/96    | 11 hosts
 TOTAL                  | 11 hosts in 1 /48, 1 /64, and 1 /96

When the v6: prefix is not present and --network-structure is used on an IPset containing IPv6 addresses, only those addresses in the ::ffff:0:0/96 netblock are visible to rwsetcat. This is similar to giving the --ipv6-policy switch an argument of asv4.

 $ rwsettool --union set1-v6.set sample.set | rwsetcat --network=v6:TS
 TOTAL| 1180591620717411303435 hosts in 2 /48s and 67 /64s

 $ rwsettool --union set1-v6.set sample.set | rwsetcat --network=TS
 TOTAL| 11 hosts in 1 /8, 1 /16, 2 /24s, and 2 /27s

Seeing a summary of an IPset

Use --print-statistics to get a summary of the IPset file.

 $ rwsetcat --print-statistics --print-filenames sample.set
 sample.set:
 Network Summary
         minimumIP = 10.1.2.250
         maximumIP = 10.1.3.4
                 11 hosts (/32s),    0.000000% of 2^32
                  1 occupied /8,     0.390625% of 2^8
                  1 occupied /16,    0.001526% of 2^16
                  2 occupied /24s,   0.000012% of 2^24
                  2 occupied /27s,   0.000001% of 2^27

 $ rwsetcat --print-statistics set1-v6.set
 Network Summary
        minimumIP = 2001:db8:0:5::
        maximumIP = 2001:db8::5f:ffff:ffff:ffff:ffff
                                              1 occupied /8
                                              1 occupied /16
                                              1 occupied /24
                                              1 occupied /32
                                              1 occupied /40
                                              1 occupied /48
                                              1 occupied /56
                                             66 occupied /64s
                                          16384 occupied /72s
                                        4194304 occupied /80s
                                     1073741824 occupied /88s
                                   274877906944 occupied /96s
                                 70368744177664 occupied /104s
                              18014398509481984 occupied /112s
                            4611686018427387904 occupied /120s
                         1180591620717411303424 hosts (/128s)

ENVIRONMENT

SILK_IP_FORMAT

This environment variable is used as the value for --ip-format when that switch is not provided. Since SiLK 3.11.0.

SILK_PAGER

When set to a non-empty string, rwsetcat automatically invokes this program to display its output a screen at a time. If set to an empty string, rwsetcat does not automatically page its output.

PAGER

When set and SILK_PAGER is not set, rwsetcat automatically invokes this program to display its output a screen at a time.

SEE ALSO

rwset(1), rwsetbuild(1), rwsettool(1), rwsetmember(1), rwbagtool(1), rwcut(1), silk(7), cut(1)