NAME

rwpmapbuild - Create a binary prefix map from a text file

SYNOPSIS

  rwpmapbuild [--input-file=FILENAME] [--output-file=FILENAME]
        [--mode={ipv4|ipv6|proto-port}] [--dry-run] [--ignore-errors]
        [--note-add=TEXT] [--note-file-add=FILENAME]
        [--invocation-strip]

  rwpmapbuild --help

  rwpmapbuild --version

DESCRIPTION

A prefix map file is a binary file that maps a value (specifically either an IP addresses or a protocol-port pair) to a string label. rwpmapbuild reads textual input to create a binary prefix map file. The syntax of this input is described in the "INPUT FILE FORMAT" section below.

Once you have created a prefix map file, you may use the file in rwfilter(1), rwstats(1), rwuniq(1), rwgroup(1), rwsort(1), or rwcut(1) to partition, count, sort and display SiLK flow records based on the string labels defined in the prefix map. See the pmapfilter(3) manual page for details. To view the contents of a prefix map file, use rwpmapcat(1). To query the contents of a prefix map, use rwpmaplookup(1).

The textual input is read from the file specified by --input-file or from the standard input when the switch is not provided. The binary output is written to the location named by --output-file or to the standard output when the switch is not provided and the standard output is not connected to a terminal.

The remainder of this section provides three example files that could be used as input to rwpmapbuild. For details on the syntax of the input, see the "INPUT FILE FORMAT" section that follows the description of the command line "OPTIONS".

Sample IPv4 input file

The following is a sample input file for rwpmapbuild that describes the registry of special-purpose IPv4 addresses. Any IP address that is not a special-purpose address get the label Normal.

 # Prefix map sample input file for special purpose IPv4 addresses
 map-name   ipv4-special
 mode       ipv4
 default    Normal

 #    Each line has an either a CIDR block or a pair of IP
 #    addresses and then a label for that range
 0.0.0.0/8          This host on this network [RFC1122 section 3.2.1.3]
 10.0.0.0/8         Private-Use [RFC1918]
 100.64.0.0/10      Shared Address Space [RFC6598]
 127.0.0.0/8        Loopback [RFC1122 section 3.2.1.3]
 169.254.0.0/16     Link Local [RFC3927]
 172.16.0.0/12      Private-Use [RFC1918]
 192.0.0.0/24       IETF Protocol Assignments [RFC6890 section 2.1]
 192.0.0.0/29       IPv4 Service Continuity Prefix [RFC7335]
 #    A range of a single IP address requires a "/32" suffix or
 #    that the IP address be repeated
 192.0.0.8/32       IPv4 dummy address [RFC7600]
 192.0.0.9/32       Port Control Protocol Anycast [RFC7723]
 192.0.0.10/32      Traversal Using Relays around NAT Anycast [draft]
 #    A range may be specified as two IP addresses
 192.0.0.170  192.0.0.171   NAT64/DNS64 Discovery [RFC7050 section 2.2]
 192.0.2.0/24       Documentation (TEST-NET-1) [RFC5737]
 192.31.196.0/24    AS112-v4 [RFC7535]
 192.52.193.0/24    AMT [RFC7450]
 192.88.99.0/24     Deprecated (6to4 Relay Anycast) [RFC7526]
 192.168.0.0/16     Private-Use [RFC1918]
 192.175.48.0/24    Direct Delegation AS112 Service [RFC7534]
 198.18.0.0/15      Benchmarking [RFC2544]
 198.51.100.0/24    Documentation (TEST-NET-2) [RFC5737]
 203.0.113.0/24     Documentation (TEST-NET-3) [RFC5737]
 240.0.0.0/4        Reserved [RFC1112 section 4]
 255.255.255.255/32 Limited Broadcast [RFC919 section 7]

Sample IPv6 input file

The following input file for rwpmapbuild describes IPv6 address space. The file demonstrates the use of the label statement. It uses a hyphen ("-") as the label for any undefined ranges.

 # Prefix map sample input file for IPv6 address space
 map-name   iana-ipv6
 mode       ipv6

 label      0   RFC3849
 label      1   RFC3879
 label      2   RFC4048
 label      3   RFC4193
 label      4   RFC4291
 label      5   RFC4291 Loopback Address
 label      6   RFC4291 Unspecified Address
 label      7   RFC4291 IPv4-mapped Address
 label      8   RFC5180
 label      9   RFC6666
 label     10   RFC7723
 label     11   -

 default        -

 0000::/8       4
 ::1/128        5
 ::/128         6
 ::ffff:0:0/96  7
 0100::/8       4
 0100::/64      9           # RFC6666
 0200::/7       2           # RFC4048
 0400::/6       4
 0800::/5       4
 1000::/4       4
 2000::/3       4
 2001:1::1/128 10           # RFC7723
 2001:2::/48    8           # Benchmarking
 2001:db8::/32  0           # Documentation
 4000::/3       4
 6000::/3       4
 8000::/3       4
 a000::/3       4
 c000::/3       4
 e000::/4       4           # You may use the label number or the
 f000::/5       RFC4291     # exact label name, but any other text
 f800::/6       4           # causes rwpmapbuild to issue an error
 fc00::/7       RFC4193
 fe00::/9       4
 fe80::/10      4
 fec0::/10      RFC3879
 ff00::/8       4

Sample protocol-port input file

This is a small sample of a file that could be used to label IP protocols, specific ports within the TCP and UDP protocols, and ICMP type and code values. When ranges overlap or one range is a specialization of another, the wider or more general range should be listed first, followed by the narrower or more specific ranges.

 map-name   protocol-port-example
 mode       proto-port

 #    The range is either a single protocol or a protocol and
 #    a port separated by a slash.
 1      1           ICMP
 #    Specify the wider categories first, then specialize
 6      6           TCP
 6/0    6/1024      TCP/Generic reserved
 #    A range of a single port requires both the starting
 #    value and the ending value
 6/21   6/21        TCP/FTP
 6/22   6/22        TCP/SSH
 6/25   6/25        TCP/SMTP
 6/80   6/80        TCP/HTTP
 6/443  6/443       TCP/HTTPS
 6/6000 6/6063      TCP/X11
 17     17          UDP
 17/0   17/1024     UDP/Generic reserved
 17/53  17/53       UDP/DNS
 17/67  17/68       UDP/DHCP
 50     50          ESP
 58     58          ICMPv6
 #    For ICMP Type/Code, convert the type and code to a port
 #    value using this expression:  type * 256 + code
 1/0    1/255       ICMP/Echo Reply
 1/768  1/1023      ICMP/Destination Unreachable
 1/1024 1/1279      ICMP/Source Quench
 1/768  1/768       ICMP/Net Unreachable
 1/769  1/769       ICMP/Host Unreachable
 1/770  1/770       ICMP/Protocol Unreachable
 1/771  1/771       ICMP/Port Unreachable

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--input-file=FILENAME

Read the textual input from FILENAME. You may use stdin or - to represent the standard input. When this switch is not provided, the input is read from the standard input unless the standard input is a terminal. rwpmapbuild will read textual input from the terminal if the standard input is explicitly specified as the input. The input file format is described below.

--output-file=FILENAME

Write the binary prefix map to FILENAME. You may use stdout or - to represent the standard output. When this switch is not provided, the prefix map is written to the standard output unless the standard output is connected to a terminal.

--mode={ipv4|ipv6|proto-port}

Specify the type of the input, as if a mode statement appeared in the input stream. The value specified by this switch must not conflict with an explicit mode statement appearing in the input.

--dry-run

Do not write the output file. Simply check the syntax of the input file.

--ignore-errors

Write the output file regardless of any errors encountered while parsing the input file.

--note-add=TEXT

Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME

Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--invocation-strip

Do not record the command used to create the prefix map in the output. When this switch is not given, the invocation is written to the file's header, and the invocation may be viewed with rwfileinfo(1). Since SiLK 3.12.0.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

INPUT FILE FORMAT

This section describes the format of the textual input file for rwpmapbuild. Three example files are shown above in the "DESCRIPTION" section.

Blank lines or lines containing only whitespace in the input file are ignored.

The file may contain comments, and these are ignored. A comment begins with the first # character on a line and extends to the end of the line. Note that # appearing in a textual label is treated as the beginning of a comment.

Each non-blank line in the input file that is not a comment is considered a statement. A statement must be completed on a single line, and only one statement may appear on a line.

The delimiter in the input file is whitespace---specifically one or more space and/or tab characters.

rwpmapbuild supports five types of statements. Four of those statements begin with a specific keyword: one of mode, map-name, label, and default. Any line that does not begin with with a keyword is expected to contain a range definition, which maps a range to a label. The format of the range definition depends on the mode.

The four statement types that begin with a keyword are all optional. They are:

mode { ipv4 | ipv6 | proto-port | ip }

Specify what types of ranges are defined in the file. The mode statement must appear before any ranges are specified. The mode may also be set using the --mode command line switch. When both the mode statement and the --mode switch are given, their values must match. When neither the mode statement nor the --mode switch is provided, rwpmapbuild processes the input in IPv4 address mode. The ip mode is deprecated; it is an alias for ipv4. The mode statement may only appear one time.

map-name simple-string

Create a name for the data in this prefix map file that other SiLK tools may use to refer to this prefix map file. When the prefix map file is used by rwfilter(1), the simple-string is used to generate the filtering switch names. When the prefix map file is used by rwcut(1), rwgroup(1), rwsort(1), rwstats(1), or rwuniq(1), the simple-string is used to generate the field names. See pmapfilter(3) for details. The simple-string may not contain whitespace, a comma, or a colon. The map-name statement may only appear one time.

label num label-text

Associate the numeric identifier num with the given label text label-text. By specifying a label statement, the identifier num is expanded to label-text when the range definitions are being defined.

Either all labels used in the file must appear in label statements, or no label statements may appear in which case rwpmapbuild creates labels as it parses the range definitions. All label statements must appear before the default statement and before the range definitions.

label-text is a textual string that begins at the first non-whitespace character and extends to the final non-whitespace character on that line that does not appear in a comment. The label-text may include embedded whitespace and non-alphanumeric characters. While a comma (,) is legal in the label-text, using a comma prevents the label from being used by the --pmap-src and --pmap-dest switches in rwfilter(1).

If no label statements appear in the input, any text containing at least one non-whitespace character may be used as the label in the default statement and the range definitions.

It is an error if num or label-text appear in any other label statement. The minimum value for num is 0 and the maximum value is 2147483647. Note that rwpmapbuild creates labels for all numeric identifiers between 0 and the maximum identifier used in the input file, and using an unnecessarily large value creates many empty entries.

default label-value

Use the label label-value for any ranges not explicitly mentioned in this input file. The label-value text is one of

  1. when label statements are used, a numerical label identifier that was specified in one of the statements

  2. when label statements are used, a string that is an exact match of the label-text that was specified in one of those statements

  3. when label statements are not used, a string that begins at the first non-whitespace character and extends to the final non-whitespace character on that line that does not appear in a comment

The default statement must appear before the range definitions are specified. If the default statement does not appear in the input, the label UNKNOWN is automatically defined and used as the default.

As mentioned above, any line that does not begin with one of the above keywords must contain a range definition, and the format of the line depends on the type of data in the input file--that is, the mode of the input.

Regardless of the input mode, the final item in each range definition is the label to assign to that range. The label has the same form as that described for the default statement above, and the label is denoted by label-value in the following.

Address Mode

When the mode is either ipv4 or ipv6, rwpmapbuild parses the file in address mode. In address mode, each range definition contains an IP range and a label-value.

A range is either a CIDR block or a pair of IP addresses that specify the starting IP and ending IP of the range. To provide a label for a single IP address, you must either use the same IP address as the starting and ending values of the range, append /32 to a single IPv4 address, or append /128 to a single IPv6 address. When the mode is ipv4, an IPv6 address in the input file raises an error. rwpmapbuild also accepts integer representations of IP addresses when in ipv4 mode,

When ranges overlap or one range is a specialization of another, the wider or more general range should be listed first, followed by the narrower or more specific ranges.

cidr-block label-value

Associate the given label identifier or label text with this CIDR block. The CIDR block is composed of an IP address in canonical notation (e.g, dotted-decimal for IPv4), a slash /, and the number of significant bits.

low-ip high-ip label-value

Associate the given label identifier or label text with this IP range, where low-ip and high-ip are in canonical notation.

low-int high-int label-value

Treat low-int and high-int as 32-bit values, convert the values to IPv4 addresses, and associate the given label identifier or label text with the IPv4 range.

Protocol/Port Mode

When the mode is proto-port, rwpmapbuild parses the file in protocol/port mode. In protocol/port mode, each range definition contains a starting value, an ending value, and the label-value.

The starting and ending values may both be integers between 0 and 255, inclusive. The numbers are treated as protocol values, where 6 is TCP, 17 is UDP, and 1 is ICMP.

The starting and ending values may also both be a number, a slash (/), and a number. The first number is treated as the protocol and the second number as a port number (or service) in that protocol. For example, 6/80 is considered the http service of TCP.

When ranges overlap or one range is a specialization of another, the wider or more general range should be listed first, followed by the narrower or more specific ranges. That is, specify the generic protocol first, then port numbers within that protocol.

proto/port proto/port label-value

Associate the given label identifier or label text with all protocols and port numbers between these two values inclusive. Note that while port is not meaningful for all protocols (specifically, it is meaningful for TCP and UDP and may contain type/code information for ICMP), rwpmapbuild accepts port numbers for any protocol.

proto proto label-value

Associate the given label identifier or label text for all protocols between these two values inclusive.

EXAMPLE

In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash (\) is used to indicate a wrapped line.

Reading from and writing to a file:

 $ rwpmapbuild --input iana-ipv6.txt --output iana-ipv6.pmap

Reading from the standard input and writing to the standard output:

 $ cat ipv4-special.txt              \
   | rwpmapbuild > ipv4-special.pmap

For example input files, see the "DESCRIPTION" section above.

ENVIRONMENT

SILK_CLOBBER

The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SEE ALSO

pmapfilter(3), rwfilter(1), rwfileinfo(1), rwpmapcat(1), rwpmaplookup(1), rwcut(1), rwgroup(1), rwsort(1), rwstats(1), rwuniq(1), silk(7)