Monitoring for Large-Scale Networks

SiLK

rwpmaplookup

NAME

rwpmaplookup - Map keys to prefix map entries

SYNOPSIS

rwpmaplookup { --map-file=MAP_FILE | --address-types[=MAP_FILE]
               | --country-codes[=MAP_FILE] }
      [--fields=FIELDS] [--ipset-files] [--no-errors]
      [--ip-format=FORMAT] [--integer-ips] [--zero-pad-ips]
      [--no-titles] [--no-columns] [--column-separator=CHAR]
      [--no-final-delimiter] [{--delimited | --delimited=CHAR}]
      [{--output-path=PATH | --pager=PAGER_PROG}]
      [--no-files ARG [ARGS...] | --xargs[=FILE] | FILE [FILES...]]

rwpmaplookup --help

rwpmaplookup --version

DESCRIPTION

rwpmaplookup finds keys in a binary prefix map file and prints the key and its value in a textual, bar (|) delimited format.

By default, rwpmaplookup expects its arguments to be the names of text files containing keys---one key per line. When the --ipset-files switch is given, rwpmaplookup takes IPset files as arguments and uses the IPs as the keys. The --no-files switch causes rwpmaplookup to treat each command line argument itself as a key to find in the prefix map.

When --no-files is not specified, rwpmaplookup reads the keys from the files named on the command line or from the standard input when no file names are specified and neither --xargs nor --no-files is present. To read the standard input in addition to the named files, use - or stdin as a file name. When the --xargs switch is provided, rwpmaplookup reads the names of the files to process from the named text file or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line.

You must tell rwpmaplookup the prefix map to use for look-ups using one of three switches:

To use an arbitrary prefix map, use the --map-file switch.
If you want to map IP addresses to country codes (see ccfilter(3)), use the --country-codes switch. To use the default country code prefix map, do not provide an argument to the switch. To use a specific country code mapping file, specify the file as the argument.
If you want to map IP addresses to address types (see addrtype(3)), use the --address-types switch. To use the default address types prefix map, do not provide an argument to the switch. To use a specific address types mapping file, specify the file as the argument.

If the --map-file switch specifies a prefix map containing protocol/port pairs, each input file should contain one protocol/port pair per line in the form PROTOCOL/PORT, where PROTOCOL is a number between 0 and 255 inclusive, and PORT is a number between 0 and 65535 inclusive. When the --ipset-files switch is specified, it is an error if the --map-file switch specifies a prefix map containing protocol/port pairs.

When querying any other type of prefix map and the --ipset-files switch is not present, each textual input file should contain one IP address per line, where the IP is a single IP address (not a CIDR block) in canonical form or the integer representation of an IPv4 address.

The --fields switch allows you to specify which columns appear in the output. The default columns are the key and the value, where the key is the IP address or protocol/port pair, and the value is the textual label for that key.

If the prefix map contains IPv6 addresses, any IPv4 address in the input is mapped into the ::ffff:0:0/96 netblock when searching.

If the prefix map contains IPv4 addresses only, any IPv6 address in the ::ffff:0:0/96 netblock is converted to IPv4 when searching. Any other IPv6 address is ignored, and it is not printed in the output unless the input field is requested.

Prefix map files are created by the rwpmapbuild(1) and rwgeoip2ccmap(1) utilities. IPset files are created most often by rwset(1) and rwsetbuild(1).

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

One of --map-file, --address-types, or --country-codes is required.

--map-file=PMAP_FILE

Find the IP addresses or protocol/port pairs in the prefix map file PMAP_FILE.

--address-types

Find the IP addresses in the address types (see addrtype(3)) mapping file specified by the SILK_ADDRESS_TYPES environment variable, or in the default address types mapping file if that environment variable is not set.

--address-types=ADDRTYPE_FILE

Find the IP addresses in the address types mapping file specified by ADDRTYPE_FILE.

--country-codes

Find the IP addresses in the country code (see ccfilter(3)) mapping file specified by the SILK_COUNTRY_CODES environment variable, or in the default country code mapping file if that environment variable is not set.

--country-codes=COUNTRY_CODE_FILE

Find the IP addresses in the country code mapping file specified by COUNTRY_CODE_FILE.

--fields=FIELDS

Specify the columns to include in the output. The columns are displayed in the order the fields are specified. FIELDS is a comma separated list of field-names. Field-names are case-insensitive. When this switch is not provided, the default fields are key,value. The list of available fields are:

key: The key used to search the prefix map.
value: The label returned from the prefix map for the key.
block: The block in the prefix map that contains the key. For a prefix map file that contains IPv4 addresses, the result will be a CIDR block such as 10.18.26.32/27.
start-block: The value at the start of the block in the prefix map that contains the key.
end-block: The value at the end of the block in the prefix map that contains the key.
input: The text read from the input file that rwpmaplookup attempted to parse. Note that blank lines, lines containing only whitespace and comments, and lines longer than 2048 characters will not be printed. In addition, any comments appearing after the text are stripped. When --ipset-files is specified, this field contains the IP address in its canonical form.

--no-files

Causes rwpmaplookup to treat the command line arguments as the text to be parsed. This allows one to look up a handful of values without having to create a temporary file. Use of the --no-files switch disables paging of the output. This switch may not be combined with --ipset-files.

--no-errors

Disables printing of errors when the input cannot be parsed as an IP address or a protocol/port pair. This switch is ignored when --ipset-files is specified.

--ipset-files

Causes rwpmaplookup to treat the command line arguments as the names of IPset files to read and use as keys into the prefix map. It is an error to use this switch when --map-file specifies a protocol/port prefix map. When --ipset-files is active, the input column of --fields contains the IP in its canonical form, regardless of the --ip-format switch. This switch may not be combined with --no-files.

--ip-format=FORMAT

When printing the key of a prefix map containing IP addresses, specify how IP addresses are printed, where FORMAT is a comma-separated list of the arguments described below. When this switch is not specified, the SILK_IP_FORMAT environment variable is checked for a value and that format is used if it is valid. The default FORMAT is canonical according to whether the prefix map file is IPv4 or IPv6. Since SiLK 3.7.0.

canonical: Print IP addresses in the canonical format. For an IPv4 prefix map, use dot-separated decimal (192.0.2.1). For an IPv6 prefix map, use colon-separated hexadecimal (2001:db8::1) or a mixed IPv4-IPv6 representation for IPv4-mapped IPv6 addresses (the ::ffff:0:0/96 netblock, e.g., ::ffff:192.0.2.1) and IPv4-compatible IPv6 addresses (the ::/96 netblock other than ::/127, e.g., ::192.0.2.1).
no-mixed: Print IP addresses in the canonical format (192.0.2.1 or 2001:db8::1) but do not used the mixed IPv4-IPv6 representations. For example, use ::ffff:c000:201 instead of ::ffff:192.0.2.1. Since SiLK 3.17.0.
decimal: Print IP addresses as integers in decimal format. For example, print 192.0.2.1 and 2001:db8::1 as 3221225985 and 42540766411282592856903984951653826561, respectively.
hexadecimal: Print IP addresses as integers in hexadecimal format. For example, print 192.0.2.1 and 2001:db8::1 as c00000201 and 20010db8000000000000000000000001, respectively. Note: This setting does not apply to CIDR prefix values which are printed as decimal.
zero-padded: Make all IP address strings contain the same number of characters by padding numbers with leading zeros. For example, print 192.0.2.1 and 2001:db8::1 as 192.000.002.001 and 2001:0db8:0000:0000:0000:0000:0000:0001, respectively. For IPv6 addresses, this setting implies no-mixed, so that ::ffff:192.0.2.1 is printed as 0000:0000:0000:0000:0000:ffff:c000:0201. As of SiLK 3.17.0, may be combined with any of the above, including decimal and hexadecimal. As of SiLK 3.18.0, the values of CIDR prefix are also zero-padded.

The following arguments modify certain IP addresses prior to printing. These arguments may be combined with the above formats.

map-v4: When the prefix map contains only IPv4 addresses, change all IPv4 addresses to IPv4-mapped IPv6 addresses (addresses in the ::ffff:0:0/96 netblock) prior to formatting. Since SiLK 3.17.0.
unmap-v6: When the prefix map contains IPv6 addresses, change any IPv4-mapped IPv6 addresses (addresses in the ::ffff:0:0/96 netblock) to IPv4 addresses prior to formatting. Since SiLK 3.17.0.

The following argument is also available:

force-ipv6: Set FORMAT to map-v4,no-mixed.

--integer-ips

Print IP addresses as integers. This switch is equivalent to --ip-format=decimal, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--zero-pad-ips

Print IP addresses as fully-expanded, zero-padded values in their canonical form. This switch is equivalent to --ip-format=zero-padded, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--no-titles

Turn off column titles. By default, titles are printed.

--no-columns

Disable fixed-width columnar output.

--column-separator=C

Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.

--no-final-delimiter

Do not print the column separator after the final column. Normally a delimiter is printed.

--delimited

--delimited=C

Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

--output-path=PATH

Write the textual output to PATH, where PATH is a filename, a named pipe, the keyword stderr to write the output to the standard error, or the keyword stdout or - to write the output to the standard output (and bypass the paging program). If PATH names an existing file, rwpmaplookup exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this option is not given, the output is either sent to the pager or written to the standard output.

--pager=PAGER_PROG

When the --no-files switch has not been specified and output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the --output-path switch is given or if the value of the pager is determined to be the empty string, no paging is performed and all output is written to the terminal.

--xargs

--xargs=FILENAME

Read the names of the input files from FILENAME or from the standard input if FILENAME is not provided. The input is expected to have one filename per line. rwpmaplookup opens each named file in turn and reads the IPset, the textual IP addresses, or the textual protocol/port pairs from it as if the filenames had been listed on the command line.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash (\) is used to indicate a wrapped line.

Country code examples

Print the country code for a list of addresses read from the standard input.

$ cat my-addrs.txt
128.2.0.0
128.2.0.1
$ cat my-addrs.txt | rwpmaplookup --country-codes
            key|               value|
      128.2.0.0|                  us|
      128.2.0.1|                  us|

Use --no-files to list the address on the command line.

$ rwpmaplookup --country-codes  128.2.0.0 128.2.0.1
            key|               value|
      128.2.0.0|                  us|
      128.2.0.1|                  us|

Use --ipset-files to read the addresses from an IPset file.

$ rwsetbuild my-addrs.txt my-addrs.set
$ rwpmaplookup --country-codes --ipset-files my-addrs.set
            key|               value|
      128.2.0.0|                  us|
      128.2.0.1|                  us|

Use the --fields switch to control which columns are printed.

$ rwpmaplookup --country-codes --fields=value my-addrs.txt
               value|
                  us|
                  us|

Add the --delimited and --no-titles switches so the output only contains the value column. Print the country code for a single address using the default country code prefix map.

$ rwpmaplookup --country-codes --fields=value --delimited \
       --no-titles --no-files 128.2.0.0
us

Alternatively

$ echo 128.2.0.0   \
  | rwpmaplookup --country-codes --fields=value --delim --no-title
us

To use a different country code mapping file, provide that file as the argument to the --country-codes switch.

$ rwpmaplookup --country-code=old-address-map.pmap --no-files 128.2.0.0
          key|value|
    128.2.0.0|   us|

CIDR block input

Note that rwpmaplookup does not parse text that contains CIDR blocks.

$ echo '128.2.0.0/31'      \
  | rwpmaplookup --country-codes
            key|value|
rwpmaplookup: Invalid IP '128.2.0.1/31' at -:1: Extra text follows value

For this case, use the IPset tool rwsetbuild(1) to parse the CIDR block list and create a binary IPset stream, and pipe the IPset to rwpmaplookup.

$ echo '128.2.0.0/31'      \
  | rwsetbuild             \
  | rwpmaplookup --country-code --ipset-files
            key|value|
      128.2.0.0|   --|
      128.2.0.1|   --|

For versions of rwpmaplookup that do not have the --ipset-files switch, you can have rwsetcat(1) read the binary IPset stream and print the IP addresses as text, and pipe that into rwpmaplookup. Be sure to include the --cidr-blocks=0 switch to rwsetcat which forces individual IP addresses to be printed.

$ echo '128.2.0.0/31'              \
  | rwsetbuild                     \
  | rwsetcat --cidr-blocks=0       \
  | rwpmaplookup --country-code
            key|value|
      128.2.0.0|   --|
      128.2.0.1|   --|

General prefix map usage

Consider a user-defined prefix map, assigned-slash-8s.pmap, that maps each /8 in the IPv4 address space to its assignment.

$ rwpmapcat assigned-slash-8s.pmap | head -4
           ipBlock|                                         label|
         0.0.0.0/8|                   IANA - Local Identification|
         1.0.0.0/8|                                         APNIC|
         2.0.0.0/8|                                      RIPE NCC|

Use the --map-file switch to map from IPs to labels using this prefix map.

$ cat my-addrs.txt
17.17.17.17
9.9.9.9
$ cat my-addrs.txt | rwpmaplookup --map-file=assigned-slash-8s.pmap
            key|               value|
    17.17.17.17| Apple Computer Inc.|
        9.9.9.9|                 IBM|

Use --ip-format=decimal to print the output as integers.

$ cat my-addrs.txt         \
  | rwpmaplookup --ip-format=decimal --map-file=assigned-slash-8s.pmap
       key|               value|
 286331153| Apple Computer Inc.|
 151587081|                 IBM|

Add the input field to see the input as well.

$ cat my-addrs.txt         \
  | rwpmaplookup --ip-format=decimal --fields=key,value,input \
       --map-file=assigned-slash-8s.pmap
       key|               value|               input|
 286331153| Apple Computer Inc.|         17.17.17.17|
 151587081|                 IBM|             9.9.9.9|

Combine the input field with the --no-errors switch to see a row for each key.

$ rwpmaplookup --fields=key,value,input --no-errors --no-files \
       --map-file=assigned-slash-8s.pmap 9.9.9.9 17.1717.17
            key|               value|               input|
        9.9.9.9| Apple Computer Inc.|             9.9.9.9|
               |                    |          17.1717.17|

The input can contain integer values.

$ echo 151587081           \
  | rwpmaplookup --fields=key,value,input --delimited=, \
       --map-file=assigned-slash-8s.pmap
key,value,input
9.9.9.9,IBM,151587081

Block output

Specifying block in the --fields switch causes rwpmaplookup to print the CIDR block that contains the address key.

$ cat my-addrs.txt
9.8.7.6
9.10.11.12
17.16.15.14
17.18.19.20
$ rwpmaplookup --map-file=assigned-slash-8s.pmap \
       --fields=key,value,block my-addrs.txt
            key|               value|             block|
        9.8.7.6|                 IBM|         9.0.0.0/8|
     9.10.11.12|                 IBM|         9.0.0.0/8|
    17.16.15.14| Apple Computer Inc.|        17.0.0.0/8|
    17.18.19.20| Apple Computer Inc.|        17.0.0.0/8|

To break the CIDR block into its starting and ending value, specify the start-block and end-block fields.

$ rwpmaplookup --map-file=assigned-slash-8s.pmap               \
       --fields=key,value,start-block,end-block my-addrs.txt
            key|               value|    start-block|      end-block|
        9.8.7.6|                 IBM|        9.0.0.0|  9.255.255.255|
     9.10.11.12|                 IBM|        9.0.0.0|  9.255.255.255|
    17.16.15.14| Apple Computer Inc.|       17.0.0.0| 17.255.255.255|
    17.18.19.20| Apple Computer Inc.|       17.0.0.0| 17.255.255.255|

To get a unique list of blocks for the input keys, do not output the key field and pipe the output of rwpmaplookup to the uniq(1) command. (This works as long as the input data is sorted).

$ cat my-addrs.txt                                 \
  | rwpmaplookup --map-file=assigned-slash-8s.pmap \
       --fields=block,value                        \
  | uniq
             block|               value|
         9.0.0.0/8|                 IBM|
        17.0.0.0/8| Apple Computer Inc.|

The values printed in the block column corresponds to the CIDR block that were used when the prefix map file was created.

$ rwpmaplookup --map=assigned-slash-8s.pmap --fields=block,value   \
       --no-files 128.2.0.1 129.0.0.1
             block|               value|
       128.0.0.0/8|Administered by ARIN|
       129.0.0.0/8|Administered by ARIN|

In the output from rwpmapcat(1), those two blocks are combined into a larger range.

$ rwpmapcat --map=assigned-slash-8s.pmap | grep 128
       128.0.0.0/6|Administered by ARIN|

Working with IPsets

Assume you have a binary IPset file, my-ips.set, that has the contents shown here, and you want to find the list of unique assignments from the assigned-slash-8s.pmap file.

$ rwsetcat --cidr-blocks=1 my-ips.set
9.9.9.0/24
13.13.13.0/24
15.15.15.0/24
16.16.16.0/24
17.17.17.0/24
18.18.18.0/24

Since the blocks in the assigned-slash-8s.pmap file are /8, use the rwsettool(1) command to mask the IPs in the IPset to the unique /8 that contains each of the IPs.

$ rwsettool --mask=8 my-ips.set    \
  | rwpmaplookup --map-file=assigned-slash-8s.pmap
           key|                        value|
       9.0.0.0|                          IBM|
      13.0.0.0|            Xerox Corporation|
      15.0.0.0|      Hewlett-Packard Company|
      16.0.0.0|Digital Equipment Corporation|
      17.0.0.0|          Apple Computer Inc.|
      18.0.0.0|                          MIT|

Protocol/port prefix maps

Assume the service.pmap prefix map file maps protocol/port pairs to the name of the service running on the named port.

$ rwpmapcat service.pmap
startPair|  endPair|    label|
      0/0|  0/65535|  unknown|
      1/0|  1/65535|     ICMP|
      2/0|  5/65535|  unknown|
      6/0|     6/21|      TCP|
     6/22|     6/22|  TCP/SSH|
...
     17/0|    17/52|      UDP|
    17/53|    17/53|  UDP/DNS|
...

To query this prefix map, the input must contain two numbers separated by a slash.

$ rwpmaplookup --map-file=service.pmap --no-files 6/80
      key|    value|
     6/80| TCP/HTTP|

Specifying block, start-block, and end-block in the --fields switch also works for Protocol/port prefix map files. The block column contains the same information as the start-block and end-block columns separated by a single space.

$ rwpmaplookup --map-file=service.pmap --no-files  \
       --fields=key,value,start,end,block          \
       6/80 6/6000 17/0 17/53 128/128
      key|     value|start-blo|end-block|              block|
     6/80|  TCP/HTTP|     6/80|     6/80|          6/80 6/80|
   6/6000|       TCP|   6/4096|   6/6143|      6/4096 6/6143|
     17/0|       UDP|     17/0|    17/31|         17/0 17/31|
    17/53|   UDP/DNS|    17/53|    17/53|        17/53 17/53|
  200/200|Unassigned|    192/0|223/65535|    192/0 223/65535|

Using the pmapfilter(3) plug-in to rwcut(1), you can print the label for the source port and destination port in the SiLK Flow file data.rw.

$ rwcut --pmap-file=service.pmap --num-rec=5       \
       --fields=proto,sport,src-service,dport,dst-service data.rw
pro|sPort|src-service|dPort|dst-service|
 17|29617|        UDP|   53|    UDP/DNS|
 17|   53|    UDP/DNS|29617|        UDP|
  6|29618|        TCP|   22|    TCP/SSH|
  6|   22|    TCP/SSH|29618|        TCP|
  1|    0|       ICMP|  771|       ICMP|

The pmapfilter plug-in does not provide a way to print the values based on the application field. You can get that information by having rwcut print the protocol and application separated by a slash, and pipe the result into rwpmaplookup.

$ rwcut --fields=proto,application --num-rec=5     \
       --delimited=/ --no-title                    \
  | rwpmaplookup --map-file=service.pmap
      key|    value|
    17/53|  UDP/DNS|
    17/53|  UDP/DNS|
     6/22|  TCP/SSH|
     6/22|  TCP/SSH|
      1/0|     ICMP|

ENVIRONMENT

SILK_IP_FORMAT: This environment variable is used as the value for --ip-format when that switch is not provided. Since SiLK 3.11.0.
SILK_PAGER: When set to a non-empty string, rwpmaplookup automatically invokes this program to display its output a screen at a time unless the --no-files switch is given. If this variable is set to an empty string, rwpmaplookup does not automatically page its output.
PAGER: When set and SILK_PAGER is not set, rwpmaplookup automatically invokes this program to display its output a screen at a time.
SILK_COUNTRY_CODES: This environment variable allows the user to specify the country code mapping file to use when the --country-codes switch is specified without an argument. The variable's value may be a complete path or a file relative to SILK_PATH. See the "FILES" section for standard locations of this file.
SILK_ADDRESS_TYPES: This environment variable allows the user to specify the address type mapping file to use when the --address-types switch is specified without an argument. The variable's value may be a complete path or a file relative to the SILK_PATH. See the "FILES" section for standard locations of this file.
SILK_CLOBBER: The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.
SILK_PATH: This environment variable gives the root of the install tree. When searching for configuration files, rwpmaplookup may use this environment variable. See the "FILES" section for details.

FILES

${SILK_COUNTRY_CODES}
${SILK_PATH}/share/silk/country_codes.pmap
${SILK_PATH}/share/country_codes.pmap
/usr/share/silk/country_codes.pmap
/usr/share/country_codes.pmap: Possible locations for the country codes mapping file when the --country-codes switch is specified without an argument.
${SILK_ADDRESS_TYPES}
${SILK_PATH}/share/silk/address_types.pmap
${SILK_PATH}/share/address_types.pmap
/usr/share/silk/address_types.pmap
/usr/share/address_types.pmap: Possible locations for the address types mapping file when the --address-types switch is specified without an argument.

NOTES

rwpmaplookup was added in SiLK 3.0.

rwpmaplookup duplicates the functionality of rwip2cc(1). rwip2cc is deprecated, and it will be removed in the SiLK 4.0 release. Examples of using rwpmaplookup in place of rwip2cc are provided in the latter's manual page.