NAME

rwpmaplookup - Map keys to prefix map entries

SYNOPSIS

  rwpmaplookup { --map-file=MAP_FILE | --address-types[=MAP_FILE]
                 | --country-codes[=MAP_FILE] }
        [--fields=FIELDS] [--ipset-files] [--no-errors]
        [--ip-format=FORMAT] [--integer-ips] [--zero-pad-ips]
        [--no-titles] [--no-columns] [--column-separator=CHAR]
        [--no-final-delimiter] [{--delimited | --delimited=CHAR}]
        [{--output-path=PATH | --pager=PAGER_PROG}]
        [--no-files ARG [ARGS...] | --xargs[=FILE] | FILE [FILES...]]

  rwpmaplookup --help

  rwpmaplookup --version

DESCRIPTION

rwpmaplookup finds keys in a binary prefix map file and prints the key and its value in a textual, bar (|) delimited format.

By default, rwpmaplookup expects its arguments to be the names of text files containing keys---one key per line. When the --ipset-files switch is given, rwpmaplookup takes IPset files as arguments and uses the IPs as the keys. The --no-files switch causes rwpmaplookup to treat each command line argument itself as a key to find in the prefix map.

When --no-files is not specified, rwpmaplookup reads the keys from the files named on the command line or from the standard input when no file names are specified and neither --xargs nor --no-files is present. To read the standard input in addition to the named files, use - or stdin as a file name. When the --xargs switch is provided, rwpmaplookup reads the names of the files to process from the named text file or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line.

You must tell rwpmaplookup the prefix map to use for look-ups using one of three switches:

If the --map-file switch specifies a prefix map containing protocol/port pairs, each input file should contain one protocol/port pair per line in the form PROTOCOL/PORT, where PROTOCOL is a number between 0 and 255 inclusive, and PORT is a number between 0 and 65535 inclusive. When the --ipset-files switch is specified, it is an error if the --map-file switch specifies a prefix map containing protocol/port pairs.

When querying any other type of prefix map and the --ipset-files switch is not present, each textual input file should contain one IP address per line, where the IP is a single IP address (not a CIDR block) in canonical form or the integer representation of an IPv4 address.

The --fields switch allows you to specify which columns appear in the output. The default columns are the key and the value, where the key is the IP address or protocol/port pair, and the value is the textual label for that key.

If the prefix map contains IPv6 addresses, any IPv4 address in the input is mapped into the ::ffff:0:0/96 netblock when searching.

If the prefix map contains IPv4 addresses only, any IPv6 address in the ::ffff:0:0/96 netblock is converted to IPv4 when searching. Any other IPv6 address is ignored, and it is not printed in the output unless the input field is requested.

Prefix map files are created by the rwpmapbuild(1) and rwgeoip2ccmap(1) utilities. IPset files are created most often by rwset(1) and rwsetbuild(1).

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

One of --map-file, --address-types, or --country-codes is required.

--map-file=PMAP_FILE

Find the IP addresses or protocol/port pairs in the prefix map file PMAP_FILE.

--address-types

Find the IP addresses in the address types (see addrtype(3)) mapping file specified by the SILK_ADDRESS_TYPES environment variable, or in the default address types mapping file if that environment variable is not set.

--address-types=ADDRTYPE_FILE

Find the IP addresses in the address types mapping file specified by ADDRTYPE_FILE.

--country-codes

Find the IP addresses in the country code (see ccfilter(3)) mapping file specified by the SILK_COUNTRY_CODES environment variable, or in the default country code mapping file if that environment variable is not set.

--country-codes=COUNTRY_CODE_FILE

Find the IP addresses in the country code mapping file specified by COUNTRY_CODE_FILE.

--fields=FIELDS

Specify the columns to include in the output. The columns are displayed in the order the fields are specified. FIELDS is a comma separated list of field-names. Field-names are case-insensitive. When this switch is not provided, the default fields are key,value. The list of available fields are:

key

The key used to search the prefix map.

value

The label returned from the prefix map for the key.

block

The block in the prefix map that contains the key. For a prefix map file that contains IPv4 addresses, the result will be a CIDR block such as 10.18.26.32/27.

start-block

The value at the start of the block in the prefix map that contains the key.

end-block

The value at the end of the block in the prefix map that contains the key.

input

The text read from the input file that rwpmaplookup attempted to parse. Note that blank lines, lines containing only whitespace and comments, and lines longer than 2048 characters will not be printed. In addition, any comments appearing after the text are stripped. When --ipset-files is specified, this field contains the IP address in its canonical form.

--no-files

Causes rwpmaplookup to treat the command line arguments as the text to be parsed. This allows one to look up a handful of values without having to create a temporary file. Use of the --no-files switch disables paging of the output. This switch may not be combined with --ipset-files.

--no-errors

Disables printing of errors when the input cannot be parsed as an IP address or a protocol/port pair. This switch is ignored when --ipset-files is specified.

--ipset-files

Causes rwpmaplookup to treat the command line arguments as the names of IPset files to read and use as keys into the prefix map. It is an error to use this switch when --map-file specifies a protocol/port prefix map. When --ipset-files is active, the input column of --fields contains the IP in its canonical form, regardless of the --ip-format switch. This switch may not be combined with --no-files.

--ip-format=FORMAT

When printing the key of an prefix map containing IP addresses, specify how IP addresses are printed. When this switch is not specified, the SILK_IP_FORMAT environment variable is checked for a format. If it is empty or contains an invalid format, IPs are printed in the canonical format. The FORMAT is one of:

canonical

Print IP addresses in their canonical form: dotted quad for IPv4 (127.0.0.1) and hexadectet for IPv6 (2001:db8::1). Note that IPv6 addresses in ::ffff:0:0/96 and some IPv6 addresses in ::/96 will be printed as a mixture of IPv6 and IPv4.

zero-padded

Print IP addresses in their canonical form, but add zeros to the output so it fully fills the width of column. The addresses 127.0.0.1 and 2001:db8::1 are printed as 127.000.000.001 and 2001:0db8:0000:0000:0000:0000:0000:0001, respectively.

decimal

Print IP addresses as integers in decimal format. The addresses 127.0.0.1 and 2001:db8::1 are printed as 2130706433 and 42540766411282592856903984951653826561, respectively.

hexadecimal

Print IP addresses as integers in hexadecimal format. The addresses 127.0.0.1 and 2001:db8::1 are printed as 7f000001 and 20010db8000000000000000000000001, respectively.

force-ipv6

Print all IP addresses in the canonical form for IPv6 without using any IPv4 notation. Any IPv4 address is mapped into the ::ffff:0:0/96 netblock. The addresses 127.0.0.1 and 2001:db8::1 are printed as ::ffff:7f00:1 and 2001:db8::1, respectively.

--integer-ips

Print IP addresses as integers. This switch is equivalent to --ip-format=decimal, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--zero-pad-ips

Print IP addresses as fully-expanded, zero-padded values in their canonical form. This switch is equivalent to --ip-format=zero-padded, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--no-titles

Turn off column titles. By default, titles are printed.

--no-columns

Disable fixed-width columnar output.

--column-separator=C

Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.

--no-final-delimiter

Do not print the column separator after the final column. Normally a delimiter is printed.

--delimited
--delimited=C

Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

--output-path=PATH

Write the textual output to PATH, where PATH is a filename, a named pipe, the keyword stderr to write the output to the standard error, or the keyword stdout or - to write the output to the standard output (and bypass the paging program). If PATH names an existing file, rwpmaplookup exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this option is not given, the output is either sent to the pager or written to the standard output.

--pager=PAGER_PROG

When the --no-files switch has not been specified and output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the --output-path switch is given or if the value of the pager is determined to be the empty string, no paging is performed and all output is written to the terminal.

--xargs
--xargs=FILENAME

Read the names of the input files from FILENAME or from the standard input if FILENAME is not provided. The input is expected to have one filename per line. rwpmaplookup opens each named file in turn and reads the IPset, the textual IP addresses, or the textual protocol/port pairs from it as if the filenames had been listed on the command line.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash (\) is used to indicate a wrapped line.

Country code examples

Print the country code for a list of addresses read from the standard input.

 $ cat my-addrs.txt
 128.2.0.0
 128.2.0.1
 $ cat my-addrs.txt | rwpmaplookup --country-codes
             key|               value|
       128.2.0.0|                  us|
       128.2.0.1|                  us|

Use --no-files to list the address on the command line.

 $ rwpmaplookup --country-codes  128.2.0.0 128.2.0.1
             key|               value|
       128.2.0.0|                  us|
       128.2.0.1|                  us|

Use --ipset-files to read the addresses from an IPset file.

 $ rwsetbuild my-addrs.txt my-addrs.set
 $ rwpmaplookup --country-codes --ipset-files my-addrs.set
             key|               value|
       128.2.0.0|                  us|
       128.2.0.1|                  us|

Use the --fields switch to control which columns are printed.

 $ rwpmaplookup --country-codes --fields=value my-addrs.txt
                value|
                   us|
                   us|

Add the --delimited and --no-titles switches so the output only contains the value column. Print the country code for a single address using the default country code prefix map.

 $ rwpmaplookup --country-codes --fields=value --delimited \
        --no-titles --no-files 128.2.0.0
 us

Alternatively

 $ echo 128.2.0.0   \
   | rwpmaplookup --country-codes --fields=value --delim --no-title
 us

To use a different country code mapping file, provide that file as the argument to the --country-codes switch.

 $ rwpmaplookup --country-code=old-address-map.pmap --no-files 128.2.0.0
           key|value|
     128.2.0.0|   us|

CIDR block input

Note that rwpmaplookup does not parse text that contains CIDR blocks.

 $ echo '128.2.0.0/31'      \
   | rwpmaplookup --country-codes
             key|value|
 rwpmaplookup: Invalid IP '128.2.0.1/31' at -:1: Extra text follows value

For this case, use the IPset tool rwsetbuild(1) to parse the CIDR block list and create a binary IPset stream, and pipe the IPset to rwpmaplookup.

 $ echo '128.2.0.0/31'      \
   | rwsetbuild             \
   | rwpmaplookup --country-code --ipset-files
             key|value|
       128.2.0.0|   --|
       128.2.0.1|   --|

For versions of rwpmaplookup that do not have the --ipset-files switch, you can have rwsetcat(1) read the binary IPset stream and print the IP addresses as text, and pipe that into rwpmaplookup. Be sure to include the --cidr-blocks=0 switch to rwsetcat which forces individual IP addresses to be printed.

 $ echo '128.2.0.0/31'              \
   | rwsetbuild                     \
   | rwsetcat --cidr-blocks=0       \
   | rwpmaplookup --country-code
             key|value|
       128.2.0.0|   --|
       128.2.0.1|   --|

General prefix map usage

Consider a user-defined prefix map, assigned-slash-8s.pmap, that maps each /8 in the IPv4 address space to its assignment.

 $ rwpmapcat assigned-slash-8s.pmap | head -4
            ipBlock|                                         label|
          0.0.0.0/8|                   IANA - Local Identification|
          1.0.0.0/8|                                         APNIC|
          2.0.0.0/8|                                      RIPE NCC|

Use the --map-file switch to map from IPs to labels using this prefix map.

 $ cat my-addrs.txt
 17.17.17.17
 9.9.9.9
 $ cat my-addrs.txt | rwpmaplookup --map-file=assigned-slash-8s.pmap
             key|               value|
     17.17.17.17| Apple Computer Inc.|
         9.9.9.9|                 IBM|

Use --ip-format=decimal to print the output as integers.

 $ cat my-addrs.txt         \
   | rwpmaplookup --ip-format=decimal --map-file=assigned-slash-8s.pmap
        key|               value|
  286331153| Apple Computer Inc.|
  151587081|                 IBM|

Add the input field to see the input as well.

 $ cat my-addrs.txt         \
   | rwpmaplookup --ip-format=decimal --fields=key,value,input \
        --map-file=assigned-slash-8s.pmap
        key|               value|               input|
  286331153| Apple Computer Inc.|         17.17.17.17|
  151587081|                 IBM|             9.9.9.9|

Combine the input field with the --no-errors switch to see a row for each key.

 $ rwpmaplookup --fields=key,value,input --no-errors --no-files \
        --map-file=assigned-slash-8s.pmap 9.9.9.9 17.1717.17
             key|               value|               input|
         9.9.9.9| Apple Computer Inc.|             9.9.9.9|
                |                    |          17.1717.17|

The input can contain integer values.

 $ echo 151587081           \
   | rwpmaplookup --fields=key,value,input --delimited=, \
        --map-file=assigned-slash-8s.pmap
 key,value,input
 9.9.9.9,IBM,151587081

Block output

Specifying block in the --fields switch causes rwpmaplookup to print the CIDR block that contains the address key.

 $ cat my-addrs.txt
 9.8.7.6
 9.10.11.12
 17.16.15.14
 17.18.19.20
 $ rwpmaplookup --map-file=assigned-slash-8s.pmap \
        --fields=key,value,block my-addrs.txt
             key|               value|             block|
         9.8.7.6|                 IBM|         9.0.0.0/8|
      9.10.11.12|                 IBM|         9.0.0.0/8|
     17.16.15.14| Apple Computer Inc.|        17.0.0.0/8|
     17.18.19.20| Apple Computer Inc.|        17.0.0.0/8|

To break the CIDR block into its starting and ending value, specify the start-block and end-block fields.

 $ rwpmaplookup --map-file=assigned-slash-8s.pmap               \
        --fields=key,value,start-block,end-block my-addrs.txt
             key|               value|    start-block|      end-block|
         9.8.7.6|                 IBM|        9.0.0.0|  9.255.255.255|
      9.10.11.12|                 IBM|        9.0.0.0|  9.255.255.255|
     17.16.15.14| Apple Computer Inc.|       17.0.0.0| 17.255.255.255|
     17.18.19.20| Apple Computer Inc.|       17.0.0.0| 17.255.255.255|

To get a unique list of blocks for the input keys, do not output the key field and pipe the output of rwpmaplookup to the uniq(1) command. (This works as long as the input data is sorted).

 $ cat my-addrs.txt                                 \
   | rwpmaplookup --map-file=assigned-slash-8s.pmap \
        --fields=block,value                        \
   | uniq
              block|               value|
          9.0.0.0/8|                 IBM|
         17.0.0.0/8| Apple Computer Inc.|

The values printed in the block column corresponds to the CIDR block that were used when the prefix map file was created.

 $ rwpmaplookup --map=assigned-slash-8s.pmap --fields=block,value   \
        --no-files 128.2.0.1 129.0.0.1
              block|               value|
        128.0.0.0/8|Administered by ARIN|
        129.0.0.0/8|Administered by ARIN|

In the output from rwpmapcat(1), those two blocks are combined into a larger range.

 $ rwpmapcat --map=assigned-slash-8s.pmap | grep 128
        128.0.0.0/6|Administered by ARIN|

Working with IPsets

Assume you have a binary IPset file, my-ips.set, that has the contents shown here, and you want to find the list of unique assignments from the assigned-slash-8s.pmap file.

 $ rwsetcat --cidr-blocks=1 my-ips.set
 9.9.9.0/24
 13.13.13.0/24
 15.15.15.0/24
 16.16.16.0/24
 17.17.17.0/24
 18.18.18.0/24

Since the blocks in the assigned-slash-8s.pmap file are /8, use the rwsettool(1) command to mask the IPs in the IPset to the unique /8 that contains each of the IPs.

 $ rwsettool --mask=8 my-ips.set    \
   | rwpmaplookup --map-file=assigned-slash-8s.pmap
            key|                        value|
        9.0.0.0|                          IBM|
       13.0.0.0|            Xerox Corporation|
       15.0.0.0|      Hewlett-Packard Company|
       16.0.0.0|Digital Equipment Corporation|
       17.0.0.0|          Apple Computer Inc.|
       18.0.0.0|                          MIT|

Protocol/port prefix maps

Assume the service.pmap prefix map file maps protocol/port pairs to the name of the service running on the named port.

 $ rwpmapcat service.pmap
 startPair|  endPair|    label|
       0/0|  0/65535|  unknown|
       1/0|  1/65535|     ICMP|
       2/0|  5/65535|  unknown|
       6/0|     6/21|      TCP|
      6/22|     6/22|  TCP/SSH|
 ...
      17/0|    17/52|      UDP|
     17/53|    17/53|  UDP/DNS|
 ...

To query this prefix map, the input must contain two numbers separated by a slash.

 $ rwpmaplookup --map-file=service.pmap --no-files 6/80
       key|    value|
      6/80| TCP/HTTP|

Specifying block, start-block, and end-block in the --fields switch also works for Protocol/port prefix map files. The block column contains the same information as the start-block and end-block columns separated by a single space.

 $ rwpmaplookup --map-file=service.pmap --no-files  \
        --fields=key,value,start,end,block          \
        6/80 6/6000 17/0 17/53 128/128
       key|     value|start-blo|end-block|              block|
      6/80|  TCP/HTTP|     6/80|     6/80|          6/80 6/80|
    6/6000|       TCP|   6/4096|   6/6143|      6/4096 6/6143|
      17/0|       UDP|     17/0|    17/31|         17/0 17/31|
     17/53|   UDP/DNS|    17/53|    17/53|        17/53 17/53|
   200/200|Unassigned|    192/0|223/65535|    192/0 223/65535|

Using the pmapfilter(3) plug-in to rwcut(1), you can print the label for the source port and destination port in the SiLK Flow file data.rw.

 $ rwcut --pmap-file=service.pmap --num-rec=5       \
        --fields=proto,sport,src-service,dport,dst-service data.rw
 pro|sPort|src-service|dPort|dst-service|
  17|29617|        UDP|   53|    UDP/DNS|
  17|   53|    UDP/DNS|29617|        UDP|
   6|29618|        TCP|   22|    TCP/SSH|
   6|   22|    TCP/SSH|29618|        TCP|
   1|    0|       ICMP|  771|       ICMP|

The pmapfilter plug-in does not provide a way to print the values based on the application field. You can get that information by having rwcut print the protocol and application separated by a slash, and pipe the result into rwpmaplookup.

 $ rwcut --fields=proto,application --num-rec=5     \
        --delimited=/ --no-title                    \
   | rwpmaplookup --map-file=service.pmap
       key|    value|
     17/53|  UDP/DNS|
     17/53|  UDP/DNS|
      6/22|  TCP/SSH|
      6/22|  TCP/SSH|
       1/0|     ICMP|

ENVIRONMENT

SILK_IP_FORMAT

This environment variable is used as the value for --ip-format when that switch is not provided. Since SiLK 3.11.0.

SILK_PAGER

When set to a non-empty string, rwpmaplookup automatically invokes this program to display its output a screen at a time unless the --no-files switch is given. If this variable is set to an empty string, rwpmaplookup does not automatically page its output.

PAGER

When set and SILK_PAGER is not set, rwpmaplookup automatically invokes this program to display its output a screen at a time.

SILK_COUNTRY_CODES

This environment variable allows the user to specify the country code mapping file to use when the --country-codes switch is specified without an argument. The variable's value may be a complete path or a file relative to SILK_PATH. See the "FILES" section for standard locations of this file.

SILK_ADDRESS_TYPES

This environment variable allows the user to specify the address type mapping file to use when the --address-types switch is specified without an argument. The variable's value may be a complete path or a file relative to the SILK_PATH. See the "FILES" section for standard locations of this file.

SILK_CLOBBER

The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SILK_PATH

This environment variable gives the root of the install tree. When searching for configuration files, rwpmaplookup may use this environment variable. See the "FILES" section for details.

FILES

${SILK_COUNTRY_CODES}
${SILK_PATH}/share/silk/country_codes.pmap
${SILK_PATH}/share/country_codes.pmap
/usr/share/silk/country_codes.pmap
/usr/share/country_codes.pmap

Possible locations for the country codes mapping file when the --country-codes switch is specified without an argument.

${SILK_ADDRESS_TYPES}
${SILK_PATH}/share/silk/address_types.pmap
${SILK_PATH}/share/address_types.pmap
/usr/share/silk/address_types.pmap
/usr/share/address_types.pmap

Possible locations for the address types mapping file when the --address-types switch is specified without an argument.

NOTES

rwpmaplookup was added in SiLK 3.0.

rwpmaplookup duplicates the functionality of rwip2cc(1). rwip2cc is deprecated, and it will be removed in the SiLK 4.0 release. Examples of using rwpmaplookup in place of rwip2cc are provided in the latter's manual page.

SEE ALSO

rwpmapbuild(1), rwpmapcat(1), ccfilter(3), addrtype(3), pmapfilter(3), rwgeoip2ccmap(1), rwcut(1), rwset(1), rwsetbuild(1), rwsetcat(1), rwsettool(1), silk(7), uniq(1)