NAME

rwgeoip2ccmap - Create a country code prefix map from a GeoIP Legacy file

SYNOPSIS

  rwgeoip2ccmap [--input-path=PATH] [--output-path=PATH] [--dry-run]
        [--mode={[auto] [ipv4|ipv6] [csv|binary] [geoip2|legacy]}]
        [--note-add=TEXT] [--note-file-add=FILENAME]
        [--invocation-strip]

  rwgeoip2ccmap --help

  rwgeoip2ccmap --version

Legacy Synopsis

  rwgeoip2ccmap {--csv-input | --v6-csv-input | --encoded-input}
        [--input-file=PATH] [--output-file=PATH] [--dry-run]
        [--note-add=TEXT] [--note-file-add=FILENAME]
        [--invocation-strip]

DESCRIPTION

Prefix maps provide a way to map field values to string labels based on a user-defined map file. The country code prefix map, typically named country_codes.pmap, is a special prefix map that maps an IP address to a two-letter country code as defined by ISO 3166 part 1. For additional information, see https://www.iso.org/iso-3166-country-codes.html and https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2.

rwgeoip2ccmap creates the country code prefix map by reading one of the country code database files distributed by MaxMind(R) http://www.maxmind.com/. rwgeoip2ccmap supports these formats:

See the "EXAMPLES" section below for the details on how to convert these files to a SiLK country-code prefix map file.

The country code prefix map file is used to map IP addresses to country codes in various SiLK tools as documented in the ccfilter(3) man page. As a brief overview, you may

The rwpmaplookup(1) command can use the country code mapping file to display the country code for textual IP addresses.

To create a general prefix map file from textual input, use rwpmapbuild(1).

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--mode==MODE_OPTIONS

Specify the type of the input and whether rwgeoip2ccmap creates a prefix map containing IPv4 or IPv6 addresses. MODE_OPTIONS is a comma-separated list of the following values. When not specified, MODE_OPTIONS defaults to auto. Since SiLK 3.12.0; changed in SiLK 3.17.0.

auto

Determine the type of the input based on the argument to --input-path, and determine the type of prefix map to create based on the IP addresses that appear on the first line of input for a CSV file or by the depth of the tree for a binary input file. This is the default mode.

ipv6

Create an IPv6 prefix map. When reading CSV input, the IPv4 addresses are mapped into the ::ffff:0:0/96 netblock. This value may not be combined with ipv4.

ipv4

Create an IPv4 prefix map. When reading CSV input, the IPv6 addresses in the ::ffff:0:0/96 netblock are mapped to IPv4 addresses and all other IPv6 addresses are ignored. When reading GeoIP2 binary data, the IPv6 addresses in the ::0:0/96 netblock are mapped to IPv4. This value may not be combined with ipv6.

csv

Read textual input containing IP addresses in a comma separated value format. and create an IPv4 prefix map. Any IPv6 addresses in the ::ffff:0:0/96 netblock are mapped to an IPv4 address and all other IPv6 addresses are ignored. This value may not be combined with binary. Since SiLK 3.17.0.

binary

Read a MaxMind binary country code database file in the GeoIP Legacy, GeoIP2, or GeoLite2 formats. Support for the GeoIP2 formats requires that SiLK was built with libmaxminddb support. This value may not be combined with csv.

geoip2

Expect the input to be the GeoIP2 or GeoLite2 country code formats (CSV or binary). GeoIP2/GeoLite2 data may not be read from the standard input. The value may not be combined with legacy. Since SiLK 3.17.0.

legacy

Expect the input to be the GeoIP Legacy country code format (CSV or binary). This mode is enabled if the input is being read from the standard input. This value may not be combined with geoip2. Since SiLK 3.17.0.

--input-path=PATH

Read the comma-separated value (CSV) or binary forms of the GeoIP2, GeoLite2, GeoIP Legacy, or GeoLite Legacy country code database from PATH. For GeoIP2 data, the --input-path switch is required, and it must either be the location of the GeoLite2-Country.mmdb file for binary data or the directory containing the GeoLite2-Country-Blocks-IPv4.csv file for CSV data. rwgeoip2ccmap supports reading GeoIP Legacy data (either binary or CSV) from the standard input. You may use stdin or - to represent the standard input; when this switch is not provided, the input is read from the standard input unless the standard input is a terminal. rwgeoip2ccmap reads read textual input from the terminal if the standard input is explicitly specified as the input. (Added in SiLK 3.17.0 as a replacement for --input-file.)

--output-path=PATH

Write the binary country code prefix map to PATH. You may use stdout or - to represent the standard output. When this switch is not provided, the prefix map is written to the standard output unless the standard output is connected to a terminal. (Added in SiLK 3.17.0 as a replacement for --output-file.)

--dry-run

Check the syntax of the input file and do not write the output file. Since SiLK 3.12.0.

--note-add=TEXT

Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool. Since SiLK 3.12.0.

--note-file-add=FILENAME

Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation. Since SiLK 3.12.0.

--invocation-strip

Do not record the command used to create the prefix map in the output. When this switch is not given, the invocation is written to the file's header, and the invocation may be viewed with rwfileinfo(1). Since SiLK 3.12.0.

--help

Print the available options and exit.

--version

Print the version number and exit the application.

Deprecated Options

The following switches are deprecated.

--csv-input

Assume the input is the CSV GeoIP Legacy country code data for IPv4. Use --mode=ipv4,csv,legacy as the replacement. Deprecated as of SiLK 3.12.0.

--v6-csv-input

Assume the input is the CSV GeoIP Legacy country code data for IPv6. Use --mode=ipv6,csv,legacy as the replacement. Deprecated as of SiLK 3.12.0.

--encoded-input

Assume the input is the specially-encoded binary form of the GeoIP Legacy country code data for either IPv4 or IPv6. Use --mode=binary,legacy as the replacement. Deprecated as of SiLK 3.12.0.

--input-file=PATH

Read the input from PATH. An alias for --input-path. Added in SiLK 3.12.0; deprecated as of SiLK 3.17.0.

--output-file=PATH

Write the binary country code prefix map to PATH. An alias for --output-path. Added in SiLK 3.12.0; deprecated as of SiLK 3.17.0.

EXAMPLES

The following examples show how to create the country code prefix map file, country_codes.pmap, from various forms of input. Once you have created the country_codes.pmap file, you should copy it to /usr/share/silk/country_codes.pmap so that the ccfilter(3) plug-in can find it. Alternatively, you can set the SILK_COUNTRY_CODES environment variable to the location of the country_codes.pmap file.

In these examples, the dollar sign ($) represents the shell prompt. Some input lines are split over multiple lines in order to improve readability, and a backslash (\) is used to indicate such lines.

MaxMind GeoIP2 or GeoLite2 Comma Separated Values Files

Download the CSV version of the MaxMind GeoIP2 or GeoLite2 country database file, e.g., GeoLite2-Country-CSV_20180327.zip. This archive is created with the zip(1) utility and contains a directory of multiple files. Expand the archive with unzip(1):

 $ unzip GeoLite2-Country-CSV_20180327.zip
 Archive:  GeoLite2-Country-CSV_20180327.zip
   inflating: GeoLite2-Country-CSV_20180327/GeoLite2-Country-...
   ...

rwgeoip2ccmap uses three of those files:

GeoLite2-Country-Blocks-IPv4.csv

A mapping from IPv4 netblocks to a geoname_ids.

GeoLite2-Country-Blocks-IPv6.csv

A mapping from IPv6 netblocks to a geoname_ids.

GeoLite2-Country-Locations-en.csv

A mapping from geoname_ids to continent and country.

Run rwgeoip2ccmap and set --input-path to the name of the directory.

 $ rwgeoip2ccmap --input-path=GeoLite2-Country-CSV_20180327 \
        --output-path=country_codes.pmap

MaxMind GeoIP2 or GeoLite2 Binary File

Support for reading GeoIP2 binary files requires that rwgeoip2ccmap was compiled with support for the libmaxminddb library.

Download the binary version of the MaxMind GeoIP2 or GeoLite2 country database file, e.g., GeoLite2-Country_20180327.tar.gz. The file is a compressed (gzip(1)) tape archive (tar(1)). Most versions of the tar program allow you to expand the archive using

 $ tar zxf GeoLite2-Country_20180327.tar.gz

Older versions of tar may require you to invoke gzip yourself

 $ gzip -d -c GeoLite2-Country_20180327.tar.gz \
   | tar cf -

The result is a directory named GeoLite2-Country_20180327 or similar.

Run rwgeoip2ccmap and set --input-path to the name of the GeoLite2-Country.mmdb file in the directory.

 $ rwgeoip2ccmap \
        --input-path=GeoLite2-Country_20180327/GeoLite2-Country.mmdb \
        --output-path=country_codes.pmap

MaxMind Legacy IPv4 Comma Separated Values File

Download the CSV version of the MaxMind GeoIP Legacy Country database for IPv4, GeoIPCountryCSV.zip. Running unzip -l on the zip file should show a single file, GeoIPCountryWhois.csv.) To expand this file, use the unzip(1) utility.

 $ unzip GeoIPCountryCSV.zip
 Archive:  GeoIPCountryCSV.zip
   inflating: GeoIPCountryWhois.csv

Create the country_codes.pmap file by running

 $ rwgeoip2ccmap --input-path=GeoIPCountryWhois.csv \
        --output-path=country_codes.pmap

You may avoid creating the GeoIPCountryWhois.csv file by using the -p option of unzip to pass the output of unzip directly to rwgeoip2ccmap:

 $ unzip -p GeoIPCountryCSV.zip  \
   | rwgeoip2ccmap --mode=ipv4 --output-path=country_codes.pmap

MaxMind Legacy IPv6 Comma Separated Values File

If you download the IPv6 version of the MaxMind GeoIP Legacy Country database, use the following command to create the country_codes.pmap file:

 $ gzip -d -c GeoIPv6.csv.gz  \
   | rwgeoip2ccmap --mode=ipv6 > country_codes.pmap

Since the GeoIPv6.csv.gz file only contains IPv6 addresses, the resulting country_codes.pmap file will display the unknown value (--) for any IPv4 address. See the next example for a solution.

MaxMind Legacy IPv6 and IPv4 Comma Separated Values Files

To create a country_codes.pmap mapping file that supports both IPv4 and IPv6 addresses, download both of the Legacy CSV files (GeoIPv6.csv.gz and GeoIPCountryCSV.zip) from MaxMind.

You need to uncompress both files and feed the result as a single stream to the standard input of rwgeoip2ccmap. This can be done in a few commands:

 $ gzip -d GeoIPv6.csv.gz
 $ unzip GeoIPCountryCSV.zip
 $ cat GeoIPv6.csv GeoIPCountryWhois.csv  \
   | rwgeoip2ccmap --mode=ipv6 > country_codes.pmap

Alternatively, if your shell supports it, you may be able to use a subshell to avoid having to store the uncompressed data:

 $ ( gzip -d -c GeoIPv6.csv.gz ; unzip -p GeoIPCountryCSV.zip )  \
   | rwgeoip2ccmap --mode=ipv6 > country_codes.pmap

SEE ALSO

ccfilter(3), rwpmaplookup(1), rwfilter(1), rwcut(1), rwsort(1), rwstats(1), rwuniq(1), rwgroup(1), rwpmapbuild(1), rwfileinfo(1), silk(7), gzip(1), zip(1), unzip(1), https://dev.maxmind.com/geoip/geoip2/geolite2/, https://dev.maxmind.com/geoip/legacy/geolite/, https://www.iso.org/iso-3166-country-codes.html, https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2

NOTES

Support for GeoIP2 and GeoLite2 input files were added in SiLK 3.17.0.

Support for the binary form of the GeoIP Legacy format was removed in SiLK 3.12.0 and restored in SiLK 3.12.2.

MaxMind, GeoIP, and related trademarks are the trademarks of MaxMind, Inc.