CERT
Software Assurance Secure Systems Organizational Security Coordinating Response Training
Child pages
  • Country Codes in SiLK
Skip to end of metadata
Go to start of metadata

Country codes are two-letter abbreviations assigned to IP addresses to designate the country to whom the IPs are registered. See The IANA root domain list for a list of codes.

Country codes are enabled in SiLK using p-maps. In some network analysis environments, a default country code p-map is already installed, and it is very easy to use it to look at country codes in flow data.

First, see if your system has a default country code p-map.

  1. Test the rwip2cc command. Try:
     rwip2cc --address=74.125.67.100
    
    If the command outputs "us" (this is a google.com address), then your system is configured with a default country code p-map.
  2. Or, you can look for the file named country_codes.pmap in one of the following directories (note: $SILK_PATH is the value of the SILK_PATH environment variable, if it is set):
    • $SILK_PATH/share/silk/
    • $SILK_PATH/share/
    • /usr/local/share/silk/
    • /usr/local/share/

Second, see if your system has the ccfilter plugin installed. This C plugin ("ccfilter.so") introduces two keys:

  • scc: "source country code", field 18, and;
  • dcc: "destination country code", field 19

for use in rwcut, rwsort and rwuniq. In SiLK 2.0 and higher, scc and dcc can also be called as keys in rwstats.

To check to see if you have the ccfilter plugin, try running the following rwcut command:

 echo 74.125.67.100 | rwtuc --fields=sip | rwcut --fields=sip,scc

If the command outputs the source IP and an "scc" column with "us" as the entry, then you have the ccfilter plugin installed by default.

If you have a default country code p-map and the ccfilter plugin, check out the rest of this tooltip for easily looking at country codes. If not, find out more about making your own country code p-map in the documentation for rwgeopip2ccmap.. Find out about the ccfilter plugin in the documentation for ccfilter .


Country codes in Flow Data

In some cases we want to look at country codes in the context of flow records and not just IP addresses. Try the following rwuniq command:

 rwuniq mydat.rw --fields=scc --bytes --flows --packets --sip-distinct

This should return a bandwidth study by country code for the flows in your file. In addition to the usual records-bytes-packets output, the sip-distinct flag will count the number of source IPs associated with each country code. You can also use --fields=dcc and --dip-distinct flags to look at country codes for destination addresses.


Country codes in IP Sets and Text files

In some cases, we would like to summarize country codes for a static set of IPs, for example a watch list or a suspected botnet. This list may have been compiled outside of SiLK, or it may have been compiled from flow data but across many different flow files or time bins.  The file may be a binary SiLK IP set, or a text file.

The rwip2cc command can be used to map a text file of addresses (1 per line) to country codes:

 rwip2cc --input-file=myips.txt

We can use UNIX tools to create a table of counts by country code. This command will sort the counts from most to least:

 rwip2cc --input-file=myips.txt --print-ips=0 | sort | uniq -c | sort -n -r

If you have a binary set created using rwset or rwsetbuild (myips.set), use rwsetcat to turn it to a text file first:

rwsetcat myips.set | rwip2cc --input-file=stdin --print-ips=0 | sort | uniq -c | sort -n -r

If you have a very large IP set, it will be quicker to use rwtuc and rwuniq, so that the sorting is done in binary instead of using UNIX text processing on the large list of country codes:

rwsetcat myips.set | rwtuc --fields=sip | rwuniq --fields=scc --flows --no-titles | sort -t'|' -k2nr,2

The python script CCSetSummary.py at the end of this tooltip uses the rwtuc trick along with rwstats to get some more polished output. It will take IP sets or text files as input, along with rwstats summary commands, and return country names and country code statistics in rwstats format. Usage is:

> python CCSetSummary.py <mysetfile.{txt,set}> <rwstats commands>

where <rwstats commands> includes, eg:

  • --count=N : Top N countries in the IP set
  • --threshold=N : All countries with a count greater than N IP Addresses
  • --percentage=N : All countries that comprise more than N percent of the set

The output looks like:

> python CCSetSummary.py myips.set --threshold=10

INPUT: 3209 IP Addresses for 56 Countries and 3209 Total IP Addresses
OUTPUT: Top 15 countries by IP Address (threshold 10)
Country                    | cc|   IP Addr|  %IP Addr|   cumul_%|
South Korea                | kr|      1761| 54.876909| 54.876909|
China                      | cn|       559| 17.419757| 72.296666|
United States              | us|       283|  8.818947| 81.115612|
Canada                     | ca|       224|  6.980368| 88.095980|
Spain                      | es|        67|  2.087878| 90.183858|
France                     | fr|        38|  1.184170| 91.368027|
Japan                      | jp|        35|  1.090682| 92.458710|
Poland                     | pl|        26|  0.810221| 93.268931|
Private/experimental       | --|        23|  0.716734| 93.985665|
Germany                    | de|        20|  0.623247| 94.608912|
Sweden                     | se|        12|  0.373948| 94.982861|
Russia                     | ru|        11|  0.342786| 95.325647|
Romania                    | ro|        11|  0.342786| 95.668433|
Hong Kong                  | hk|        11|  0.342786| 96.011218|
Netherlands                | nl|        10|  0.311624| 96.322842|

Because it uses rwtuc and text processing, the script can take a long time to run on a large IP set (eg. 1 million IP addresses) for SiLK versions under 2.0; you may want to use the --threshold=1 flag and write the full list of counts to a file for further processing, instead of running the python script multiple times.

CCSetSummary.py
#!/bin/env/ python
# call as eg:  python CCSetSummary.py <setname> <rwstats options>
# Do not tell rwstats to change a delimiter from the pipe symbol "|", or to suppress column headers.
# Those options will break this script

import os, sys, commands, string, atexit

def is_setfile(fname):
    a = commands.getoutput("head -1 " + fname + " | rwip2cc --input-file=stdin --print-ip=0")
    if(len(a) > 2):
        val=1
    else:
        val=0
    return(val)

def cleanup_temp():
    os.system("rm -f _tmpIPsport.txt")

def pad_name(countryname):
    l = len(countryname)
    while(l < 26):
        countryname = countryname + " "
        l = len(countryname)
    return(countryname)

def is_v20_or_higher():
    com = "rwfilter --version | head -1 | grep -o -e '[0-9]\.[0-9]\.[0-9]'"
    a = commands.getoutput(com)
    b = int(a.split(".")[0])
    return(b >= 2)

def sport_number(cc, dict):
    if(not(dict.has_key(cc))):
        dict[cc] = "Unrecognized Name"
    a = dict.keys().index(cc)
    return(a)

def country_code(spn, dict):
    if(spn < len(dict)):
        a = dict.keys()[spn]
    else:
        a = "??"
    return(a)

def get_cname(cc, dict):
    if(dict.has_key(cc)):
        a = dict[cc]
    else:
        a = dict["??"]
    return(a)

countries = {
    "ac": "Ascension Island",
    "ax": "Aland Islands",
    "af": "Afghanistan",
    "al": "Albania",
    "dz": "Algeria",
    "as": "American Samoa",
    "ad": "Andorra",
    "ao": "Angola",
    "ai": "Anguilla",
    "aq": "Antarctica",
    "ag": "Antigua & Barbuda",
    "ar": "Argentina",
    "am": "Armenia",
    "aw": "Aruba",
    "au": "Australia",
    "at": "Austria",
    "az": "Azerbaijan",
    "bs": "Bahamas",
    "bh": "Bahrain",
    "bd": "Bangladesh",
    "bb": "Barbados",
    "by": "Belarus",
    "be": "Belgium",
    "bz": "Belize",
    "bj": "Benin",
    "bm": "Bermuda",
    "bt": "Bhutan",
    "bo": "Bolivia",
    "ba": "Bosnia & Herzegowina",
    "bw": "Botswana",
    "bv": "Bouvet Island",
    "br": "Brazil",
    "io": "UK Indian Ocean Terr.",
    "bn": "Brunei",
    "bg": "Bulgaria",
    "bf": "Burkina Faso",
    "bi": "Burundi",
    "kh": "Cambodia",
    "cm": "Cameroon",
    "ca": "Canada",
    "cv": "Cape Verde",
    "ky": "Cayman Islands",
    "cf": "Central African Rep.",
    "td": "Chad",
    "cl": "Chile",
    "cn": "China",
    "cx": "Christmas Island",
    "cc": "Cocos Islands",
    "co": "Colombia",
    "km": "Comoros",
    "cg": "Congo",
    "cd": "Dem. Rep. of Congo",
    "zr": "Dem. Rep. of Congo",
    "ck": "Cook Islands",
    "cr": "Costa Rica",
    "ci": "Cote D'Ivoire",
    "hr": "Croatia",
    "cu": "Cuba",
    "cy": "Cyprus",
    "cz": "Czech Republic",
    "cs": "Czechoslovakia (Fmr)",
    "dk": "Denmark",
    "dj": "Djibouti",
    "dm": "Dominica",
    "do": "Dominican Republic",
    "tp": "East Timor",
    "ec": "Ecuador",
    "eg": "Egypt",
    "sv": "El Salvador",
    "gq": "Equatorial Guinea",
    "er": "Eritrea",
    "ee": "Estonia",
    "et": "Ethiopia",
    "fk": "Falkland Islands",
    "fo": "Faroe Islands",
    "fj": "Fiji",
    "fi": "Finland",
    "fr": "France",
    "fx": "France, Metropolitan",
    "gf": "French Guiana",
    "pf": "French Polynesia",
    "tf": "French S. Territories",
    "ga": "Gabon",
    "gm": "Gambia",
    "ge": "Georgia",
    "de": "Germany",
    "gh": "Ghana",
    "gi": "Gibraltar",
    "gr": "Greece",
    "gl": "Greenland",
    "gd": "Grenada",
    "gp": "Guadeloupe",
    "gu": "Guam",
    "gt": "Guatemala",
    "gn": "Guinea",
    "gw": "Guinea-Bissau",
    "gy": "Guyana",
    "ht": "Haiti",
    "hm": "Heard & Mcdonald Is.",
    "va": "Vatican City",
    "hn": "Honduras",
    "hk": "Hong Kong",
    "hu": "Hungary",
    "is": "Iceland",
    "in": "India",
    "id": "Indonesia",
    "ir": "Iran",
    "iq": "Iraq",
    "ie": "Ireland",
    "il": "Israel",
    "im": "Isle of Man",
    "it": "Italy",
    "je": "Jersey",
    "jm": "Jamaica",
    "jp": "Japan",
    "jo": "Jordan",
    "kz": "Kazakhstan",
    "ke": "Kenya",
    "ki": "Kiribati",
    "kp": "North Korea",
    "kr": "South Korea",
    "kw": "Kuwait",
    "kg": "Kyrgyzstan",
    "la": "Laos",
    "lv": "Latvia",
    "lb": "Lebanon",
    "ls": "Lesotho",
    "lr": "Liberia",
    "ly": "Libya",
    "li": "Liechtenstein",
    "lt": "Lithuania",
    "lu": "Luxembourg",
    "mo": "Macau",
    "mk": "Macedonia",
    "mg": "Madagascar",
    "mw": "Malawi",
    "my": "Malaysia",
    "mv": "Maldives",
    "ml": "Mali",
    "mt": "Malta",
    "mh": "Marshall Is.",
    "mq": "Martinique",
    "mr": "Mauritania",
    "mu": "Mauritius",
    "yt": "Mayotte",
    "mx": "Mexico",
    "fm": "Micronesia",
    "md": "Moldova",
    "mc": "Monaco",
    "me": "Montenegro",
    "mn": "Mongolia",
    "ms": "Montserrat",
    "ma": "Morocco",
    "mz": "Mozambique",
    "mm": "Myanmar",
    "na": "Namibia",
    "nr": "Nauru",
    "np": "Nepal",
    "nl": "Netherlands",
    "an": "Netherlands Antilles",
    "nt": "Neutral Zone",
    "nc": "New Caledonia",
    "nz": "New Zealand",
    "ni": "Nicaragua",
    "ne": "Niger",
    "ng": "Nigeria",
    "nu": "Niue",
    "nf": "Norfolk Island",
    "mp": "Northern Mariana Is.",
    "no": "Norway",
    "om": "Oman",
    "pk": "Pakistan",
    "pw": "Palau",
    "ps": "Palestinian Territory",
    "pa": "Panama",
    "pg": "Papua New Guinea",
    "py": "Paraguay",
    "pe": "Peru",
    "ph": "Philippines",
    "pn": "Pitcairn",
    "pl": "Poland",
    "pt": "Portugal",
    "pr": "Puerto Rico",
    "qa": "Qatar",
    "re": "Reunion",
    "ro": "Romania",
    "ru": "Russia",
    "rw": "Rwanda",
    "sh": "St. Helena",
    "kn": "St. Kitts and Nevis",
    "lc": "St. Lucia",
    "pm": "St. Pierre & Miquelon",
    "vc": "St. Vincent & Grenadines",
    "ws": "Samoa",
    "sm": "San Marino",
    "st": "Sao Tome & Principe",
    "sa": "Saudi Arabia",
    "sn": "Senegal",
    "sc": "Seychelles",
    "sl": "Sierra Leone",
    "sg": "Singapore",
    "sk": "Slovakia",
    "si": "Slovenia",
    "sb": "Solomon Islands",
    "so": "Somalia",
    "za": "South Africa",
    "gs": "S. Georgia & Sandwich Is.",
    "es": "Spain",
    "lk": "Sri Lanka",
    "sd": "Sudan",
    "sr": "Suriname",
    "sj": "Svalbard & Jan Mayen",
    "sz": "Swaziland",
    "se": "Sweden",
    "ch": "Switzerland",
    "sy": "Syria",
    "tw": "Taiwan",
    "tj": "Tajikistan",
    "tz": "Tanzania",
    "th": "Thailand",
    "tg": "Togo",
    "tk": "Tokelau",
    "to": "Tonga",
    "tt": "Trinidad & Tobago",
    "tn": "Tunisia",
    "tr": "Turkey",
    "tm": "Turkmenistan",
    "tc": "Turks & Caicos Is.",
    "tv": "Tuvalu",
    "ug": "Uganda",
    "ua": "Ukraine",
    "ae": "United Arab Emirates",
    "gb": "United Kingdom",
    "us": "United States",
    "um": "US Minor Outlying Is.",
    "uy": "Uruguay",
    "su": "USSR (former)",
    "uz": "Uzbekistan",
    "vu": "Vanuatu",
    "ve": "Venezuela",
    "vn": "Viet Nam",
    "vg": "Virgin Is., British",
    "vi": "Virgin Is., U.S.",
    "wf": "Wallis and Futuna",
    "eh": "Western Sahara",
    "ye": "Yemen",
    "yu": "Yugoslavia (former)",
    "zm": "Zambia",
    "zw": "Zimbabwe",
    "--": "Private/experimental",
    "a1": "Anonymous proxy",
    "a2": "Satellite provider",
    "o1": "Other",
    "??": "Unrecognized Name"
     }

atexit.register(cleanup_temp)
l = len(sys.argv)
passoptions = ""

filename = sys.argv[1]
is_delimited = 0
use_v20 = is_v20_or_higher()
j = 2
while(j < l):
    passoptions = passoptions + sys.argv[j] + " "
    if(sys.argv[j][0:5] == "--del"):
        is_delimited = 1
    j = j + 1

command = ""
if(is_setfile(filename)):
    command = command + "rwsetcat " + filename + " | "
else:
    command = command + "cat " + filename + " | "

if(use_v20):
    c2 = "rwtuc stdin --fields=sip | rwstats --field=scc "
    sedst1 = " | sed -e 's/Bins/Countries/g; s/bins/countries/g; s/scc/cc/g; "
    sedst2 = "1s/Records/IP Addresses/g; 2s/Records/IP Address/g; 3s/Records/IP Addr/g'"
    command = command + c2 + passoptions + sedst1 + sedst2
    newoutput = commands.getoutput(command)
    newoutput = newoutput.splitlines()
else:
    command = command + "rwip2cc --input-file=stdin --print-ips=0"
    o1 = commands.getoutput(command)
    o1 = o1.splitlines()
    i1 = file("_tmpIPsport.txt", "w")
    for line in o1:
        i1.write(str(sport_number(line.strip(),countries)) + "\n")
    i1.close()
    prostatement = "cat _tmpIPsport.txt | rwtuc --fields=sport | rwstats --sport "
    sedst1 = " | sed -e '1s/records/IP Addresses/g; 1s/unique keys/countries/g;"
    sedst2 = " 2s/SOURCE PORT Key/COUNTRY CODE/g; 2s/flow counts/countries/g;"
    sedst3 = " 3s/Records/IP Addr/g; 3s/sPort/cc/g'"
    command2 =  prostatement + passoptions + sedst1 + sedst2 + sedst3
    newoutput = commands.getoutput(command2)
    os.system("rm _tmpIPsport.txt")
    newoutput = newoutput.splitlines()

j = 0
for line in newoutput:
    if(j == 2):
        if(is_delimited == 0):
            if(not use_v20):
                line = "Country" + 20*" " + "|" + line.strip()
            else:
                line = "Country" + 20*" " + "| " + line.strip()
        else:
            line = "Country|" + line.strip()
    if(j > 2):
        if(not use_v20):
            keynum = int((line.split("|")[0]).strip())
            cc = country_code(keynum, countries)
        else:
            cc = (line.split("|")[0]).strip()

        if(is_delimited == 0):
            cname = pad_name(get_cname(cc,countries)) + " |"
        else:
            cname = get_cname(cc, countries) + "|"

        if(not use_v20):
            line = cname + cc + (line[line.index('|'):]).strip()
        else:
            line = cname + line
    print(line)
    j = j + 1

  • No labels