Monitoring for Large-Scale Networks

fixbuf-tools

ipfix2json

NAME

ipfix2json - Print contents of an IPFIX file as json text

SYNOPSIS

ipfix2json [--in FILE_NAME]
           [--out FILE_NAME]
           [--cert-element-path DIRECTORY_NAME]
           [--element-file FILE_NAME]
           [--no-cert-elements]
           [--rfc5610]
           [--show {data | templates} [,...]]
           [--allow-duplicates]
           [--octet-format FORMAT]
           [--string-format FORMAT]

ipfix2json [--version]

ipfix2json [--help]

DESCRIPTION

ipfix2json is a tool to read an IPFIX stream and print its contents as JSON. ipfix2json is able to print templates, data records, options templates, and options records to the output.

By default, only data records and option data records are written. Use the --show option to select other or additional output.

ipfix2json supports IPFIX structured data in the form of basicLists, subTemplateLists, and subTemplateMultiLists, though these container structures are not represented as they are in IPFIX, favoring a simpler format more conducive to the json format.

By default, ipfix2json reads the IPFIX file from the standard input and writes JSON to the standard output. To specify the input or output file's location, use the --in or --out option, respectively.

ipfix2json does not include paddingOctets elements (IE-210) in the output.

Information elements whose data type is a time stamp are printed as a string in ISO-8601 format with no T between the date and time and fractional seconds if needed, for example, "2005-09-21 13:03:56.748Z". The number of digits following the decimal (if any) indicate the precision of the timestamp (e.g., 6 digits for microsecords).

IP addresses are printed as a string in the dotted-quad format for IPv4 and colon-separated hexadecimal for IPv6.

Information elements of data type string are printed as Unicode surrounded by quotation marks if the value is valid UTF-8, where escape sequences are used for certain characters (e.g., quotation mark), control characters, and non-printable characters. For a complete description, see the documentation for the string option under --string-format in the "OPTIONS" section below.

If the value of an element of type string is invalid UTF-8, by default ipfix2json uses an ASCII representation which is described under the ascii option for --string-format. The user may force Unicode output for invalid UTF-8-encoded values with the unicode option to --string-format.

The --string-format option allows for other formats as described under "OPTIONS".

By default, elements of type octetArray (except paddingOctets) are represented as a quoted string containing a base64 encoding of the element's value. The --octet-format option may be used to change this as described under "OPTIONS".

All integer values are presented as numbers regardless of their size. Doubles are presented with a decimal point and optional exponent if needed.

In ipfix2json's output of an IPFIX data record, there is no indication of the type of a value or, for strings and octetArrays, the format being used to print the value.

ipfix2json requires the input file to contain the IPFIX templates that describe the data records within the file, and the template must appear before the records that use it. Any records that do not have a corresponding template are ignored.

Information Model

The default information model used by ipfix2json includes the standard information elements defined by IANA provided by libfixbuf.

ipfix2json also attempts (unless the --no-cert-elements option is given) to read the CERT private enterprise information elements defined in a file named cert_ipfix.xml and exits with an error if it is unable to do so. These elements are used by the NetSA tools yaf(1), pipeline(8), super_mediator(1), and rwsilk2ipfix(1). See https://tools.netsa.cert.org/cert-ipfix-registry/ for additional information about this file.

ipfix2json checks several directories to locate the cert_ipfix.xml file, stopping at the first file it finds. An additional directory path may be added to the search path by using the --cert-element-path option, which may be repeated. The list of directories, in search order, is

directories specified by --cert-element-path in the order they appear on the command line, if any
the directory ../share/fixbuf-tools relative to the directory containing the application
the fixbuf-tools subdirectory of the datadir directory specified when ipfix2json was configured (defaults to $prefix/share)
the share/fixbuf-tools subdirectory installation folder for the GLib-2 library
the fixbuf-tools subdirectory of the directories specified by the $XDG_DATA_DIRS environment variable, or /usr/local/share and /usr/share/fixbuf-tools when that variable is empty

Unless the --no-cert-elements option is given, ipfix2json exits with an error if it is unable to find the cert_ipfix.xml file.

There are two ways to augment the set of elements in ipfix2json's information model:

The --rfc5610 option instructs ipfix2json to watch the input for options records that define private enterprise information elements (as defined by RFC5610) and to add those elements to the information model.
The --element-file=FILE_NAME option tells ipfix2json to parse the contents of FILE_NAME and add those information elements to the information model. These elements are loaded after the standard information model and the CERT NetSA elements. The argument is an XML file whose schema is that used by IANA's XML Information Element Registry, with the following additions:

cert:enterpriseId

A number representing the Private Enterprise Number of the element

cert:reversible

A boolean value (true, yes, or 1 for true; false, no, or 0 for false) that specifies whether the element may have a separate identity in a reverse flow.

The --element-file option may be used multiple times to load multiple files, and the loaded elements replace existing elements with the same identifier.

OPTIONS

The following options are available for ipfix2json. The complete option name must be used. For options that take an argument, you may use either --option=argument or --option argument.

Input/Output Location Options

These options specify the source and destination file names or streams.

--in FILE_NAME: Sets the input file name to FILE_NAME. When the option is not specified, ipfix2json reads from the standard input or exits with an error when the standard input is a terminal. ipfix2json reads from the standard input if FILE_NAME is '-'. Short option: -i.
--out FILE_NAME: Sets the output file name to FILE_NAME. If FILE_NAME exists, it is overwritten. The string '-' may be used to write to standard output (the default). Short option: -o.

Display Options

These options control what is printed and how it is presented.

--allow-duplicates

Allows a JSON object to contain duplicate keys, which may cause tools processing ipfix2json's output to ignore entries or to raise warnings or errors. By default, when a template contains repeated information elements, ipfix2json prints the first instance using the element's name (for example templateId), the second instance using the name and the suffix "-2" (templateId-2), the third instance with the suffix "-3", et cetera. A similar suffix is added when record a contains multiple subTemplateLists that use the same template. Using this option causes ipfix2json to disable use of these suffixes, which may make the output more human-readable.

--show={data | templates}[,...]

Causes the output to contain only the categories specified in SHOW_LIST, a comma-separated list containing data and/or templates. When --show is not given, the default is to print data records and option data records only. Short option: -s.

data: Tells ipfix2json to print data records.
templates: Tells ipfix2json to print template records.

--octet-format FORMAT

Determines the output format of data record elements whose type is octetArray. The default format is base64. This option takes the same FORMAT argument as --string-format, described next.

--string-format FORMAT

Determines the output format of data record elements whose type is string. The default format is string.

The FORMAT argument to --string-format and --octet-format may be one of these values:

ascii

Displays octets from 32 (space) to 126 (tilde ~) inclusive as ASCII except for octet value 34 (quotation mark, displayed as \") and 92 (reverse solidus, displayed as \\). All other values are displayed as a five-character sequence \\xhh where h is a single lowercase hexadecimal digit. For example, "foo\\x0a" where \\x0A represents a newline. This is the fallback format if FORMAT is string but the value is not valid UTF-8.

base64

Displays the value using base-64 encoding, e.g., "Zm9vCg==". This is the default output format for octetArray elements.

empty

Outputs an empty string as the value, "". Equivalent to none.

hexadecimal

Outputs a string containing the hexadecimal value for each octet in the value with no delimiter or space between the values, e.g., "666f6f0a".

octets

Outputs a string containing the hexadecimal value for each octet in the value with a single space between the values, e.g., "66 6f 6f 0a".

string

If the value is valid UTF-8, displays the value as Unicode where a two-character escape sequence is used for quotation mark (U+0022, \"), reverse solidus (U+005C, \\), tab (U+0009, \t), line feed (U+000A, \n), and carriage return (U+000D, \r), and the sequence \uHHHH is used for non-printable characters in the Basic Multilingual Plane, (where H represents an uppercase hexadecimal digit). Non-printable characters outside the Basic Multilingual Plane are represented as \uHHHH\uHHHH encoding the UTF-16 surrogate pair. Example output: "foo\n".

If the value is invalid UTF-8, the ascii format is used instead. To force Unicode output, select the unicode format. string is the default format for elements of type string.

unicode

Displays a valid UTF-8 value using the string format, otherwise octets that cannot be interpreted as Unicode are replaced with the Unicode replacement character, U+FFFD, and the result is printed in the string format.

Information Model Options

These options control loading of information elements in addition to those defined by IANA. By default, ipfix2json tries to load information elements from a file named cert_ipfix.xml and exits the application if it cannot. The cert_ipfix.xml file must be located to properly process YAF and Super Mediator output.

--cert-element-path DIRECTORY_NAME: Adds DIRECTORY_NAME to the search path for the CERT NetSA information element XML file, cert_ipfix.xml. This option may be repeated to add multiple directories to the search path. The directories are added in the order they appear on the command line and before the default locations. To tell ipfix2json not to load this file, use the --no-cert-elements option. Short option: -c.
--element-file FILE_NAME: Loads the XML file FILE_NAME and incorporates information element information found in it. The format of the file is described above. This option is processed after the elements defined by IANA and CERT NetSA have been loaded. The option may be used multiple times to load multiple files, and later elements replace existing elements when they have the same identifier. When reading cert_ipfix.xml with this option, you should also specify --no-cert-elements to disable the search for that file. Short option: -e.
--no-cert-elements: Tells ipfix2json not to search for the cert_ipfix.xml file. Without this option, ipfix2json exits if it cannot find the CERT NetSA information element file.
--rfc5610: Tells ipfix2json to scan the IPFIX input file for options records that define private enterprise information elements and to add the unknown elements to the information model. These options records are defined in RFC5610

Help Options

--help: Causes ipfix2json to print a brief usage message to the standard output and exit. Short option: -h.
--version: Causes ipfix2json to print the version and copyright information to the standard output and exit. Short option: -V.

EXAMPLES

In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line.

$ ipfix2json --in - --out -

$ ipfix2json --in /data/ipfix.ipfix --out /data/text.txt

Here is a example of a DNS record. The __templateId and __templateName provide the template ID and template name (provided by the template metadata records) of the template that describes the data record. Note the use of the jq tool to format the JSON.

$ ipfix2json --in /data/ipfix.ipfix --octet-format ascii | jq
...
{
  "__templateId": "0xb100(45312)",
  "__templateName": "yaf_flow_total_rle_ip4_micro",
  "flowStartMicroseconds": "2015-09-14 18:06:49.157337Z",
  "flowEndMicroseconds": "2015-09-14 18:06:49.157337Z",
  "octetTotalCount": 125,
  "packetTotalCount": 1,
  "sourceIPv4Address": "192.168.111.223",
  "destinationIPv4Address": "192.168.111.94",
  "sourceTransportPort": 5355,
  "destinationTransportPort": 50200,
  "flowAttributes": 0,
  "protocolIdentifier": 17,
  "flowEndReason": 1,
  "silkAppLabel": 53,
  "vlanId": 0,
  "ipClassOfService": 0,
  "subTemplateMultiList": [
    {
      "__templateId": "0xce00(52736)",
      "__templateName": "yaf_dns",
      "subTemplateList": [
        {
          "__templateId": "0xcf00(52992)",
          "__templateName": "yaf_dns_qr",
          "subTemplateList": [
            {
              "__templateId": "0xce06(52742)",
              "__templateName": "yaf_dns_ptr",
              "dnsPTRDName": "DIVDHQ005."
            }
          ],
          "dnsQName": "223.111.168.192.in-addr.arpa.",
          "dnsTTL": 30,
          "dnsQRType": 12,
          "dnsQueryResponse": 1,
          "dnsAuthoritative": 0,
          "dnsNXDomain": 0,
          "dnsRRSection": 1,
          "dnsID": 37364
        }
      ]
    }
  ]
}

This is ipfix2json output for the template used by the above top-level record:

$ ipfix2json --in /data/ipfix.ipfix --show templates | grep 0xb100 | jq
{
  "templateId": "0xb100(45312)",
  "templateName": "yaf_flow_total_rle_ip4_micro",
  "templateFields": [
    {
      "informationElementName": "flowStartMicroseconds",
      "informationElementId": 154,
      "templateFieldLength": 8,
      "informationElementDataType": "dateTimeMicroseconds",
      "informationElementUnits": "microseconds"
    },
    {
      "informationElementName": "flowEndMicroseconds",
      "informationElementId": 155,
      "templateFieldLength": 8,
      "informationElementDataType": "dateTimeMicroseconds",
      "informationElementUnits": "microseconds"
    },
    {
      "informationElementName": "octetTotalCount",
      "informationElementId": 85,
      "templateFieldLength": 4,
      "informationElementDataType": "unsigned64",
      "informationElementUnits": "octets"
    },
    {
      "informationElementName": "packetTotalCount",
      "informationElementId": 86,
      "templateFieldLength": 4,
      "informationElementDataType": "unsigned64",
      "informationElementUnits": "packets"
    },
    {
      "informationElementName": "sourceIPv4Address",
      "informationElementId": 8,
      "templateFieldLength": 4,
      "informationElementDataType": "ipv4Address"
    },
    {
      "informationElementName": "destinationIPv4Address",
      "informationElementId": 12,
      "templateFieldLength": 4,
      "informationElementDataType": "ipv4Address"
    },
    {
      "informationElementName": "sourceTransportPort",
      "informationElementId": 7,
      "templateFieldLength": 2,
      "informationElementDataType": "unsigned16"
    },
    {
      "informationElementName": "destinationTransportPort",
      "informationElementId": 11,
      "templateFieldLength": 2,
      "informationElementDataType": "unsigned16"
    },
    {
      "informationElementName": "flowAttributes",
      "informationElementId": 40,
      "privateEnterpriseNumber": 6871,
      "templateFieldLength": 2,
      "informationElementDataType": "unsigned16"
    },
    {
      "informationElementName": "protocolIdentifier",
      "informationElementId": 4,
      "templateFieldLength": 1,
      "informationElementDataType": "unsigned8"
    },
    {
      "informationElementName": "flowEndReason",
      "informationElementId": 136,
      "templateFieldLength": 1,
      "informationElementDataType": "unsigned8"
    },
    {
      "informationElementName": "silkAppLabel",
      "informationElementId": 33,
      "privateEnterpriseNumber": 6871,
      "templateFieldLength": 2,
      "informationElementDataType": "unsigned16"
    },
    {
      "informationElementName": "vlanId",
      "informationElementId": 58,
      "templateFieldLength": 2,
      "informationElementDataType": "unsigned16"
    },
    {
      "informationElementName": "ipClassOfService",
      "informationElementId": 5,
      "templateFieldLength": 1,
      "informationElementDataType": "unsigned8"
    },
    {
      "informationElementName": "subTemplateMultiList",
      "informationElementId": 293,
      "templateFieldLength": 65535,
      "informationElementDataType": "subTemplateMultiList"
    }
  ]
}

AUTHORS

Emily Sarneso, Matt Coates and the CERT Network Situational Awareness Group Engineering Team, <http://www.cert.org/netsa>.

Bug reports may be sent directly to the Network Situational Awareness team at <netsa-help@cert.org>.