NAME

ipfixDump - Print contents of an IPFIX file as human-readable text

SYNOPSIS

ipfixDump [--in FILE_NAME]
          [--out FILE_NAME]
          [--cert-element-path DIRECTORY_NAME]
          [--element-file FILE_NAME]
          [--no-cert-elements]
          [--rfc5610]
          [{ --show=SHOW_LIST | --templates | --data |
             --template-counts | --file-times }]
          [--octet-format FORMAT]
          [--string-format FORMAT]
          [--suppress=ITEM_LIST]
          [--yaf]

ipfixDump [--version]

ipfixDump [--help]

DESCRIPTION

ipfixDump is a tool to read an IPFIX stream and print its contents as human-readable text to assist the user in analyzing the file. By default, ipfixDump prints all message headers, templates, data records, options templates, and options records to the output, plus a one line summary of the file's content.

To select different output, use the --show, --templates, --data, --template-counts, or --file-times switches. It is possible to suppress parts of the default output with the --suppress option; this option's purpose is to make it easier to compare files.

ipfixDump supports IPFIX structured data in the form of basicLists, subTemplateLists, and subTemplateMultiLists.

By default, ipfixDump reads the IPFIX file from the standard input and writes the text to the standard output. To specify the input or output file's location, use the --in or --out option, respectively.

ipfixDump requires the input file to contain the IPFIX templates that describe the data records within the file, and the template must appear before the records that use it. Any records that do not have a corresponding template are ignored.

Information Model

The default information model used by ipfixDump includes the standard information elements defined by IANA provided by libfixbuf.

ipfixDump also attempts (unless the --no-cert-elements option is given) to read the CERT private enterprise information elements defined in a file named cert_ipfix.xml and exits with an error if it is unable to do so. These elements are used by the NetSA tools yaf(1), pipeline(8), super_mediator(1), and rwsilk2ipfix(1). See https://tools.netsa.cert.org/cert-ipfix-registry/ for additional information about this file.

(Prior to ipfixDump v4, ipfixDump required the --yaf option to load the CERT NetSA element file. That is no longer required, and the --yaf option is a no-op.)

ipfixDump checks several directories to locate the cert_ipfix.xml file, stopping at the first file it finds. An additional directory path may be added to the search path by using the --cert-element-path option, which may be repeated. The list of directories, in search order, is

Unless the --no-cert-elements option is given, ipfixDump exits with an error if it is unable to find the cert_ipfix.xml file.

There are two ways to augment the set of elements in ipfixDump's information model:

  1. The --rfc5610 option instructs ipfixDump to watch the input for options records that define private enterprise information elements (as defined by RFC5610) and to add those elements to the information model.

  2. The --element-file=FILE_NAME option tells ipfixDump to parse the contents of FILE_NAME and add those information elements to the information model. These elements are loaded after the standard information model and the CERT NetSA elements. The argument is an XML file whose schema is that used by IANA's XML Information Element Registry, with the following additions:

    cert:enterpriseId

    A number representing the Private Enterprise Number of the element

    cert:reversible

    A boolean value (true, yes, or 1 for true; false, no, or 0 for false) that specifies whether the element may have a separate identity in a reverse flow.

    The --element-file option may be used multiple times to load multiple files, and the loaded elements replace existing elements with the same identifier.

OPTIONS

The following options are available for ipfixDump. The complete option name must be used. For options that take an argument, you may use either --option=argument or --option argument.

Input/Output Location Options

These options specify the source and destination file names or streams.

--in FILE_NAME

Sets the input file name to FILE_NAME. When the option is not specified, ipfixDump reads from the standard input or exits with an error when the standard input is a terminal. ipfixDump reads from the standard input if FILE_NAME is '-'. Short option: -i.

--out FILE_NAME

Sets the output file name to FILE_NAME. If FILE_NAME exists, it is overwritten. The string '-' may be used to write to standard output (the default). Short option: -o.

Display Choice Options

These options choose what to print. By default, ipfixDump prints all message headers, templates, data records, options templates, and options records to the output, plus a one line summary of the file's content.

--data

Equivalent to --show=data: Tells ipfixDump to print only data records and the message headers and footers for messages that contain a data set. May not be used with --file-times, --templates, or --template-counts. Short option: -d.

--file-times

Equivalent to --show=file-times: Tells ipfixDump to print only the flowEndMilliseconds timestamps for the first and last record in the input, the minimum and maximum flowEndMilliseconds timestamps seen, the message export times of the first and last IPFIX messages, and the total number of templates and data records read. May not be used with --data, --templates, or --template-counts. Short option: -f.

--show=SHOW_LIST

Causes the output to contain only the categories specified in SHOW_LIST, a comma-separated list of the names described below. When none of --show or the other output selector options is given (--data, --file-times, --templates, --template-counts), ipfixDump defaults to --show=data,templates. This option has precedence over and clears the other display choices. Short option: -s.

When --show includes any of templates, data, messages, or flow-times, the last line of the output summarizes the number of messages, number of data records, and number of templates records contained in the input.

data

Tells ipfixDump to print the data records that were read from the input and to print the message header and footer (statistics) for messages that contain data records.

file-times

Tells ipfixDump to print, at the end of the data and after the template-count table, the flowEndMilliseconds timestamps for the first and last record in the input, the minimum and maximum flowEndMilliseconds timestamps seen, and the message export times of the first and last IPFIX messages.

messages

Tells ipfixDump to print the message header and message footer (statistics) for all messages. This can be used to see the message boundaries that would normally be suppressed when only data or template records are being displayed.

templates

Tells ipfixDump to print the template records that appear in the input and to print the message header and footer (statistics) for messages that contain template records.

template-counts

Causes ipfixDump to print, after the data and template records, a table showing the counts and sizes of templates defined and used by a record and/or sub-record. The table contains five or six columns where each row includes (1)a template ID defined in the input, (2)the number of times data using that template was seen if greater than 0, (3)the smallest sized record that used that template, (4)the average size for records that used that template, (5)the largest sized record that used that template, and (6)the template name if the input contains template description option records. The counts include uses in subTemplateLists and subTemplateMultiLists when a structured data record is present. If a list specifies a template ID but the list is empty, the count for that template ID is unchanged. When computing record sizes for structured data records, a sub-record's size is also included in its parent's size.

unused-counts

Produces output identical to template-counts except that output includes templates where column two is 0; that is, templates that were defined in the input but not used for data records or sub-records.

--suppress=ITEM_LIST

Disables the display of the items in ITEM_LIST. This may be useful when using ipfixDump to compare IPFIX files. The following items may be suppressed:

record-number

For top-level records, do not display the record number.

export-time

Do not display the export time in the message header.

stream-offset

Do not display the stream offset in the message header.

message-length

Do not display the message length in the message header.

sequence-number

Do not display the sequence number in the message header.

messages

Disables printing of the message header and message footer. The output still includes the number of messages, records, and templates seen in the file once the end of the input is reached. This overrides --show=messages.

--templates

Equivalent to --show=templates: Tells ipfixDump to print only template definitions and the message headers and footers for messages that contain a template set. May not be used with --data, --file-times, or --template-counts. Short option: -t.

--template-counts

Equivalent to --show=template-counts: Tells ipfixDump to print only a table showing the template IDs defined in the input, the number of records and sub-records that used that template (for those with counts greater than zero), and the minimum, average, and maximum sizes of records that used that template. May not be used with --data, -file-times, or --templates. Short option: -c.

Presenation Options

These options control how the output is formatted.

--octet-format=FORMAT

Sets the output for data record elements whose type is octetArray. The output always includes the length of the octetArray surrounded by square brackets (e.g., [4]) before the displayed value, if any. If the value's length is 0 or if the element is paddingOctets (IE-210), only the length is printed. The default format is short-hex. See --string-format for a description of the formats.

--string-format=FORMAT

Sets the output for data record elements whose type is string. The output always includes the octet-length of the element's data surrounded by square brackets ([4]) before the displayed value, if any. The default format is string.

With a few exceptions, the formats described here are valid for both the --octet-format and --string-format options.

ascii

Outputs a string in double quotes (") where octets from 32 (space) to 126 (tilde) inclusive are displaced as ASCII except for octet value 34 (displayed as \") and 92 (\\). All other values are displayed as a five-character sequence \\xhh where h is a single lowercase hexadecimal digit. For example, "foo\\x0a" where \\x0a represents a newline. This is the fallback format when the argument is string and the value is not valid UTF-8. When the value's length is 0, the output is [0] "".

base64

Displays the value using base-64 encoding, e.g., [4] Zm9vCg==. The value is not surrounded by quotation marks. If the value's length is 0, only the length is printed.

empty

Does not display anything other than the length. Equivalent to none.

hexadecimal

Outputs 0x and then the hexadecimal value for each octet in the octetArray or string with no delimiter or space between the values, e.g., [4] 0x666f6f0a. If the value's length is 0, only the length is printed.

hexdump

Displays the value in the style of the hexdump(1) utility when invoked with the arguments -v -C. The output displays 16 octets of the value on each line. Each line has the same prefix as that of the element name followed by a vertical bar that aligns with the element name. That is followed by the line's offset displayed in hexadecimal, up to 16 octets each presented as a hexadecimal number, a vertical bar, those 16 octets presented as ASCII with a period for any non-printable or control character, followed by another vertical bar and a newline. The final line displays the length of the value in hexadecimal. For example:

[4]
| 00000000  66 6f 6f 0a                                       |foo.            |
| 00000004
octets

Displays a string in double-quotes (") containing a space-separated list of the hexadecimal value for each octet in the value, e.g., [4] "66 6f 6f 0a". If the value's length is 0, the output is [0] "".

none

Does not display anything other than the length. Equivalent to empty.

short-hex

(Valid only for --octet-format and is its default format.) If the octetArray is a fixed-width element and its length is eight or less, displays the value in the hexadecimal format, otherwise displays nothing.

quoted

(Valid only for --string-format.) An alias for string.

raw

(Valid only for --string-format.) Displays the raw string with no surrounding quotes and making no attempt to escape control characters or unprintable characters. This mimics the behavior of ipfixDump before version 3.0.0.

string

If the value is valid UTF-8, prints the value as Unicode surrounded by quotation marks ("), where a two-character escape sequence is used for quotation mark (\"), reverse solidus (\\), tab (\t), line feed (\n), and carriage return (\r), and the six-character sequence \uHHHH is used for non-printable characters in the Basic Multilingual Plane (where H represents an uppercase hexadecimal digit). Non-printable characters outside the Basic Multilingual Plane are represented as \uHHHH\uHHHH encoding the UTF-16 surrogate pair. For example, [4] "foo\n".

An invalid UTF-8 value is printed in the ascii format. To force Unicode output, choose the unicode format.

When the value's length is 0, the output is [0] "". string is the default format for elements of type string.

unicode

Prints a valid UTF-8 encoded value using the string format. For an invalid UTF-8 value, octets that cannot be interpreted as Unicode are replaced with the Unicode replacement character, U+FFFD, and the result is printed in the string format.

Information Model Options

These options control loading of information elements in addition to those defined by IANA. By default, ipfix2json tries to load information elements from a file named cert_ipfix.xml and exits the application if it cannot. The cert_ipfix.xml file must be located to properly process YAF and Super Mediator output.

--cert-element-path DIRECTORY_NAME

Adds DIRECTORY_NAME to the search path for the CERT NetSA information element XML file, cert_ipfix.xml. This option may be repeated to add multiple directories to the search path. The directories are added in the order they appear on the command line and before the default locations. To tell ipfixDump not to load this file, use the --no-cert-elements option. Short option: -c.

--element-file FILE_NAME

Loads the XML file FILE_NAME and incorporates information element information found in it. The format of the file is described above. This option is processed after the elements defined by IANA and CERT NetSA have been loaded. The option may be used multiple times to load multiple files, and later elements replace existing elements when they have the same identifier. When reading cert_ipfix.xml with this option, you should also specify --no-cert-elements to disable the search for that file. Short option: -e.

--no-cert-elements

Tells ipfix2json not to search for the cert_ipfix.xml file. Without this option, ipfix2json exits if it cannot find the CERT NetSA information element file.

--rfc5610

Tells ipfixDump to scan the IPFIX input file for options records that define private enterprise information elements and to add the unknown elements to the information model. These options records are defined in RFC5610

--yaf

Does nothing and is provided for backward compatibility. (In previous releases, this switch was required to load the cert_ipfix.xml file.)

Help Options

--help

Causes ipfixDump to print a brief usage message to the standard output and exit. Short option: -h.

--version

Causes ipfixDump to print the version and copyright information to the standard output and exit. Short option: -V.

EXAMPLES

In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line.

$ ipfixDump --in - --out -

$ ipfixDump --in /data/ipfix.ipfix --out /data/text.txt --yaf

The ipfixDump output for a complete IPFIX message that contains a single options template record is:

--- Message Header ---
export time: 2021-02-09 19:56:53       observation domain id: 0
message length: 62                     sequence number: 0 (0)
stream offset: 0

--- Options Template Record --- tid: 53254 (0xd006), fields: 9, scope: 2 ---
  privateEnterpriseNumber              (346) <uint32>       [4] {scope}
  informationElementId                 (303) <uint16>       [2] {scope}
  informationElementDataType           (339) <uint8>        [1]
  informationElementSemantics          (344) <uint8>        [1]
  informationElementUnits              (345) <uint16>       [2]
  informationElementRangeBegin         (342) <uint64>       [8]
  informationElementRangeEnd           (343) <uint64>       [8]
  informationElementName               (341) <string>   [65535]
  informationElementDescription        (340) <string>   [65535]

*** Msg Stats: 1 Template Records ***

Here is an example record that includes a subTemplateList:

--- Data Record 1332161 --- fields: 7, tmpl: 53252 (0xd004) tombstone_record ---
  observationDomainId                 (149){s} : 0
  exportingProcessId                  (144){s} : 45982
  exporterConfiguredId           (6871/551){s} : 4646
  paddingOctets                       (210)    : [6]
  tombstoneId                    (6871/550)    : 0
  observationTimeSeconds              (322)    : 2021-02-09 20:01:53
  tombstoneAccessList            (6871/554)    : count: 1 {noneOf} tmpl: 53253 (0xd005) tombstone_access
  +--- STL Record 1 --- fields: 2, tmpl: 53253 (0xd005) tombstone_access ---
  + certToolId                     (6871/553) : 1
  + observationTimeSeconds              (322) : 2021-02-09 20:01:53

AUTHORS

Emily Sarneso and the CERT Network Situational Awareness Group Engineering Team, <http://www.cert.org/netsa>.

Bug reports may be sent directly to the Network Situational Awareness team at <netsa-help@cert.org>.

SEE ALSO

ipfix2json(1), yaf(1), yafscii(1), yafdpi(1), super_mediator(1), pipeline(8), rwsilk2ipfix(1), https://tools.netsa.cert.org/cert-ipfix-registry/, https://www.iana.org/assignments/ipfix/ipfix.xhtml