YAF: Deep Packet Inspection

Introduction

With Deep Packet Inspection (DPI) enabled, yaf can examine packet payloads, capture useful information for a specific protocol, and export it in a protocol-specific IPFIX template. DPI in yaf is directly related to application labeling as it will only perform DPI if a match was found during the application labeling phase, and it will only execute an inspection specific to the protocol denoted by the application label (applabel).

Note: The DPI support in YAF 3 differs greatly from that in prior releases. If using a previous release of yaf, please consult the manual pages for your installation.

Required build time options: --enable-applabel --enable-dpi

Deep packet inspection support is not included in yaf by default; the --enable-applabel and --enable-dpi options must be passed to configure.

To check whether your yaf installation has DPI enabled, run yaf --version and check the setting of "Deep Packet Inspection Support".

Minimum required run time options: --dpi --max-payload

Deep packet inspection always requires payload capture to be enabled with the --max-payload option. Since DPI requires application labeling, the --dpi option will enable both features.

A minimum payload capture length of 2048 bytes is recommended, but 4096 bytes is ideal for best results (including capture of full certificate chains):

yaf --daemonize --live pcap --in eth0               \
    --out localhost --ipfix-port=18000 --ipfix tcp  \
    --dpi --max-payload=4096

DPI is enabled for all supported protocols by default, but you can also specify the dpi-select option to select on which protocols to perform DPI. The below command will perform DPI for only SMTP, DNS, and HTTP labeled flows:

yaf --daemonize --live pcap --in eth0               \
    --out localhost --ipfix-port=18000 --ipfix tcp  \
    --dpi --dpi-select=25,53,80 --max-payload=4096

DPI rules, or where to find them, are taken from a configuration file read by yaf at startup time. This is the same configuration file that yaf application labeling uses and is written in Lua. The rule file can be given on the command line with the --dpi-rules-file option, or yaf will try to read it from the default location of /usr/local/etc/yafDPIRules.conf. (The location may be different depending on how your yaf installation was built.)

yaf --daemonize --live pcap --in eth0               \
    --out localhost --ipfix-port=18000 --ipfix tcp  \
    --dpi --dpi-rules-file=*FILE* --max-payload=4096

Additional DNS Build Time Options

The default behavior of the yaf's DNS DPI support is to capture and export all DNS responses. One or both of the following options may be given to configure to limit which DNS responses yaf is capable of exporting.

Once built, yaf only supports these DNS responses. It will be necesssary to recompile and reinstall yaf to change its behavior.

When these build-time options are used, yaf still requires the --dpi and --max-payload options to export the DNS elements.

Configuration File

While both applabel and DPI-related configuration options are in the configuration file, here we are focusing only on options relevant to DPI. The syntax for application labeling is described here.

The configuration file is written in Lua. For specifics of the Lua language, see http://www.lua.org/manual/5.3/manual.html.

Comments in Lua start with double hyphens (--) and continue to the end of the line.

The file has some DPI-specific variables that may be set:

Currently the per_field_limit and per_record_limit are global settings and apply to all protocols. In a future release, we may allow configuration of these limits per protocol.

Rule Format

DPI related rule entries are ultimately optional and are only given after required applabel keys within an applabel rule if yaf supports DPI for the associated protocol.

The file must define a variable named applabels which is an array of applabel tables. Each applabel table defines a rule that tells yaf how to assign an applabel, what information elements to create when doing DPI, and how to assign values to those elements.

An applabel rule has the form

 {label=<APP>,
  label_type="<LABEL_TYPE>",
  value=[=[<EXPRESSION>]=],
  ports={<PORTS>},
  protocol=<PROTO>,
  active=<true|false>,
  <DPI-RELATED-ENTRIES>}

Here, we expand on the available options which may be given for <DPI-RELATED-ENTRIES>. (See application labeling for a desciption of the other key-value pairs.)

If yaf supports DPI for a given protocol, the associated applabel rule will include at least the following:

dpi_type="<DPI_TYPE>"

where <DPI_TYPE> is how to perform DPI. The value is a string, and the recognized values are: none, regex, plugin, and regex-plugin**. If dpi_type is not present or has the value "none", no DPI is performed. More details are in the "Assigning..." sections below.

Assigning DPI Using Regular Expressions

A "regex" dpi_type is typically used with label_type="regex" rules and has the following form:

 {label=<APP>,
  label_type="regex",
  active=true,
  protocol=<PROTO>,
  value=[=[<APP_PCRE_REGEX>]=],
  dpi_type="regex",
  dpi_template_id=<TID>,
  dpi_name="<NAME>",
  dpi_rules={
      {elem_name="<ELEMENT>",
       active=<true|false>,
       regex=[=[DPI_PCRE_REGEX]=]},
      ...
  }}

where the following additional key-value pairs are required or strongly recommended:

When creating the template that holds the basicLists, YAF first checks for an information element of whose name is elem_name+"List" and whose type is basicList. If found, that element is used, otherwise the generic basicList element is used.

Assigning DPI Using A Plugin

Rules with label_type="plugin" may have a dpi_type of either "plugin" or "regex-plugin". The latter is rarely used; see the DNP 3.0, Modbus, and Ethernet/IP rules for examples.

Typically, a "plugin" dpi_type is used with label_type="plugin" rules and the same <LIBRARY> is used to perform DPI. In this case, no additional key-value pairs are required and the rule has the following form:

{label=<APP>,
 label_type="plugin",
 active=true,
 value=[=[<LIBRARY>]=],
 args={[[ARG1]], [[ARG2]], ...},
 dpi_type="plugin"}

Creating New Information Elements

The variable elements may be used to define new information elements within the CERT private enterprise domain (Private Enterprise Number (PEN) 6871). If defined, it must be an array of tables, where each table represents an element:

elements = {
  {name="<NAME>", id=<NUM>, is_string=<true|false>},
  ...
}

Each table in elements must contain the keys name and id, where <NAME> is the name of the element and <NUM> is its ID. <NAME> must be an unused name and <NUM> must be an unused ID between 1 and 16383 inclusive. If you attempt to use an existing name or ID, yaf exists with an error message. To find an unused name and ID, consult the CERT IPFIX Registry for the defined elements.

The key is_string accepts a boolean; if present and true, the type of the new element is set to string, otherwise the new element's type is octetArray. Typically string should be true unless the object contains binary data.

Keep in mind that user defined information elements may be added for any "regex" dpi_type, but there is a total limit of 40 regexes per protocol.

To find out if yaf accepted your elements, run yaf with --verbose and review the terminal output or log file.

Upon yaf startup and capture, you will be able to see if the rule files and their regular expressions were accepted.

[2021-10-15 17:20:14] DPI Running for ALL Protocols
[2021-10-15 17:20:14] Reading packets from packets.pcap
[2021-10-15 17:20:14] Initializing Rules from DPI File /usr/local/etc/yafDPIRules.conf
[2021-10-15 17:20:14] DPI rule scanner accepted 63 rules from the DPI Rule File

An unacceptable regular expression will be brought to your attention with the above statements. If you choose certain protocols for inspection using the --dpi-select option, only the appropriate rule statements will be loaded into the DPI Rule Scanner.

DPI Data Export

yaf's output consists of an IPFIX message stream. yaf uses a variety of templates for IPFIX data records; As of yaf 3.0, DPI information is now housed in its own named subTemplateList: yafDPIList. Rather than the previous subTemplateMultiList which could have a variety of nested templates for each record, the yafDPIList will only use one template per record, corresponding to the DPI for the protocol specified in the silkAppLabel field.

Below are the templates that may appear depending on the application label of the flow. For more information on yaf information elements see <todo link to basic flow record>. For more information on IPFIX Structured lists, see Export of Structured Data in IPFIX, RFC 6313.

Most of the values are exported as a basicList, which represents a list of zero or more instances of any Information Element. In YAF 3, there is a specific element of type basicList that holds a single type of element. For example, the HTTP GET DPI is exported in the httpGetList element, which is a basicList that holds zero or more httpGet elements.

HTTP

Hypertext Transfer Protocol (HTTP) Deep Packet Inspection is based on RFC 2616.

Optional HTTP Elements

The following information elements are defined but not enabled by default. To enable any of the following fields individually, set the field's active key to true in the yafDPIRules.conf file.

HTTP Mobile Fields

These fields may be enabled all together by setting the http_mobile variable to true in the yafDPIRules.conf file.

HTTP Extra Fields

These fields may be enabled all together by setting the http_extra variable to true in the yafDPIRules.conf file.

SSH

Secure Shell (SSH) Deep Packet Inspection is based on RFC 4253.

SMTP

Simple Mail Transfer Protocol (SMTP) Deep Packet Inspection is based on RFC 2821.

As of YAF 2.12, the SMTP DPI data is represented as a hierarchy. The outer layer captures the SMTP connection information. For each message sent during the connection, a separate sub-record is created. Within the subrecord, another subrecord exists that contains the header fields found in the DATA section exported as key-value pairs.

DNS

Domain Name System (DNS) Deep Packet Inspection is based on RFC 1035.

DNS information is exported in the yafDPIList as zero or more DNS Resource Record Templates. Each Resource Record (RR) entry contains generic RR information such as type, TTL, and name. There is also a subTemplateList element, dnsDetailRecordList, that contains RR specific information based on the type of RR (A Record vs NS Record, for example). The main subTemplateList contains one entry for each RR in the flow.

The following elements are contained in the DNS Resource Record Template.

DNS Resource Record Types

This section describes the templates that may be used in the dnsDetailRecordList according to the type of resource record (the dnsRRType).

DNSSEC Resource Record Types

DNSSEC information is not exported by default. To export DNSSEC information, set dnssec_enabled to true in the yafDPIRules.conf file. DNSSEC DPI is based on RFC 4034, Resource Records for the DNS Security Extensions and, for NSEC3, RFC 5155, DNS Security (DNSSEC) Hashed Authenticated Denial of Existence.

FTP

File Transfer Protocol (FTP) Deep Packet Inspection is based on RFC 959.

TLS/SSL

Transport Layer Security (TLS)/Secure Socket Layer (SSL) Deep Packet Inspection can identify and export handshake and certificate information if it is contained in the payload of the flow. The TLS/SSL DPI is presented in a nested structure, detailed in this section. At the top level are the TLS handshake-related elements.

Each certificate identified by yaf is exported as an entry in the sslCertList subTemplateList or/and the sslBinaryCertificateList basicList. sslCertList is used when the yafDPIRules.conf file has cert_export_enabled set to false (the default) or cert_hash_enabled set to true (false is the default). sslBinaryCertificateList is used when cert_export_enabled is true. (To get both lists, set both those variable to true.)

An entry in sslCertList contains basic certificate elements such as serial numbers, validity timestamps, and optionally the certificate's hash when cert_hash_enabled is set to true. sslCertList also contains three nested subTemplateLists: sslIssuerFieldList for Issuer fields, sslSubjectFieldList for Subject fields, and sslExtensionFieldList for Extension fields. Each of these subTemplateLists contains an ID and it associated value, where the IDs correspond to the attributes associated with X.509 Certificates, object identifiers id-ce and id-at. See below.

sslBinaryCertificateList is a basicList of sslBinaryCertificate entries, where each entry hold an entire binary certificate.

Note that most certificate chains are about 3000 bytes. In order to capture the entire certificate chain, the --max-payload option to yaf should be set appropriately.

TLS/SSL Handshake-related Elements

TLS/SSL X.509 Certificate-related Elements

When cert_export_enabled is false or cert_hash_enabled is true, the top-level TLS/SSL record contains:

TLS/SSL Issuer and Subject Field Lists

The sslIssuerFieldList and sslSubjectFieldList are encoded as a sequence of Relative Distinguished Names, which are basically type-value pairs. These lists contain zero or more occurrences of the RelativeDistinguishedName id-value pairs pulled from the X.509 Certificate. sslIssuerFieldList entries are pulled from the Issuer RDNSequence, and sslIssuerFieldList entries from the Subject RDNSequence. There is one entry in the list for each pair. Each entry contains:

TLS/SSL Extension Field List

Extensions are only defined for X.509 v3 certificates and provide methods for associating additional attributes with the Issuer and Subject information. The sslExtensionFieldList contains zero or more occurrences of the following element pair:

Full Certificate Export Template

When cert_export_enabled is true, the top-level TLS/SSL record contains:

SLP

Service Location Protocol (SLP) Deep Packet Inspection is based on RFC 2608.

IMAP

Internet Message Access Protocol (IMAP) Deep Packet Inspection is based on RFC 3501.

IRC

Internet Relay Chat (IRC) Deep Packet Inspection is based on RFC 2812.

RTSP

Real Time Streaming Protocol (RTSP) Deep Packet Inspection is based on RFC 2326.

SIP

Session Initiation Protocol (SIP) Deep Packet Inspection is based on RFC 3261.

NNTP

Network News Transfer Protocol (NNTP) Deep Packet Inspection is based on RFC 977.

TFTP

Trivial File Transfer Protocol (TFTP) Deep Packet Inspection is based on RFC 1350.

MySQL

MySQL Deep Packet Inspection is based on information found in the MYSQL Developer Documentation.

POP3

Post Office Protocol 3 (POP3) Deep Packet Inspection is based on RFC 1939.

RTP

Real-time Transport Protocol (RTP) Deep Packet Inspection is based on RFC 3550. The Payload Type indicates the format of the payload and how it should be interpreted by the receiving application.

DNP3

Distributed Network Protocol 3 (DNP3) Deep Packet Inspection is slightly different than other plugin-based protocols, operating as a "regex-plugin" dpi_type. YAF will export the following information if the yafDPIRules.conf contain regular expressions under the label ID 20000. The regular expressions are compared against the payload of DNP3 packets starting with the function code in the DNP Application Layer header. YAF will loop through all the the available DNP3 packets contained in the captured payload. For each packet that matches one of the regular expressions listed in yafDPIRules.conf, YAF will include an entry in the exported subTemplateList. The DNP3 DPI is disabled by default. To enable it, uncomment the DPI-related entries in the DNP3 rule in the yafDPIRules.conf file.

Modbus

Modbus DPI is similar to DNP3 DPI. YAF will export any patterns matched by the regular expressions labeled with the ID 502 found in the yafDPIRules.conf file. The regular expressions are compared against the payload of all valid Modbus packets starting right after the MBAP header (offset 7), beginning with the Modbus function code. The Modbus DPI is disabled by default. To enable it, uncomment the DPI-related entries in the Modbus rule in the yafDPIRules.conf file.

Ethernet/IP

Ethernet/IP DPI is similar to DNP3 and Modbus DPI. YAF will export any patterns matched by the regular expressions labeled with the ID 44818 in the yafDPIRules.conf file. The regular expressions are compared against the start of the payload of all valid Ethernet/IP packets (Command in the Encapsulation Header is the first byte). The Ethernet/IP DPI is disabled by default. To enable it, uncomment the DPI-related entries in the Ethernet/IP rule in the yafDPIRules.conf file.