YAF

What is New with YAF 3.0

• Records and template IDs have changed
  • DPI Location
  • Named IPFIX Lists
• Improved AppLabel and DPI processing
• Single AppLabel and DPI Configuration File - Written in Lua.
  • Applabeling
    • Regular Expression Applabel (regex)
    • Plug-in Applabel (plugin)
    • Signature Applabel (signature)
  • DPI
    • Regex DPI
    • Plugin DPI
    • Regex-plugin DPI
• DPI Enhancements
  • TLS
  • SSH
• Template metadata enhancements
• Increased Payload Capture
• Selective Payload Export
• Easier application of dpi
• Field Name Changes
  • General Rules for Renaming
  • IE Renames, Sorted by Element ID
  • IE Renames, Sorted by Old Name

This page documents the new features and incompatible changes in YAF 3.

Records and template IDs have changed

Flow records emitted by YAF have been streamlined to fields more accessible as the variety of YAF flow customizations are enable. Things like --payload, --flow-stats, and --mac have their fields moved from the subTemplateMultiList in yaf 2.x into the main record. This puts more fields in the main part of the record, making them easier to find downstream.

A consequence of moving fields into the main flow record is that each combination of possible user-enabled features creates a new template ID. If looking for a specific template ID when processing YAF 3 data, you maybe have to broaden your scope to find what you are looking for.

DPI Location

Within a flow record, the DPI information is now housed in its own named subTemplateList: yafDPIList (IE 6871/432). Rather than the previous subTemplateMultiList which could have a variety of nested templates for each record, the yafDPIList will only use one template per record, corresponding to the DPI for the protocol specified in the silkAppLabel field.

The subTemplateMultiList is still used to house data from non-DPI plug-ins, such as the DHCP fingerprinting.

Named IPFIX Lists

In YAF 3, most of the IPFIX list structures now have meaningful names that make the data more readable and accessible. These lists are still of IPFIX type basicList, subTemplateList, and subTemplateMultiList, but have names we hope an analyst finds more meaningful. YAF has always thoroughly utilized IPFIX's list structures for efficiency. This allows YAF to only have fields in its output that it has valid data for, preventing generally empty records that try to account for every possible field in fixed columns.

Examples:

• DPI information is now stored in yafDPIList, a subTemplateList.

• In DNS subrecords:

  • The sub-record that holds the DNS details is a dnsDetailRecordList, a subTemplateList whose template varies by the type of DNS record. Previously this used the subTemplateList element.

• In TLS/SSL records:

  • The list of certificates is in sslCertList, a subTemplateList.

  • sslIssuerFieldList is a subTemplateList holding a key and value of fields for the certificate issuer. The sslSubjectFieldList and sslExtensionFieldList are similar.

  • The list of TLS ciphers is in sslCipherList, a basicList of sslCipher elements.

• In HTTP records:

  • The HTTP GET requests are in httpGetList, a basicList of httpGet elements.

  • The HTTP User-Agent strings are in httpUserAgentList, a basicList of httpUserAgent elements.

Improved AppLabel and DPI processing

The regular expressions to determine the application label have been improved.

New AppLabels have been added.

DPI regexes have been updated.

DPI regexes have been added.

DPI templates have changed.

Single AppLabel and DPI Configuration File - Written in Lua.

As part of the DPI changes in YAF 3, the configuration files for application labeling and DPI options have been merged into a single file, and the file's syntax has been moved from a custom format to a Lua based file. Below is the new syntax for the config file, this information can also be found in the file itself.

The file yafApplabelRules.conf is no longer used and may be deleted.

NOTE: If you have customized yafApplabelRules.conf or yafDPIRules.conf, make copies of your files prior to installing YAF, install YAF, and then modify the new yafDPIRules.conf file with your modifications.

As is standard in Lua, -- starts a comment, and the rest of the line is a comment.

The file must define a variable named applabels which holds an array of tables, each of which represents an application label and optional DPI setting.

applabels = {
 -- HTTP
 <ENTRY FOR APPLABEL 80>,
 -- SSH
 <ENTRY FOR APPLABEL 22>,
 ...
 }

Applabeling

Each applabel is represented by a table in the format of:

{label = <APP>,
 label_type = "<TYPE>",
 value = [[<EXPRESSION>]],
 ports = <PORT_LIST>,
 protocol = <PROTO>
 dpi_type = "<DPI_TYPE>",
 <OPTIONAL_DPI_SETTINGS>}

where

• <APP> is the application label to apply (an unsigned 16-bit decimal integer in the range 0 to 65535)

• <EXPRESSION> specifies how to recognize the given application protocol. The contents of <EXPRESSION> depend on the <TYPE>.

• <TYPE> specifies the format of <EXPRESSION>. Three types of applabel rules are supported: regex, plugin, and signature.

• The optional ports key has a list of port numbers as its value, for example { 8080, 8888 }. The optional <PORT_LIST> tells YAF to check this rule when the flow's source or destination port matches these values. Values in the <PORT_LIST> must be unique across all application labels (<APP>) and <PORT_LIST> values defined in the configuration file.

• <PROTO> limits rule to being tested only when the flow record's protocolIdentifier matches <PROTO> or when <PROTO> is 0. If the protocol key is not present, the test is performed. <PROTO> is a value between 0 and 255 inclusive, but only 0, 6, and 17 are relevant since yaf only runs applabel checks for TCP and UDP records.

• The dpi_type key-value pair is optional; see the [DPI][] section below.

• <OPTIONAL_DPI_SETTINGS> are additional settings that depend on the value given to the dpi_type key.

Regular Expression Applabel (regex)

Regular expression (label_type="regex") rules have the following form:

{label = <APP>,
 label_type = "regex",
 value = [[<PATTERN>]],
 ports = { <PORT_LIST> },
 protocol = <PROTO>,
 dpi_type = "<DPI_TYPE>",
 <OPTIONAL_DPI_SETTINGS>}

The regular expression <PATTERN> is compared against the available payload of both directions of the flow, and if the expression matches either direction of the payload, the label <APP> is applied to the flow. <PATTERN> is a PCRE regular expression; for the syntax and semantics of <PATTERN>, see the PCRE documentation for patterns.

Note: The square brackets that surround the expression are "Lua long brackets" and surround a "long literal", which is useful for writing regular expressions. Zero or more equal signs (=) may appear between each set of brackets (the same number on each side), [====[as an example]====]. At least one equal sign is recommended when specifying a regex.

When applying applabel rules to a flow, YAF first checks whether the flow's source or destination port matches a rule's "label" (<APP>) or a port in the <PORT_LIST>; if no rule matching the ports is found, YAF tests the flow against all applabel rules in the order in which they appear.

Example: Here is the applabel rule for rsync traffic:

-- RSYNC
{label = 873, label_type = "regex",
 value = [=[^@RSYNCD:]=]},

The dpi_type is not specified since YAF does not extract any DPI data for rsync traffic.

Plug-in Applabel (plugin)

Plugin rules (label_type="plugin") are used to label application payload using a C dynamically loaded library, and have the following form:

{label = <APP>,
 label_type = "plugin",
 value = "<LIBRARY>",
 protocol = <PROTO>,
 dpi_type = "<DPI_TYPE>",
 <OPTIONAL_DPI_SETTINGS>}

where <LIBRARY> is the name of a dynamically loadable library that exists somewhere within the LD_LIBRARY_PATH, the LTDL_LIBRARY_PATH, or a system library path, without the library name extension (usually .so). If the plug-in returns 1, the flow will be labeled with <APP>. Otherwise, the flow will be labeled with whatever value the plug-in returns (useful for a plug-in that can identify more than 1 protocol). See the source code to the plug-ins that ship with YAF for details.

Example: Here is the applabel rule for dhcp traffic:

-- DHCP
{label = 67, label_type = "plugin",
 value = "dhcpplugin"},

Note again the lack of dpi_type.

Signature Applabel (signature)

Signature rules (label_type="signature") have the following form:

{label = <APP>,
 label_type = "signature",
 value = [[<EXPRESSION>]],
 dpi_type = "<DPI_TYPE>"}

The regular expression <EXPRESSION> is compared against the available payload of the flow, and if the expression matches, the label <APP> is applied to the flow.

Signatures rules are similar to Regex Rules except YAF tests the signature rules first and they are applied to all flow records regardless of source and destination port values. For expressions that you want to search for first before port-based matching, use this format. The expression is a PCRE regular expression.

DPI

The process of deep packet inspection into the labeled flows is now specified as part of the same configuration. DPI configuration is specified using one of three dpi_type values: regex, plugin, or regex-plugin. If no DPI processing available or desired, do not include the dpi_type key.

Regex DPI

Regex DPI (dpi_type="regex") can be used with with any type of applabeling. Using the regex DPI type requires that you specify three additional fields in the applabeling/dpi line: "dpi_name", "dpi_template_id", and "dpi_rules":

• dpi_name (string): This field specifies the name for the protocol being labeled and inspected. This string will be used to automatically create things like the output template name.

• dpi_template_id (number): This field specifies the numeric value (ideally specified in hexadecimal for clarity) to be used for the template ID for this protocol's DPI. Must be unique.

• dpi_rules (array): This field specifies the list of regex rules that will be applied to the payload to generate the output DPI elements. Each rule looks like the following:

  {elem_name = "<NAME>", regex = [[<REGEX>]]}

  where both directional payloads are checked for matches of <REGEX> and any matches are stored in a basicList of the IPFIX information element whose name is <NAME>. If there is an information element of type basicList having the name "<NAME>List", that element is used in place of a generic basicList.

  If capturing parentheses are used in the pattern, only the text captured by the first set of parentheses is stored.

Example: Here is the applabel and a couple of DPI rules for HTTP:

-- HTTP
{label = 80, label_type = "regex", active = true,
 value = [=[HTTP/\d]=],
 dpi_type = "regex", dpi_name = "http", dpi_template_id = 0xC600,
 dpi_rules = {
   {elem_name = "httpServerString",
    regex = [=[^Server: ?([-a-zA-Z0-9.~!/:;@#$%^&*=+,_() ]+)]=]},
   {elem_name = "httpUserAgent",
    regex = [=[^User-Agent: ?([-a-zA-Z0-9.~!/:;@#$%^&*=+,_() ]+)]=]},
 }
},

Note the use of capturing parentheses to include only the values of the Server and User-Agent settings but not the keywords.

Matches for the Server string are stored in the httpServerStringList element, a basicList of type httpServerString, and similarly for the httpUserAgentList and httpUserAgent.

Plugin DPI

Plugin DPI (dpi_type="plugin") can only be used when the applabel type is also a plugin, as the same plugin will be responsible for both labeling and doing the deep packet inspection. The plugin DPI type requires that you specify additional fields in the applabeling/dpi line: The "dpi_templates" field is required; some plugins also require "plugin_rules":

• dpi_templates (table of arrays of string): A table where each entry represents one template created by this plugin. The set of keys are fixed, and each key matches the name of a template defined by the plugin. The value for each key is an array of strings containing the information elements exported as part of that template. The user may comment out (or remove) entries to prevent those elements from being exported by the plugin.

• plugin_rules (table of strings): A table where each key names a variable defined in the plugin (the set of keys is fixed). Each value is a string containing a PCRE regular expression used by the plugin.

Example: Here is the applabel and DPI setting for TFTP:

-- TFTP
{label = 69, label_type = "plugin", active = true,
 value = "tftpplugin", dpi_type = "plugin",
 dpi_templates = {
   yaf_tftp = {
     "tftpFilename",
     "tftpMode",
   },
 },
 plugin_rules = {
   tftpRegex = [=[\x00[\x01|\x02]([-a-zA-Z1-9. ]+)\x00(?i)(netascii|octet|mail)\x00]=],
 },
},

Regex-plugin DPI

This type (dpi_type="regex-plugin") of DPI indicates that a plugin is used for application labeling but that user specified regexes are used for extracting the DPI, similar to Regex DPI. The only parameter needed is the same dpi_rules parameter as mentioned above.

DPI Enhancements

TLS

TLS (SSL) deep packet inspection now includes the JA3 hash for server and client and the string used as the basic for the JA3 hash.

SSH

SSH deep packet inspection is greatly enhanced and is now handled by a C plugin. Information includes the negotiated algorithms, the MD5 of the server key, the HASSH hash for the server and client, and the string used as the basis for the HASSH.

Template metadata enhancements

Increased Payload Capture

YAF now supports collecting more than 65535 bytes of payload for performing deep packet inspection. Note the amount of raw payload YAF may export is still limited by the maximum size of an IPFIX message.

Selective Payload Export

YAF can now emit the forward and reverse payloads (up to the lesser of --max-payload or --max-export, if specified) for only a user-specified list of application labels using --payload-applabel-select. Previously payload export was all or none, but now maybe you only want payloads for flows YAF was unable to label, or only the details for DNS flows.

Easier application of dpi

Field Name Changes

There are significant changes to the names of CERT enterprise-specific information elements for data emitted by YAF.

There were several motivations for this change. In some cases the name veered far from the IPFIX naming idiom. The contents of other elements had evolved over time necessitating a new name. Some elements were redundant, while others represented unrelated entities. Some elements related to a single protocol (such as DNS) used inconsistent naming.

Data is more accessible if field are used in only one place in the record. While names maybe longer, they are more clear which is the point of naming fields for analysts to use. A full list of name changes is below.

General Rules for Renaming

Rules we more-or-less ended up with:

1. Initialisms and acronyms generally in all-caps in the middle of a name or all-small at the start of a name.

2. However, "Id" and "id" for identifiers.

3. DNS RR type names are in all-caps.

4. Prefix names that are a concept specific to our tools with a tool name or "certTool" for cross-tool stuff.

5. Do not prefix names that may only be currently produced by our tools, but which could be produced by and meaningful to anyone monitoring Internet traffic.

For everything we've talked about, this is a rundown on what's changed or not changed and why, grouped by the rationale of the changes.

YAF 3 does not rename these elements which contain initialisms:

YAF 3 renames these information elements to make it clear that they are for communicating information across CERT tools. (Currently for diagnostic purposes):

YAF 3 renames these to make it clear that these IEs are YAF-specific statistics information, and not for any other "mean flow rate", etc.:

YAF 3 renames these TLS/SSL-related elements to make their meaning more clear:

YAF 3 gives names to these super_mediator-specific dedup-related elements that more clearly denote their meaning and origin:

1. Made more generic, because this field is used for other DNS features of super_mediator, not just for dedup.
2. Unified these two into one item, since the meaning here is simply "how many times did you see this deduplicated thing".
3. Made it clear that this is a representation of data that is being deduplicated.

YAF 3 gives these pipeline-specific elements names that aren't completely alien, and labels them to say they're for pipeline:

YAF 3 uses "Id" as short for identifier instead of "ID":

YAF 3 changes these so the first segment of the name (nDPI) isn't mixed-case on its own:

YAF 3 treats "fingerprint" as a single word, since all of the literature around p0f fingerprinting and DHCP fingerprinting does so:

YAF 3 uses a name for "EtherNet/IP™" that makes people less likely to incorrectly assume that it is ethernet-over-IP:

YAF 3 does not use hyphens in these names, since those really do not fit with the IPFIX naming idiom at all:

YAF 3 splits up existing DNSSEC information elements that are used in multiple places in order to disambiguate them, and renames existing single-use IEs that do not say what resource record types they're for:

YAF 3 renames these DNS-related names so they are a bit cleaner and much more clear about their meaning in relation to DNS information.

YAF 3 renames this so that the RR type name is in all caps:

YAF 3 continues using RR type names in all caps and does not change these:

YAF 3 takes ownership of these elements from Analysis Pipeline, gives them names consistent with the other DNS-related elements, and uses them in DNS subrecords place of sourceIPv4Address, sourceIPv6Address, and protocolIdentifier.

YAF 3 adds these additional DNSSEC information elements:

IE Renames, Sorted by Element ID

This table lists the renamed information elements, sorted by the element identifier. All IEs use the CERT Enterprise Number, 6871.

IE Renames, Sorted by Old Name

This table lists the renamed information elements, sorted by the previous name. All IEs use the CERT Enterprise Number, 6871.