What is New with YAF 3.0

This page documents the new features and incompatible changes in YAF 3.

Records and template IDs have changed

Flow records emitted by YAF have been streamlined to fields more accessible as the variety of YAF flow customizations are enable. Things like --payload, --flow-stats, and --mac have their fields moved from the subTemplateMultiList in yaf 2.x into the main record. This puts more fields in the main part of the record, making them easier to find downstream.

A consequence of moving fields into the main flow record is that each combination of possible user-enabled features creates a new template ID. If looking for a specific template ID when processing YAF 3 data, you maybe have to broaden your scope to find what you are looking for.

DPI Location

Within a flow record, the DPI information is now housed in its own named subTemplateList: yafDPIList (IE 6871/432). Rather than the previous subTemplateMultiList which could have a variety of nested templates for each record, the yafDPIList will only use one template per record, corresponding to the DPI for the protocol specified in the silkAppLabel field.

The subTemplateMultiList is still used to house data from non-DPI plug-ins, such as the DHCP fingerprinting.

Named IPFIX Lists

In YAF 3, most of the IPFIX list structures now have meaningful names that make the data more readable and accessible. These lists are still of IPFIX type basicList, subTemplateList, and subTemplateMultiList, but have names we hope an analyst finds more meaningful. YAF has always thoroughly utilized IPFIX's list structures for efficiency. This allows YAF to only have fields in its output that it has valid data for, preventing generally empty records that try to account for every possible field in fixed columns.

Examples:

Improved AppLabel and DPI processing

The regular expressions to determine the application label have been improved.

New AppLabels have been added.

DPI regexes have been updated.

DPI regexes have been added.

DPI templates have changed.

Single AppLabel and DPI Configuration File - Written in Lua.

As part of the DPI changes in YAF 3, the configuration files for application labeling and DPI options have been merged into a single file, and the file's syntax has been moved from a custom format to a Lua based file. Below is the new syntax for the config file, this information can also be found in the file itself.

The file yafApplabelRules.conf is no longer used and may be deleted.

NOTE: If you have customized yafApplabelRules.conf or yafDPIRules.conf, make copies of your files prior to installing YAF, install YAF, and then modify the new yafDPIRules.conf file with your modifications.

As is standard in Lua, -- starts a comment, and the rest of the line is a comment.

The file must define a variable named applabels which holds an array of tables, each of which represents an application label and optional DPI setting.

applabels = {
 -- HTTP
 <ENTRY FOR APPLABEL 80>,
 -- SSH
 <ENTRY FOR APPLABEL 22>,
 ...
 }

Applabeling

Each applabel is represented by a table in the format of:

{label=<N>,
 label_type="<TYPE>",
 value=[[<EXPRESSION>]],
 dpi_type="<DPI_TYPE>" }

where

Regular Expression Applabel (regex)

Regular expression (label_type="regex") rules have the following form:

{label=<N>,
 label_type="regex",
 value=[[<PATTERN>]],
 ports={PORT_LIST},
 dpi_type="<DPI_TYPE>"}

The regular expression <PATTERN> is compared against the available payload of both directions of the flow, and if the expression matches either direction of the payload, the label <N> is applied to the flow. <PATTERN> is a PCRE regular expression; for the syntax and semantics of <PATTERN>, see the PCRE documenation for patterns.

Note: The square backets that surround the expression are "Lua long brackets" and surround a "long literal", which is useful for writing regular expressions. Zero or more equal signs (=) may appear between each set of brackets (the same number on each side), [====[as an example]====]. At least one equal sign is recommended when specifying a regex.

When applying applabel rules to a flow, YAF first checks whether the flow's source or destination port matches a rule's "label" (<N>); if no rule matching the ports is found, YAF tests the flow against all applabel rules in the order in which they appear.

The optional ports key has a list of port numbers as its value. The optional <PORT_LIST> tells YAF to check this rule when the flow's source or destination port matches these values. Values in the <PORT_LIST> must be unique across all application labels (<N>) and <PORT_LIST> values defined in the configuration file.

Example: Here is the applabel rule for rsync traffic:

-- RSYNC
{label=873, label_type="regex",
 value=[=[^@RSYNCD:]=]},

The dpi_type is not specified since YAF does not extract any DPI data for rsync traffic.

Plug-in Applabel (plugin)

Plugin rules (label_type="plugin") are used to label application payload using a C dynamically loaded library, and have the following form:

{label=<N>,
 label_type="plugin",
 value=[[<LIBRARY>]],
 dpi_type="<DPI_TYPE>"}

where <LIBRARY> is the name of a dynamically loadable library that exists somewhere within the LD_LIBRARY_PATH, the LTDL_LIBRARY_PATH, or a system library path, without the library name extension (usually .so). If the plug-in returns 1, the flow will be labeled with <N>. Otherwise, the flow will be labeled with whatever value the plug-in returns (useful for a plug-in that can identify more than 1 protocol). See the source code to the plug-ins that ship with YAF for details.

Example: Here is the applabel rule for dhcp traffic:

-- DHCP
{label=67, label_type="plugin",
 value=[[dhcpplugin]]},

Note again the lack of dpi_type.

Signature Applabel (signature)

Signature rules (label_type="signature") have the following form:

{label=<N>,
 label_type="signature",
 value=[[<EXPRESSION>]],
 dpi_type="<DPI_TYPE>"}

The regular expression <EXPRESSION> is compared against the available payload of the flow, and if the expression matches, the label <N> is applied to the flow.

Signatures rules are similar to Regex Rules except YAF tests the signature rules first and they are applied to all flow records regardless of source and destination port values. For expressions that you want to search for first before port-based matching, use this format. The expression is a PCRE regular expression.

DPI

The process of deep packet inspection into the labeled flows is now specified as part of the same configuration. DPI configuration is specified using one of three dpi_type values: regex, plugin, or regex-plugin. If no DPI processing available or desired, do not include the dpi_type key.

Regex DPI

Regex DPI (dpi_type="regex") can be used with with any type of applabeling. Using the regex DPI type requires that you specify three additional fields in the applabeling/dpi line: "dpi_name", "dpi_template_id", and "dpi_rules":

Example: Here is the applabel and DPI rules for secure shell:

-- SSH
{label=22,
 label_type="regex",
 value=[=[^SSH-\d]=],
 dpi_type="regex",
 dpi_name="ssh",
 dpi_template_id=0xCC00,
 dpi_rules={
   {elem_name="sshVersion",
    regex=[=[^(SSH-\d\.\d-?[-_.a-zA-Z0-9 ]*)\r\n]=]},
 }
},

Note the use of capturing parantheses to include everything to the end of the line but not end-of-line character sequence.

Matches for the ssh version string are stored in the sshVersionList element, a basicList of type sshVersion.

Plugin DPI

Plugin DPI (dpi_type="plugin") can only be used when the applabel type is also a plugin, as the same plugin will be responsible for both labeling and doing the deep packet inspection. No additional fields are needed except where otherwise specified for specific plugins by the configuration file.

Example: Here is the applabel and DPI setting for IRC:

-- IRC -- Include the de facto port
{label=194,
 ports={6667},
 label_type="plugin",
 value=[[ircplugin]],
 dpi_type="plugin"},

Note the use of ports to include the de facto port for IRC.

Regex-plugin DPI

This type (dpi_type="regex-plugin") of DPI indicates that a plugin is used for application labeling but that user specified regexes are used for extracting the DPI, similar to Regex DPI. The only parameter needed is the same dpi_rules parameter as mentioned above.

Template metadata enhancements

Selective Payload Export

YAF can now emit the forward and reverse payloads (up to the lesser of --max-payload or --max-export, if specified) for only a user-specified list of application labels using --payload-applabel-select. Previously payload export was all or none, but now maybe you only want payloads for flows YAF was unable to label, or only the details for DNS flows.

Easier application of dpi

Field Name Changes

There are significant changes to the names of CERT enterprise-specific information elements for data emitted by YAF.

There were several motivations for this change. In some cases the name veered far from the IPFIX naming idiom. The contents of other elements had evolved over time necessitating a new name. Some elements were redundant, while others represented unrelated entities. Some elements related to a single protocol (such as DNS) used inconsistent naming.

Data is more accessible if field are used in only one place in the record. While names maybe longer, they are more clear which is the point of naming fields for analysts to use. A full list of name changes is below.

General Rules for Renaming

Rules we more-or-less ended up with:

  1. Initialisms and acronyms generally in all-caps in the middle of a name or all-small at the start of a name.

  2. However, "Id" and "id" for identifiers.

  3. DNS RR type names are in all-caps.

  4. Prefix names that are a concept specific to our tools with a tool name or "certTool" for cross-tool stuff.

  5. Do not prefix names that may only be currently produced by our tools, but which could be produced by and meaningful to anyone monitoring Internet traffic.

For everything we've talked about, this is a rundown on what's changed or not changed and why, grouped by the rationale of the changes.

YAF 3 does not rename these elements which contain initialisms:

ID NAME
138 imapStartTLS
143 rtspURL
199 dnsTTL
257 httpIMEI
258 httpIMSI
259 httpMSISDN
270 httpDNT
326 smtpStartTLS
329 smtpURL
335 smtpURLList
362 imapStartTLSList
367 rtspURLList
397 httpIMEIList
398 httpIMSIList
399 httpMSISDNList
409 httpDNTList
432 yafDPIList

YAF 3 renames these information elements to make it clear that they are for communicating information across CERT tools. (Currently for diagnostic purposes):

ID OLD NAME NEW NAME
550 tombstoneId certToolTombstoneId
551 exporterConfiguredId certToolExporterConfiguredId
552 exporterUniqueId certToolExporterUniqueId
554 tombstoneAccessList certToolTombstoneAccessList

YAF 3 renames these to make it clear that these IEs are YAF-specific statistics information, and not for any other "mean flow rate", etc.:

ID OLD NAME NEW NAME
100 expiredFragmentCount yafExpiredFragmentCount
101 assembledFragmentCount yafAssembledFragmentCount
102 meanFlowRate yafMeanFlowRate
103 meanPacketRate yafMeanPacketRate
104 flowTableFlushEventCount yafFlowTableFlushEventCount
105 flowTablePeakCount yafFlowTablePeakCount

YAF 3 gives names to these super_mediator-specific dedup-related elements that more clearly denote their meaning and origin:

ID OLD NAME NEW NAME NOTE
927 dnsRName smDNSData 1
928 dnsHitCount DEPRECATED 2
929 observedDataTotalCount smDedupHitCount 2
930 observedData smDedupData 3
  1. Made more generic, because this field is used for other DNS features of super_mediator, not just for dedup.
  2. Unified these two into one item, since the meaning here is simply "how many times did you see this deduped thing".
  3. Made it clear that this is a representation of data that is being deduplicated.

YAF 3 gives these pipeline-specific elements names that aren't completely alien, and labels them to say they're for pipeline:

ID OLD NAME NEW NAME
305 DNS_A_Record pipelineDNSARecord
306 DNS_AAAA_Record pipelineDNSAAAARecord
307 DNS_RESOURCE_RECORD pipelineDNSResourceRecord

YAF 3 uses "Id" as short for identifier instead of "ID":

ID OLD NAME NEW NAME
226 dnsID dnsId
292 mptcpAddressID mptcpAddressId

YAF 3 changes these so the first segment of the name (nDPI) isn't mixed-case on its own:

ID OLD NAME NEW NAME
300 nDPIL7Protocol ndpiL7Protocol
301 nDPIL7SubProtocol ndpiL7SubProtocol

YAF 3 treats "fingerprint" as a single word, since all of the literature around p0f fingerprinting and DHCP fingerprinting does so:

ID OLD NAME NEW NAME
107 osFingerPrint osFingerprint
242 dhcpFingerPrint dhcpFingerprint

YAF 3 uses a name for "EtherNet/IP™" that makes people less likely to incorrectly assume that it is ethernet-over-IP:

ID OLD NAME NEW NAME
286 ethernetIPData enipData
421 ethernetIPDataList enipDataList

YAF 3 does not use hyphens in these names, since those really do not fit with the IPFIX naming idiom at all:

ID OLD NAME NEW NAME
254 httpX-Forwarded-For httpXForwardedFor
271 httpX-Forwarded-Proto httpXForwardedProto
272 httpX-Forwarded-Host httpXForwardedHost
273 httpX-Forwarded-Server httpXForwardedServer
274 httpX-DeviceID httpXDeviceId
275 httpX-Profile httpXProfile
280 httpX-UA-Compatible httpXUaCompatible
394 httpX-Forwarded-ForList httpXForwardedForList
410 httpX-Forwarded-ProtoList httpXForwardedProtoList
411 httpX-Forwarded-HostList httpXForwardedHostList
412 httpX-Forwarded-ServerList httpXForwardedServerList
413 httpX-DeviceIDList httpXDeviceIdList
414 httpX-ProfileList httpXProfileList
419 httpX-UA-CompatibleList httpXUaCompatibleList

YAF 3 splits up existing DNSSEC information elements that are used in multiple places in order to disambiguate them, and renames existing single-use IEs that do not say what resource record types they're for:

ID OLD NAME NEW NAME
227 dnsAlgorithm DEPRECATED, replaced with:
423 NEW dnsDNSKEYAlgorithm
433 NEW dnsDSAlgorithm
435 NEW dnsNSEC3Algorithm
441 NEW dnsNSEC3PARAMAlgorithm
447 NEW dnsRRSIGAlgorithm
 
228 dnsKeyTag DEPRECATED, replaced with
434 NEW dnsDSKeyTag
448 NEW dnsRRSIGKeyTag
 
229 dnsSigner dnsRRSIGSigner
230 dnsSignature dnsRRSIGSignature
231 dnsDigest dnsDSDigest
232 dnsPublicKey dnsDNSKEYPublicKey
 
233 dnsSalt DEPRECATED, replaced with
439 NEW dnsNSEC3Salt
444 NEW dnsNSEC3PARAMSalt
 
234 dnsHashData DEPRECATED, replaced with
445 NEW dnsNSECNextDomainName
438 NEW dnsNSEC3NextHashedOwnerName
 
235 dnsIterations DEPRECATED, replaced with
437 NEW dnsNSEC3Iterations
443 NEW dnsNSEC3PARAMIterations
 
236 dnsSignatureExpiration dnsRRSIGSignatureExpiration
237 dnsSignatureInception dnsRRSIGSignatureInception
238 dnsDigestType dnsDSDigestType
239 dnsLabels dnsRRSIGLabels
240 dnsTypeCovered dnsRRSIGTypeCovered
241 dnsFlags dnsDNSKEYFlags
304 dnsKeyProtocolIdentifier dnsDNSKEYProtocol
 
449 NEW [split from dnsTTL] dnsRRSIGOriginalTTL

YAF 3 renames these DNS-related names so they are a bit cleaner and much more clear about their meaning in relation to DNS information.

ID OLD NAME NEW NAME
175 dnsQRType dnsRRType
177 dnsNXDomain dnsResponseCode
178 dnsRRSection dnsSection
179 dnsQName dnsName
431 dnsQRDetailRecordList dnsDetailRecordList

YAF 3 renames this so that the RR type name is in all caps:

ID OLD NAME NEW NAME
180 dnsCName dnsCNAME

YAF 3 continues using RR type names in all caps and does not change these:

ID NAME
181 dnsMXPreference
182 dnsMXExchange
183 dnsNSDName
184 dnsPTRDName
208 dnsTXTData
209 dnsSOASerial
210 dnsSOARefresh
211 dnsSOARetry
212 dnsSOAExpire
213 dnsSOAMinimum
214 dnsSOAMName
215 dnsSOARName
216 dnsSRVPriority
217 dnsSRVWeight
218 dnsSRVPort
219 dnsSRVTarget
302 dnsA
303 dnsAAAA

YAF 3 adds these additional DNSSEC information elements:

ID OLD NAME NEW NAME DATA TYPE
446 NEW dnsNSECTypeBitmaps octetArray
440 NEW dnsNSEC3TypeBitMaps octetArray
436 NEW dnsNSEC3Flags unsigned8
442 NEW dnsNSEC3PARAMFlags unsigned8

IE Renames, Sorted by Element ID

This table lists the renamed information elements, sorted by the element identifier. All IEs use the CERT Enterprise Number, 6871.

ID OLD NAME NEW NAME
100 expiredFragmentCount yafExpiredFragmentCount
101 assembledFragmentCount yafAssembledFragmentCount
102 meanFlowRate yafMeanFlowRate
103 meanPacketRate yafMeanPacketRate
104 flowTableFlushEventCount yafFlowTableFlushEventCount
105 flowTablePeakCount yafFlowTablePeakCount
107 osFingerPrint osFingerprint
175 dnsQRType dnsRRType
177 dnsNXDomain dnsResponseCode
178 dnsRRSection dnsSection
179 dnsQName dnsName
180 dnsCName dnsCNAME
226 dnsID dnsId
 
227 dnsAlgorithm DEPRECATED, replaced with:
423 NEW dnsDNSKEYAlgorithm
433 NEW dnsDSAlgorithm
435 NEW dnsNSEC3Algorithm
441 NEW dnsNSEC3PARAMAlgorithm
447 NEW dnsRRSIGAlgorithm
 
228 dnsKeyTag DEPRECATED, replaced with
434 NEW dnsDSKeyTag
448 NEW dnsRRSIGKeyTag
 
229 dnsSigner dnsRRSIGSigner
230 dnsSignature dnsRRSIGSignature
231 dnsDigest dnsDSDigest
232 dnsPublicKey dnsDNSKEYPublicKey
 
233 dnsSalt DEPRECATED, replaced with
439 NEW dnsNSEC3Salt
444 NEW dnsNSEC3PARAMSalt
 
234 dnsHashData DEPRECATED, replaced with
445 NEW dnsNSECNextDomainName
438 NEW dnsNSEC3NextHashedOwnerName
 
235 dnsIterations DEPRECATED, replaced with
437 NEW dnsNSEC3Iterations
443 NEW dnsNSEC3PARAMIterations
 
236 dnsSignatureExpiration dnsRRSIGSignatureExpiration
237 dnsSignatureInception dnsRRSIGSignatureInception
238 dnsDigestType dnsDSDigestType
239 dnsLabels dnsRRSIGLabels
240 dnsTypeCovered dnsRRSIGTypeCovered
241 dnsFlags dnsDNSKEYFlags
242 dhcpFingerPrint dhcpFingerprint
254 httpX-Forwarded-For httpXForwardedFor
271 httpX-Forwarded-Proto httpXForwardedProto
272 httpX-Forwarded-Host httpXForwardedHost
273 httpX-Forwarded-Server httpXForwardedServer
274 httpX-DeviceID httpXDeviceId
275 httpX-Profile httpXProfile
280 httpX-UA-Compatible httpXUaCompatible
286 ethernetIPData enipData
292 mptcpAddressID mptcpAddressId
300 nDPIL7Protocol ndpiL7Protocol
301 nDPIL7SubProtocol ndpiL7SubProtocol
304 dnsKeyProtocolIdentifier dnsDNSKEYProtocol
305 DNS_A_Record pipelineDNSARecord
306 DNS_AAAA_Record pipelineDNSAAAARecord
307 DNS_RESOURCE_RECORD pipelineDNSResourceRecord
394 httpX-Forwarded-ForList httpXForwardedForList
410 httpX-Forwarded-ProtoList httpXForwardedProtoList
411 httpX-Forwarded-HostList httpXForwardedHostList
412 httpX-Forwarded-ServerList httpXForwardedServerList
413 httpX-DeviceIDList httpXDeviceIdList
414 httpX-ProfileList httpXProfileList
419 httpX-UA-CompatibleList httpXUaCompatibleList
421 ethernetIPDataList enipDataList
431 dnsQRDetailRecordList dnsDetailRecordList
550 tombstoneId certToolTombstoneId
551 exporterConfiguredId certToolExporterConfiguredId
552 exporterUniqueId certToolExporterUniqueId
554 tombstoneAccessList certToolTombstoneAccessList
927 dnsRName smDNSData
 
928 dnsHitCount DEPRECATED, merged into smDedupHitCount(929)
 
929 observedDataTotalCount smDedupHitCount
930 observedData smDedupData
 
449 NEW [split from dnsTTL] dnsRRSIGOriginalTTL
446 NEW dnsNSECTypeBitMaps
440 NEW dnsNSEC3TypeBitMaps
436 NEW dnsNSEC3Flags
442 NEW dnsNSEC3PARAMFlags

IE Renames, Sorted by Old Name

This table lists the renamed information elements, sorted by the previous name. All IEs use the CERT Enterprise Number, 6871.

ID OLD NAME NEW NAME
101 assembledFragmentCount yafAssembledFragmentCount
242 dhcpFingerPrint dhcpFingerprint
 
227 dnsAlgorithm DEPRECATED, replaced with:
423 NEW dnsDNSKEYAlgorithm
433 NEW dnsDSAlgorithm
435 NEW dnsNSEC3Algorithm
441 NEW dnsNSEC3PARAMAlgorithm
447 NEW dnsRRSIGAlgorithm
 
180 dnsCName dnsCNAME
231 dnsDigest dnsDSDigest
238 dnsDigestType dnsDSDigestType
241 dnsFlags dnsDNSKEYFlags
 
234 dnsHashData DEPRECATED, replaced with
445 NEW dnsNSECNextDomainName
438 NEW dnsNSEC3NextHashedOwnerName
 
928 dnsHitCount DEPRECATED, merged into smDedupHitCount(929)
 
226 dnsID dnsId
 
235 dnsIterations DEPRECATED, replaced with
437 NEW dnsNSEC3Iterations
443 NEW dnsNSEC3PARAMIterations
 
304 dnsKeyProtocolIdentifier dnsDNSKEYProtocol
 
228 dnsKeyTag DEPRECATED, replaced with
434 NEW dnsDSKeyTag
448 NEW dnsRRSIGKeyTag
 
239 dnsLabels dnsRRSIGLabels
177 dnsNXDomain dnsResponseCode
232 dnsPublicKey dnsDNSKEYPublicKey
179 dnsQName dnsName
431 dnsQRDetailRecordList dnsDetailRecordList
175 dnsQRType dnsRRType
927 dnsRName smDNSData
178 dnsRRSection dnsSection
 
233 dnsSalt DEPRECATED, replaced with
439 NEW dnsNSEC3Salt
444 NEW dnsNSEC3PARAMSalt
 
230 dnsSignature dnsRRSIGSignature
236 dnsSignatureExpiration dnsRRSIGSignatureExpiration
237 dnsSignatureInception dnsRRSIGSignatureInception
229 dnsSigner dnsRRSIGSigner
240 dnsTypeCovered dnsRRSIGTypeCovered
306 DNS_AAAA_Record pipelineDNSAAAARecord
305 DNS_A_Record pipelineDNSARecord
307 DNS_RESOURCE_RECORD pipelineDNSResourceRecord
286 ethernetIPData enipData
421 ethernetIPDataList enipDataList
100 expiredFragmentCount yafExpiredFragmentCount
551 exporterConfiguredId certToolExporterConfiguredId
552 exporterUniqueId certToolExporterUniqueId
104 flowTableFlushEventCount yafFlowTableFlushEventCount
105 flowTablePeakCount yafFlowTablePeakCount
274 httpX-DeviceID httpXDeviceId
413 httpX-DeviceIDList httpXDeviceIdList
254 httpX-Forwarded-For httpXForwardedFor
394 httpX-Forwarded-ForList httpXForwardedForList
272 httpX-Forwarded-Host httpXForwardedHost
411 httpX-Forwarded-HostList httpXForwardedHostList
271 httpX-Forwarded-Proto httpXForwardedProto
410 httpX-Forwarded-ProtoList httpXForwardedProtoList
273 httpX-Forwarded-Server httpXForwardedServer
412 httpX-Forwarded-ServerList httpXForwardedServerList
275 httpX-Profile httpXProfile
414 httpX-ProfileList httpXProfileList
280 httpX-UA-Compatible httpXUaCompatible
419 httpX-UA-CompatibleList httpXUaCompatibleList
102 meanFlowRate yafMeanFlowRate
103 meanPacketRate yafMeanPacketRate
292 mptcpAddressID mptcpAddressId
300 nDPIL7Protocol ndpiL7Protocol
301 nDPIL7SubProtocol ndpiL7SubProtocol
930 observedData smDedupData
929 observedDataTotalCount smDedupHitCount
107 osFingerPrint osFingerprint
554 tombstoneAccessList certToolTombstoneAccessList
550 tombstoneId certToolTombstoneId
 
449 NEW [split from dnsTTL] dnsRRSIGOriginalTTL
446 NEW dnsNSECTypeBitMaps
440 NEW dnsNSEC3TypeBitMaps
436 NEW dnsNSEC3Flags
442 NEW dnsNSEC3PARAMFlags