Monitoring for Large-Scale Networks

YAF

yaf

NAME

yaf - Yet Another Flowmeter

SYNOPSIS

yaf     [--in INPUT_SPECIFIER] [--out OUTPUT_SPECIFIER]
        [--config CONFIGURATION_FILE]
        [--live LIVE_TYPE] [--ipfix TRANSPORT_PROTOCOL]
        [--no-output]
        [--decompress DECOMPRESS_DIR]
        [--filter BPF_FILTER]
        [--rotate ROTATE_DELAY] [--lock] [--caplist]
        [--daemonize] [--pidfile] [--promisc-off]
        [--stats INTERVAL][--no-stats] [--noerror]
        [--no-tombstone] [--tombstone-configured-id IDENTIFIER]
        [--export-interface]
        [--gre-decode] [--no-frag]
        [--vxlan-decode] [--vxlan-decode-ports PORTS]
        [--geneve-decode] [--geneve-decode-ports PORTS]
        [--max-frags FRAG_TABLE_MAX]
        [--ip4-only] [--ip6-only]
        [--idle-timeout IDLE_TIMEOUT]
        [--active-timeout ACTIVE_TIMEOUT]
        [--udp-temp-timeout TEMPLATE_TIMEOUT]
        [--no-vlan-in-key]
        [--force-read-all]
        [--flow-stats] [--delta]
        [--ingress INGRESS_INT] [--egress EGRESS_INT]
        [--no-element-metadata] [--no-template-metadata]
        [--max-payload PAYLOAD_OCTETS] [--udp-payload]
        [--max-export PAYLOAD_OCTETS]
        [--max-flows FLOW_TABLE_MAX]
        [--export-payload] [--payload-applabel-select LABELS]
        [--silk] [--udp-uniflow PORT]
        [--uniflow] [--mac] [--force-ip6-export]
        [--observation-domain DOMAIN_ID] [--entropy]
        [--applabel] [--dpi] [--dpi-select LABELS]
        [--dpi-rules-file RULES_FILE]
        [--ndpi] [--ndpi-protocol-file FILE]
        [--ipfix-port PORT] [--tls] [--tls-ca CA_PEM_FILE]
        [--tls-cert CERT_PEM_FILE] [--tls-key KEY_PEM_FILE]
        [--become-user UNPRIVILEGED_USER]
        [--become-group UNPRIVILEGED_GROUP]
        [--log LOG_SPECIFIER] [--loglevel LOG_LEVEL]
        [--verbose] [--version]
        [--p0fprint] [--p0f-fingerprints FILENAME]
        [--fpexport]
        [--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]]
        [--plugin-opts "OPTIONS[,OPTIONS...]"]
        [--plugin-conf CONF_FILE_PATH[,CONF_FILE_PATH...]]
        [--pcap PCAP_FILE_PREFIX] [--pcap-per-flow]
        [--max-pcap MAX_FILE_MB] [--pcap-timer PCAP_ROTATE_DELAY]
        [--pcap-meta-file META_FILE_PREFIX] [--index-pcap]
        [--hash FLOW_KEY_HASH] [--stime FLOW_START_TIMEMS]

DESCRIPTION

yaf is Yet Another Flowmeter and yaf is a suite of tools to do flow metering. yaf is used as a sensor to capture flow information on a network and export that information in IPFIX format. It reads packet data from pcap(3) dumpfiles as generated by tcpdump(1), from live capture from an interface using pcap(3), pf_ring, an Endace DAG capture device, a Napatech adapter, or Netronome NFE card aggregates these packets into flows, and exports flow records via IPFIX over SCTP, TCP or UDP, or into serialized IPFIX message streams (IPFIX files) on the local file system.

Since yaf is designed to be deployed on white-box sensors attached to local network segments or span ports at symmetric routing points, it supports bidirectional flow assembly natively. Biflow export is done via the export method specified in RFC 5103 Bidirectional Flow Export using IPFIX. See the OUTPUT section below for information on this format.

yaf also supports experimental partial payload capture, specifically for banner-grabbing applications and protocol verification purposes.

The output of yaf is designed to be collected and manipulated by flow processing toolchains supporting IPFIX. The yafscii(1) tool, which is installed as part of yaf, can be used to print yaf output in a human-readable format somewhat reminiscent of tcpdump(1), though it does not support printing structured data (e.g., subTemplateLists). The ipfixDump(1) and ipfix2json(1) tools from fixbuf also print IPFIX data as text, and they support structured data. yaf output may also be analyzed using the SiLK suite, Super Mediator (https://tools.netsa.cert.org/super_mediator/), and Analysis Pipeline (https://tools.netsa.cert.org/analysis-pipeline5/), all available from the CERT NetSA group.

OPTIONS

Configuration File

The YAF configuration file can be used instead of or in addition to command line arguments. Lua must be installed for use of the yaf configuration file.

--config CONFIGURATION_FILE: If present, use the variables set in the CONFIGURATION_FILE. The CONFIGURATION_FILE is a Lua configuration file, a plain text file that can also be a Lua program. A template configuration file can be found in /Users/mthomas/netsa/work/Dans-TemplateMetaData/share/yaf/yaf.init; the file must be modified before it can be used by yaf. yaf will use the variables set in the configuration file along with any command line arguments. See yaf.init(1) for a description of this file.

Input Options

These options control where yaf will take its input from. yaf can read packets from a pcap dumpfile (as generated by tcpdump -w) or live from an interface via libpcap, libdag, or libnapatech, or Netronome API. By default, if no input options are given, yaf reads a pcap dumpfile on standard input.

--in INPUT_SPECIFIER: INPUT_SPECIFIER is an input specifier. If --live is given, this is the name of an interface (e.g. eth0, en0, dag0, nt3g, nt3g0:1, 0:0) to capture packets from. Otherwise, it is a filename; the string - may be used to read from standard input (the default). See --live for more information on formats for Napatech, Dag, and Netronome Interface formats.
--caplist: If present, treat the filename in INPUT_SPECIFIER as an ordered newline-delimited list of pathnames to pcap(3) dumpfiles. Blank lines and lines beginning with the character '#' within this are ignored. All pathnames are evaluated with respect to the working directory yaf is run in. These dumpfiles are processed in order using the same flow table, so they must be listed in ascending time order. This option is intended to ease the use of yaf with rotated or otherwise split tcpdump(1) output.
--noerror: Used with the --caplist option. When present, this prevents yaf from exiting when processing a list of dumpfiles in the middle due to an error in a file. yaf will continue to process all files given in the INPUT_SPECIFIER despite errors within those files.
--live LIVE_TYPE: If present, capture packets from an interface named in the INPUT_SPECIFIER. LIVE_TYPE is one of pcap for packet capture via libpcap, pfring for packet capture via libpfring, or dag for packet capture via an Endace DAG interface using libdag, or napatech for packet capture via a Napatech Adapter, or netronome for packet capture via a Netronome NFE card, or zc for packet capture via PF_RING ZC. pfring is only available if yaf was built with PF_RING support. See the yafzcbalance(1) man page for using yaf with PF_RING ZC. dag is only available if yaf was built with Endace DAG support. napatech is only available if yaf was built with Napatech API support. If LIVE_TYPE is napatech, the INPUT_SPECIFIER given to --in should be in the form nt3g[<streamID>:<ports>]. StreamID and Ports are optional. StreamID if given, is the ID that the traffic stream will be assigned to on the incoming ports. Ports may be a comma-separated list of ports to listen on. If [ports] is not specified, the default is to listen on All ports. StreamID defaults to 0. netronome is only available if yaf was built with Netronome API support. If LIVE_TYPE is netronome, the INPUT_SPECIFIER given to --in should be in the form <device>:<ring> where device is the NFE card ID, typically 0. Ring is the capture ring ID which is configured via a modprobe configuration file and resides in /etc/modprobe.d/pcd.conf.
--export-interface: If present, the interface on which a packet was received will be noted internally within yaf. When flow records are exported from yaf, an ingressInterface and an egressInterface set of fields will be added to the output. The ingressInterface field will be the physical interface which captured the packet while the egressInterface will be the physical interface | 0x100. This can be used to separate traffic based on DAG physical ports. For use with the DAG card, traffic received on separate ports will be separated into different flows if yaf is configured with the --enable-daginterface option. Otherwise the physical port will simply be exported in the ingressInterface or egressInterface fields in the IPFIX record (flows can exist over multiple interfaces). This option requires building DAG, Netronome, or Napatech support in yaf with the --with-dag, --with-napatech, or --with-netronome switch. In previous versions of yaf this option was enabled using the --dag-interface or --napatech-interface switch. It is now enabled by default when yaf is built with DAG, Netronome, or Napatech support. It can be disabled by configuring yaf with --enable-interface=no. To separate traffic received on separate ports into separate flows, you must use --enable-daginterface when configuring yaf.
--filter BPF_FILTER: If present, enable Berkeley Packet Filtering (BPF) in yaf with BPF_FILTER as the incoming traffic filter. Syntax of BPF_FILTER follows the expression format described in the tcpdump(1) man page. This option is not currently supported if --live is set to dag or napatech or netronome as BPF filtering is implemented with libpcap. However, you may be able to use a BPF filter by running yaf with the DAG, Napatech, or Netronome implementations of libpcap.
--decompress DECOMPRESS_DIR: If present and the input file(s) are compressed (gzip'd), decompress the file to a temporary file within DECOMPRESS_DIR. If --caplist is also present, all files will be decompressed to DECOMPRESS_DIR. If this option is not present, yaf will decompress files to the variable specified by the TMPDIR environment variable or /tmp if TMPDIR is not set. The zlib library must be installed to use this feature.
--promisc-off: If present, yaf will not put the interface in promiscuous mode

Output Options

These options control where yaf will send its output. yaf can write flows to an IPFIX file or export flows to an IPFIX collector over SCTP, TCP, and UDP. By default, if no output options are given, yaf writes an IPFIX file to standard output.

--out OUTPUT_SPECIFIER

OUTPUT_SPECIFIER is an output specifier. If --ipfix is present, the OUTPUT_SPECIFIER specifies the hostname or IP address of the collector to which the flows will be exported. Otherwise, if --rotate is present, OUTPUT_SPECIFIER is the prefix name of each output file to write to. Otherwise, OUTPUT_SPECIFIER is a filename in which the flows will be written; the string - may be used to write to standard output (the default).

Examples

Output to file: --out flows.yaf
Output to collector on port 18000 at IP address 1.2.3.4: --out 1.2.3.4 --ipfix-port 18000 --ipfix tcp
Connect to the machine identified by the domain name "host.domain.edu" on port 4803.: --out 4803@host.domain.edu
Connect to the machine identified by the IP address "x.y.123.45" on port 18000.: --out x.y.123.45 --ipfix-port 18000

--ipfix TRANSPORT_PROTOCOL

If present, causes yaf to operate as an IPFIX exporter, sending IPFIX Messages via the specified transport protocol to the collector (e.g., SiLK's rwflowpack(8) or flowcap(8) facilities) named in the OUTPUT_SPECIFIER. Valid TRANSPORT_PROTOCOL values are tcp, udp, and sctp; sctp is only available if yaf was built with SCTP support; UDP is not recommended, as it is not a reliable transport protocol, and cannot guarantee delivery of messages. As per the recommendations in RFC 5101, yaf will retransmit templates three times within the template timeout period (configurable using --udp-temp-timeout or by default, 10 minutes). Use the --ipfix-port, --tls, --tls-ca, --tls-cert, and --tls-key options to further configure the connection to the IPFIX collector.

--rotate ROTATE_DELAY

If present, causes yaf to write output to multiple files, opening a new output file every ROTATE_DELAY seconds in the input data. Rotated files are named using the prefix given in the OUTPUT_SPECIFIER, followed by a suffix containing a timestamp in YYYYMMDDhhmmss format, a decimal serial number, and the file extension .yaf.

--lock

Use lockfiles for concurrent file access protection on output files. This is recommended for interoperating with the Airframe filedaemon facility.

--stats INTERVAL

If present, causes yaf to export process statistics every INTERVAL seconds. The default value for INTERVAL is 300 seconds or every 5 minutes. yaf uses IPFIX Options Templates and Records to export flow, fragment, and decoding statistics. If INTERVAL is set to zero, stats will not be exported.

--no-stats

If present, yaf will not export process statistics. yaf uses IPFIX Options Templates and Records to export flow, fragment, and decoding statistics. --no-stats takes precedence over --stats. When --no-stats is given, yaf also disables export of tombstone records.

--no-tombstone

If present, yaf will not export tombstone records. yaf uses IPFIX Options Templates and Records to export tombstone records. Tombstone records will only be exported if stats exporting is also active.

--tombstone-configured-id IDENTIFIER

If present, overrides the default "certToolExporterConfiguredId" value in tombstone records. This value should be less than 0xFFFF (65535). The default value is 0.

--no-output

If present, yaf will not export IPFIX data. It will ignore any argument provided to --out.

--daemonize

If present, yaf will run in daemon mode.

--pidfile

Used to specify complete path to the process ID file path.

Decoder Options

These options are used to modify the yaf packet decoder's behavior. None of these options are required; the default behavior for each option when not present is noted.

--no-frag: If present, ignore all fragmented packets. By default, yaf will reassemble fragments with a 30 second fragment timeout.
--max-frags FRAG_TABLE_MAX: If present, limit the number of outstanding, not-yet reassembled fragments in the fragment table to FRAG_TABLE_MAX by prematurely expiring fragments from the table. This option is provided to limit yaf resource usage when operating on data from very large networks or networks with abnormal fragmentation. The fragment table may exceed this limit slightly due to limits on how often yaf prunes the fragment table (every 5 seconds). By default, there is no fragment table limit, and the fragment table can grow to resource exhaustion.
--ip4-only: If present, ignore all IPv6 packets and export IPv4 flows only. The default is to process both IPv4 and IPv6 packets.
--ip6-only: If present, ignore all IPv4 packets and export IPv6 flows only. The default is to process both IPv4 and IPv6 packets.
--gre-decode: If present, attempt to decode GRE version 0 encapsulated packets. Flows will be created from packets within the GRE tunnels. Undecodeable GRE packets will be dropped. Without this option, GRE traffic is exported as IP protocol 47 flows. This option is presently experimental.
--vxlan-decode: If present, attempt to decode UDP-VxLAN encapsulated packets over specified ports. The default port is 4789. Flows will be created from packets within the VxLAN tunnels. Undecodeable VxLAN packets will be dropped. Exports the VNI to yafLayer2SegmentId. VxLAN VNI exporting takes precedence over Geneve VNI exporting when Geneve is also enabled.
--vxlan-decode-ports PORTS_LIST: If VxLAN decoding is enabled, only attempt to decode UDP-VxLAN encapsulated packets if the packet's destination port is specified in PORTS_LIST, a comma separated list of ports (integers from 0 to 65535 inclusive).
--geneve-decode: If present, attempt to decode Geneve encapsulated packets over specified ports. The default port is 6081. Flows will be created from packets within the Geneve tunnels. Undecodeable Geneve packets will be dropped. Exports the VNI to yafLayer2SegmentId unless VxLAN decoding is also enabled.
--geneve-decode-ports PORTS_LIST: If Geneve decoding is enabled, only attempt to decode Geneve encapsulated packets if the packet's destination port is specified in PORTS_LIST, a comma separated list of ports (integers from 0 to 65535 inclusive).

Flow Table Options

These options are used to modify the flow table behavior within yaf. None of these options are required; the default behavior for each option when not present is noted.

--idle-timeout IDLE_TIMEOUT

Set flow idle timeout in seconds. Flows are considered idle and flushed from the flow table if no packets are received for IDLE_TIMEOUT seconds. The default flow idle timeout is 300 seconds (5 minutes). Setting IDLE_TIMEOUT to 0 creates a flow for each packet.

--active-timeout ACTIVE_TIMEOUT

Set flow active timeout in seconds. Any flow lasting longer than ACTIVE_TIMEOUT seconds will be flushed from the flow table. The default flow active timeout is 1800 seconds (30 minutes).

--udp-temp-timeout TEMPLATE_TIMEOUT

Set UDP template timeout in seconds if --ipfix is set to udp. As per RFC 5101 recommendations, yaf will attempt to export templates three times within TEMPLATE_TIMEOUT. The default template timeout period is 600 seconds (10 minutes).

--max-payload PAYLOAD_OCTETS

If present, capture at most PAYLOAD_OCTETS octets from the start of each direction of each flow. Non-TCP flows will only capture payload from the first packet unless --udp-payload is set. If not present, yaf will not attempt to capture payload. Payload capture must be enabled for payload export (--export-payload), application labeling (--applabel), and entropy evaluation (--entropy). Note that payload capture is still an experimental feature.

--max-flows FLOW_TABLE_MAX

If present, limit the number of open flows in the flow table to FLOW_TABLE_MAX by prematurely expiring the flows with the least recently received packets; this is analogous to an adaptive idle timeout. This option is provided to limit yaf resource usage when operating on data from large networks. By default, there is no flow table limit, and the flow table can grow to resource exhaustion.

--udp-payload

Enable packet payload capture for all packets in a UDP flow. When this option is not given, yaf only captures the payload from the first packet for non-TCP flows. The maximum payload to capture is determined by the argument to --max-payload.

--silk

If present, export flows in "SiLK mode". As of yaf 2.0, this will export TCP information (flags, ISN) in the main flow record instead of within the SubTemplateMultiList. This flag must be used when exporting to SiLK for it to collect TCP flow information. This also introduces the following incompatibilities with standard IPFIX export:

totalOctetCount and reverseTotalOctetCount are clamped to 32 bits. Any packet that would cause either of these counters to overflow 32 bits will cause the flow to close with flowEndReason 0x02 (active timeout), and will become the first packet of a new flow. This is analogous to forcing an active timeout when the octet counters overflow.
The high-order bit of the flowEndReason IE is set on any flow created on a counter overflow, as above.
The high-order bit of the flowEndReason IE is set on any flow created on an active timeout.

Since this changes the semantics of the exported flowEndReason IE, it should only be used when generating flows and exporting to rwflowpack(8), flowcap(8), or writing files for processing with rwipfix2silk(1).

--force-read-all

If present, yaf will process out-of-sequence packets. However, it will still reject out-of-sequence fragments.

--no-vlan-in-key

If present, yaf will NOT use the VLAN ID in the flow key hash calculation for flows. This means that packets within the active/idle timeouts that have the same 5-tuple (sIP, dIP, sport, dport, protocol) but different VLAN IDs will be aggregated into 1 flow and the VLAN ID of the first packet in each direction will be exported in the vlanId and reverseVlanId fields.

--no-mpls

If yaf is built with Multiprotocol Label Switching (MPLS) support (use yaf --version to check), yaf uses the top three MPLS labels in the MPLS stack along with the five-tuple and VLAN ID to determine flow records, and it exports the top three MPLS labels in the IPFIX records. (The exported fields will not include the experimental bits and the bottom of stack bit.) The --no-mpls option may be used to disable the use of the MPLS labels and export of the MPLS elements. The switch is not available if yaf was built without MPLS support.

Export Options

These options are used to modify the data exported by yaf.

--export-payload: If present and payload capture is active (see --max-payload), add payload and reversePayload elements to the flow templates and export the first N octets of payload from each direction of the flow record, where N is the smaller of the arguments to --max-payload and --max-export. Non-TCP flows will only export payload from the first packet. By default, yaf does not export flow payload. See also --payload-applabel-select.
--payload-applabel-select APPLABEL_LIST: Enable payload export (as --export-payload) but only for the application labels specified in APPLABEL_LIST, a comma separated list of applabel values (integers from 0 to 65535 inclusive).
--max-export MAX_PAY_OCTETS: If present and payload export is active, export at most MAX_PAY_OCTETS octets from the start of each direction of each flow, where MAX_PAY_OCTETS may not be greater than the argument to --max-payload. If --max-export is not specified, all captured payload octets are exported. Payload export is only active when either --export-payload or --payload-applabel-select is given.
--uniflow: If present, export biflows using the Record Adjacency method in section 3 of RFC 5103. This is useful when exporting to IPFIX Collecting Processes that are not biflow-aware.
--mac: If present, export MAC-layer information; presently, exports source and destination MAC addresses.
--force-ip6-export: If present, force IPv4 flows to be exported with IPv6-mapped IPv4 addresses in ::ffff/96. This will cause all flows to appear to be IPv6 flows.
--observation-domain DOMAIN_ID: Set the observationDomainId on each exported IPFIX message to the given integer value. If not present, the observationDomainId defaults to 0. This value is also used as the exportingProcessId in the yaf statistics Option Record as a Scope Field.
--udp-uniflow PORT: If present, export each UDP packet on the given port (or 1 for all ports) as a single flow, with flowEndReason set to YAF_END_UDPFORCE (0x1F). This will not close the flow. The flow will stay open until it closes naturally by the idle and active timeouts. Most useful with --export-payload in order to export every UDP payload on a specific port.
--flow-stats: If present, export extra flow attributes and statistics in the subTemplateMultiList field. This will maintain information such as small packet count, large packet count, nonempty packet count, average interarrival times, total data octets, and max packet size. See the flow statistics template below for more information about each of the fields yaf exports.
--delta: If present, export octet and packet total counts in the delta count information elements. octetTotalCount will be exported in octetDeltaCount (IE 1), reverseOctetTotalCount will be exported in reverseOctetDeltaCount. packetTotalCount will be exported in packetDeltaCount (IE 2), and reversePacketTotalCount will be exported in reversePacketDeltaCount.
--ingress INGRESS_INT: If present, set the ingressInterface field in the flow template to INGRESS_INT. This field will also be populated if yaf was configured with --enable-daginterface or --enable-napatechinterface or --with-bivio. If yaf is running on a dag, napatech, or bivio, and the physical interface is available, this value will override INGRESS_INT.
--egress EGRESS_INT: If present, set the egressInterface field in the flow template to EGRESS_INT. This field will also be populated if yaf was configured with --enable-daginterface or --enable-napatechinterface or --with-bivio. If yaf is running on a dag, napatech, or bivio, and the physical interface is available, this value will override EGRESS_INT.
--no-element-metadata: If present, disable the export of information element metadata (see RFC 5610) before data records.
--no-template-metadata: If present, disable the export of template metadata (name, description, and other data) before data records.

Application Labeler / Deep Packet Inspection Options

If yaf is built with application labeler support enabled (using the --enable-applabel option to ./configure when yaf is built), then yaf can examine packet payloads and determine the application protocol in use within a flow, and export a 16-bit application label with each flow.

Additionally, if yaf is built with deep packet inspection support enabled (using the --enable-dpi option to ./configure when yaf is built), then yaf can examine packet payloads, capture useful information for a specific protocol, and export it in a protocol-specific template within a SubTemplateList.

The exported application label uses the common port number for the protocol. For example, HTTP traffic, independent of what port the traffic is detected on, will be labeled with a value of 80, the default HTTP port.

Application labeling and DPI requires payload capture to be enabled with the --max-payload option. A minimum payload capture length of 384 octets is recommended for best results.

--dpi-rules-file RULES_FILE: Read applabel and deep packet inspection rules from RULES_FILE. If not present, rules are read by default from /Users/mthomas/netsa/work/Dans-TemplateMetaData/etc/yafDPIRules.conf.
--applabel: If present, export application label data. Requires --max-payload to enable payload capture.
--dpi: If present, export deep packet inspection data and application label data. Requires --max-payload to enable payload capture.
--dpi-select APPLABEL_LIST: By default yaf will enable DPI for all applabels, but this flag can be used to provide a comma separated list of which applabels DPI processing should be run on.

nDPI Options

nDPI is a version of OpenDPI as maintained by ntop. You can read more about nDPI and the applications supported at: http://www.ntop.org/products/deep-packet-inspection/ndpi/

If yaf is built with nDPI support enabled (using the --with-ndpi option to ./configure when yaf is built), then yaf can examine packet payloads and determine the application protocol in use within a flow, and export the application protocol and sub-protocol with each flow.

nDPI requires payload capture to be enabled with the --max-payload option. A minimum payload capture length of 384 octets is recommended for best results.

--ndpi: If present, export nDPI data. Requires --max-payload to enable payload capture.
--ndpi-protocol-file FILE: Specify protocol file for sub-protocol and port-based protocol detection

Entropy Measurement

If yaf is built with entropy measurement enabled (using the --enable-entropy option to ./configure when yaf is built,) then yaf can examine the packet payloads and determine a Shannon Entropy value for the payload. The entropy calculation does not include the network (IP) or transport (UDP/TCP) headers. The entropy is calculated in terms of bits per byte, (log base 2.) The calculation generates a real number value between 0.0 and 8.0. That number is then converted into an 8-bit integer value between 0 and 255. Roughly, numbers above 230 are generally compressed (or encrypted) and numbers centered around approximately 140 are English text. Lower numbers carry even less information content. Another useful piece of information is that SSL/TLS tends to zero pad its packets, which causes the entropy of those flows to drop quite low.

--entropy: If present, export the entropy values for both the forward and reverse payloads. Requires the --max-payload option to operate.

IPFIX Connection Options

These options are used to configure the connection to an IPFIX collector.

--ipfix-port PORT: If --ipfix is present, export flows to TCP, UDP, or SCTP port PORT. If not present, the default IPFIX port 4739 is used. If --tls is also present, the default secure IPFIX port 4740 is used.
--tls: If --ipfix is present, use TLS to secure the connection to the IPFIX collector. Requires the TRANSPORT_PROTOCOL to be tcp, as DTLS over UDP or SCTP is not yet supported. Requires the --tls-ca, --tls-cert, and --tls-key options to specify the X.509 certificate and TLS key information.
--tls-ca CA_PEM_FILE: Use the Certificate Authority or Authorities in CA_PEM_FILE to verify the remote IPFIX Collecting Process' X.509 certificate. The connection to the Collecting Process will fail if its certificate was not signed by this CA (or by a certificate signed by this CA, recursively); this prevents export to unauthorized Collecting Processes. Required if --tls is present.
--tls-cert CERT_PEM_FILE: Use the X.509 certificate in CERT_PEM_FILE to identify this IPFIX Exporting Process. This certificate should contain the public part of the private key in KEY_PEM_FILE. Required if --tls is present.
--tls-key KEY_PEM_FILE: Use the private key in KEY_PEM_FILE for this IPFIX Exporting Process. This key should contain the private part of the public key in CERT_PEM_FILE. Required if --tls is present. If the key is encrypted, the password must be present in the YAF_TLS_PASS environment variable.

Privilege Options

These options are used to cause yaf to drop privileges when running as root for live capture purposes.

--become-user UNPRIVILEGED_USER

After opening the live capture device in --live mode, drop privilege to the named user. Using --become-user requires yaf to be run as root or setuid root. This option will cause all files written by yaf to be owned by the user UNPRIVILEGED_USER and the user's primary group; use --become-group as well to change the group yaf runs as for output purposes.

If running as root for live capture purposes and --become-user is not present, yaf will warn that privilege is not being dropped. We highly recommend the use of this option, especially in production environments, for security purposes.

--become-group UNPRIVILEGED_GROUP

--become-group can be used to change the group from the default of the user given in --become-user. This option has no effect if given without the --become-user option as well.

PCAP Options

These options are used to turn on and configure yaf's PCAP export capability.

--pcap PCAP_FILE_PREFIX

This option turns on rolling PCAP export in yaf. It will capture and write packets for all network traffic yaf has received and processed to PCAP files with the given PCAP_FILE_PREFIX. yaf will not create file directories. If yaf can't write to the file, yaf will turn off PCAP export. Pcap files will have names in the form of PCAP_FILE_PREFIX[datetime]_serialno.pcap". yaf will write to a file until the file size has reached --max-pcap or every --pcap-timer seconds (whichever happens first). By default, yaf rotates files every 5 MB. Files will be "locked" (".lock" will be appended to the filename) until yaf has closed the file. Be aware that your Operating System will have a limit on the maximum number of files in a directory and a maximum file size. If this limit is reached, yaf will write warning messages and terminate PCAP export. This may effect flow generation if yaf is also writing IPFIX files. Optionally, you can also export meta information about the flows in each rolling PCAP file with the --pcap-meta-file switch. If --pcap is used in conjunction with --hash and --stime, the PCAP_FILE_PREFIX should be the name of the PCAP file to write to (it will not be used as a file prefix).

--pcap-per-flow

If present, yaf will write a pcap file for each flow in the output directory given to --pcap. PCAP_FILE_PREFIX given to --pcap must be a file directory. This option is experimental and should only be used when reading pcap files of reasonable size. yaf only writes up to --max-payload bytes of each packet to the pcap file. Therefore, --max-payload must be set to an appropriate size to prevent packets from being truncated in the pcap file. yaf will use the last three digits of the flowStartMilliseconds as the directory and the flow key hash, flowStartMilliseconds, and serial number as the filename. See the included getFlowKeyHash program to easily calculate the name of the file for a given flow. When the pcap file has reached --max-pcap size, yaf will close the file, increment the serial number, and open a new pcap file with the same naming convention. Note that your operating system has a limit to the number of open file handles yaf can maintain at any given time. Therefore, the performance of yaf degrades when the number of open flows is greater than the maximum number of file handles.

--max-pcap MAX_FILE_MB

If present, set the maximum file size of pcap files to MAX_FILE_MB MB. The default is 5 MB.

--pcap-timer PCAP_ROTATE_DELAY

If present, yaf will rotate rolling pcap files every PCAP_ROTATE_DELAY seconds or when the file reaches --max-pcap size, whichever happens first. By default, yaf only rotates files based on file size.

--pcap-meta-file META_FILENAME

If present and --pcap is also present, yaf will export metadata on the flows contained in each rolling pcap file yaf is writing to the filename specified by META_FILENAME. yaf will write a line in the form:

flow_key_hash | flowStartMilliseconds | pcap_file_name

for each flow in the pcap. If a flow exists across 3 pcap files, there will be 3 lines in META_FILENAME for that flow (each line having a different filename). The pcap-meta-file will rotate approximately every 4,500,000 lines (or approx 2G). A new file will be created in the form META_FILENAME[datetime]_serialno.meta. This file can be uploaded to a database for flow correlation and flow-to-pcap analysis.

If --pcap-meta-file is present and --pcap is not present, yaf will export information about the pcap file(s) it is presently reading, as opposed to the pcap files yaf is writing.

--index-pcap

If present and --pcap and --pcap-meta-file are also present, export offset and length information about the packets yaf is writing to the rolling pcap files. This option can also be used when --pcap is not present, in which case it will write information about the file it is reading. Adding this option will force yaf to write one line per packet to the pcap-meta-file in the form:

flow_key_hash | flowStartMilliseconds | pcap_file_name/file_num | offset | length

If --pcap is present, the pcap_file_name is the name of the PCAP file yaf is writing. Otherwise, file_num will represent the sequential file number that yaf has processed. If yaf was given a single pcap file, this number will always be 0. offset is the offset into the pcap file of the beginning of the packet, at the start of the pcap packet header. length is the length of the packet including the pcap packet header. Using this offset, a separate program, such as yafMeta2Pcap, will be able to quickly extract packets for a flow. This file only rotates if META_FILE reaches max size.

--hash FLOW_KEY_HASH

If present, only write PCAP data for the flow(s) with FLOW_KEY_HASH. This option is only valid with the --pcap option.

--stime FLOW_START_TIMEMS

If present, only write PCAP data for the flow(s) with FLOW_START_TIMEMS and FLOW_KEY_HASH given to --hash. This option is only valid when used with the --hash and --pcap options.

Logging Options

These options are used to specify how log messages are routed. yaf can log to standard error, regular files, or the UNIX syslog facility.

--log LOG_SPECIFIER: Specifies destination for log messages. LOG_SPECIFIER can be a syslog(3) facility name, the special value stderr for standard error, or the absolute path to a file for file logging. The default log specifier is stderr if available and yaf is not running as a daemon, user otherwise.
--loglevel LOG_LEVEL: Specify minimum level for logged messages. In increasing levels of verbosity, the supported log levels are quiet, error, critical, warning, message, info, and debug. The default logging level is warning.
--verbose: Equivalent to --loglevel debug.
--version: If present, print version and copyright information to standard error and exit.

Plugin Options

These options are used to load, configure, and run a yaf plugin.

--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]: Specify the plugin to load. The loaded plugin must follow the yaf plugin framework. LIBPLUGIN_NAME must be the full path to the plugin library name. One plugin is included with yaf, a DHCP Fingerprinting plugin. This flag will only be recognized if yaf is configured with --enable-plugins. There are also configure options to export only DNS Authoritative and NXDomain responses. Read the plugin's documentation for more information.
--plugin-opts "OPTIONS[,OPTIONS...]": Specify the arguments to the plugin given to --plugin-name. This flag will only be recognized if yaf is configured with --enable-plugins and --plugin-name is set to a valid plugin.
--plugin-conf CONF_FILE_PATH[,CONF_FILE_PATH...]: Specify the path to a configuration file for the plugin given to --plugin-name. This flag will only be recognized if yaf is configured with --enable-plugins and --plugin-name is set to a valid plugin. If this switch is not used, but the plugin requires a configuration file, the default location /Users/mthomas/netsa/work/Dans-TemplateMetaData/etc will be used.

Passive OS Fingerprinting (p0f)

These options are used to enable p0f in yaf. p0f is presently experimental. There is no support in yafscii or SiLK for printing p0f related data. Currently, yaf uses the p0f Version 2 SYN fingerprints (see p0f.fp).

--p0fprint: If present, export p0f data. This data consists of three related information elements; osName, osVersion, osFingerprint. This flag requires yaf to be configured with --with-p0f.
--p0f-fingerprints: Location of the p0f fingerprint file(s), p0f.fp. Default is /Users/mthomas/netsa/work/Dans-TemplateMetaData/etc/p0f.fp. This version of yaf includes the updated CERT p0f fingerprints. See https://tools.netsa.cert.org/p0f/index.html for updates.
--fpexport: If present, enable export of handshake headers for external OS fingerprinters. The related information elements are firstPacketBanner and secondPacketBanner. This flag requires yaf to be configured with --enable-fpexporter.

OUTPUT

Basic Flow Record

yaf's output consists of an IPFIX message stream. yaf uses a variety of templates for IPFIX data records; the information elements that may appear in these templates are enumerated below. For further information about the IPFIX information model and IPFIX message stream, see RFC 5102, RFC 5101, and RFC 5103. As of yaf 2.0, yaf nests some templates in an IPFIX subTemplateMultiList. In order to retain compatibility with the SiLK Tools, use --silk to prevent yaf from nesting TCP Information Elements. Below are descriptions of each of the templates yaf will export. See the Internet-Draft Export of Structured Data in IPFIX for more information on IPFIX lists.

yaf assigns information element numbers to reverse flow elements in biflow capture based on the standard IPFIX PEN 29305. This applies only for information elements defined in the standard IPFIX Information Model (RFC 5102) that do not have a reverse information element already defined. For information elements defined under the CERT PEN, a standard method is used to calculate their reverse element identifier. The method is that bit fourteen is set to one in the IE field, (e.g. 16384 + the forward IE number.)

flowStartMilliseconds IE 152, 8 octets, unsigned

Flow start time in milliseconds since 1970-01-01 00:00:00 UTC. Always present.

flowEndMilliseconds IE 153, 8 octets, unsigned

Flow end time in milliseconds since 1970-01-01 00:00:00 UTC. Always present.

octetTotalCount IE 85, 8 octets, unsigned

Number of octets in packets in forward direction of flow. Always present (unless --delta is used.) May be encoded in 4 octets using IPFIX reduced-length encoding.

reverseOctetTotalCount Reverse (PEN 29305) IE 85, 8 octets, unsigned

Number of octets in packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.

packetTotalCount IE 86, 8 octets, unsigned

Number of packets in forward direction of flow. Always present (unless --delta is used.) May be encoded in 4 octets using IPFIX reduced-length encoding.

reversePacketTotalCount Reverse (PEN 29305) IE 86, 8 octets, unsigned

Number of packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.

octetDeltaCount IE 1, 8 octets, unsigned

Number of octets in packets in forward direction of flow. Only present if --delta is used. May be encoded in 4 octets using IPFIX reduced-length encoding.

reverseOctetDeltaCount Reverse (PEN 29305) IE 1, 8 octets, unsigned

Number of octets in reverse direction of flow. Only present if --delta is used and non-zero. May be encoded in 4 octets using IPFIX reduced-length encoding.

packetDeltaCount IE 2, 8 octets, unsigned

Number of packets in forward direction of flow. Only present if --delta is used. May be encoded in 4 octets using IPFIX reduced-length encoding.

reversePacketDeltaCount Reverse (PEN 29305) IE 2, 8 octets, unsigned

Number of packets in reverse direction of flow. Only present if --delta is used and non-zero. May be encoded in 4 octets using IPFIX reduced-length encoding.

reverseFlowDeltaMilliseconds CERT (PEN 6871) IE 21, 4 octets, unsigned

Difference in time in milliseconds between first packet in forward direction and first packet in reverse direction. Correlates with (but does not necessarily represent) round-trip time. Present if flow has a reverse direction.

sourceIPv4Address IE 8, 4 octets, unsigned

IPv4 address of flow source or biflow initiator. Present for IPv4 flows without IPv6-mapped addresses only.

destinationIPv4Address IE 12, 4 octets, unsigned

IPv4 address of flow source or biflow responder. Present for IPv4 flows without IPv6-mapped addresses only.

sourceIPv6Address IE 27, 16 octets, unsigned

IPv6 address of flow source or biflow initiator. Present for IPv6 flows or IPv6-mapped IPv4 flows only.

destinationIPv6Address IE 28, 16 octets, unsigned

IPv6 address of flow source or biflow responder. Present for IPv6 flows or IPv6-mapped IPv4 flows only.

sourceTransportPort IE 7, 2 octets, unsigned

TCP or UDP port on the flow source or biflow initiator endpoint. Always present.

destinationTransportPort IE 11, 2 octets, unsigned

TCP or UDP port on the flow destination or biflow responder endpoint. Always present. For ICMP flows, contains ICMP type * 256 + ICMP code. This is non-standard, and an open issue in yaf.

flowAttributes CERT (PEN 6871) IE 40, 2 octets, unsigned

Miscellaneous flow attributes for the forward direction of the flow. Always present (yaf 2.1 or later). Current flag values:

Bit 1: All packets in the forward direction have fixed size: For TCP flows, only packets that have payload will be considered (to avoid TCP handshakes and teardowns).
Bit 2: Packet(s) in the forward direction was received out-of-sequence
Bit 3: Host may be MP_CAPABLE (MPTCP-capable): For TCP flows, this bit will be set if a packet in the flow was seen that had the MP_CAPABLE TCP option or attempted an MP_JOIN operation.

reverseFlowAttributes CERT (PEN 6871) IE 16424, 2 octets, unsigned

Miscellaneous flow attributes for the reverse direction of the flow. Always present (yaf 2.1 or later). Current flag values:

Bit 1: All packets in the reverse direction have fixed size
Bit 2: Packet(s) in the reverse direction was received out-of-sequence
Bit 3: Host may be MP_CAPABLE (MPTCP-capable): For TCP flows, this bit will be set if a packet in the flow was seen that had the MP_CAPABLE TCP option or attempted an MP_JOIN operation.

protocolIdentifier IE 4, 1 octet, unsigned

IP protocol of the flow. Always present.

flowEndReason IE 136, 1 octet, unsigned

Flow end reason code, as defined by the IPFIX Information Model. Always present. In --silk mode, the high-order bit is set if the flow was created by continuation.

silkAppLabel CERT (PEN 6871) IE 33, 2 octets, unsigned

Application label, defined as the primary well-known port associated with a given application. Present if the application labeler is enabled, and was able to determine the application protocol used within the flow.

vlanId IE 58, 2 octets, unsigned

802.1q VLAN tag of the first packet in the forward direction of the flow.

reverseVlanId Reverse (PEN 29305) IE 58, 2 octets, unsigned

802.1q VLAN tag of the first packet in the reverse direction of the flow. Present if the flow has a reverse direction.

ipClassOfService IE 5, 1 octet, unsigned

For IPv4 packets, this is the value of the TOS field in the IPv4 header. For IPv6 packets, this is the Traffic Class field in the IPv6 header.

reverseIpClassOfService Reverse (PEN 29305) IE 5, 1 octet, unsigned

The TOS field in the IPv4 header for packets in the reverse direction, and Traffic Class field in the IPv6 header for packets in the reverse direction.

payloadEntropy CERT (PEN 6871) IE 35, 1 octet, unsigned

Shannon Entropy calculation of the forward payload data. This element is contained in the yaf Entropy template within the subTemplateMultiList.

reversePayloadEntropy CERT (PEN 6871) IE 16419, 1 octet, unsigned

Shannon Entropy calculation of the reverse payload data. This element is contained in the yaf Entropy template within the subTemplateMultiList.

mptcpInitialDataSequenceNumber, CERT (PEN 6871) IE 289, 8 octets, unsigned

The initial data sequence number found in the MPTCP Data Sequence Signal (DSS) Option.

mptcpReceiverToken, CERT (PEN 6871) IE 290, 4 octets, unsigned

The token used to identify an MPTCP connection over multiple subflows. This value is found in the MP_JOIN TCP Option for the initial SYN of a subflow.

mptcpMaximumSegmentSize, CERT (PEN 6871) IE 291, 2 octets, unsigned

The maximum segment size reported in the Maximum Segment Size TCP Option. This should be consistent over all subflows.

mptcpAddressId, CERT (PEN 6871), IE 292, 1 octet, unsigned

The address ID of the subflow found in the SYN/ACK of an MP_JOIN operation.

mptcpFlags, CERT (PEN 6871), IE 293, 1 octet, unsigned

Various MPTCP Values:

Bit 1: Priority was changed during the life of the subflow (MP_PRIO was seen)
Bit 2: Subflow has priority at setup (backup flag was not set at initialization).
Bit 3: Subflow failed. (MP_FAIL option was seen).
Bit 4: Subflow experienced fast close. (MP_FASTCLOSE options was seen).

sourceMacAddress, IE 56, 6 octets, unsigned

Source MAC Address of the first packet in the forward direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.

destinationMacAddress, IE 80, 6 octets, unsigned

Destination MAC Address of the first packet in the reverse direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.

osName CERT (PEN 6871) IE 36, variable-length

p0f OS Name for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

reverseOsName CERT (PEN 6871) IE 16420, variable-length

p0f OS Name for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

osVersion CERT (PEN 6871) IE 37, variable-length

p0f OS Version for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

reverseOsVersion CERT (PEN 6871) IE 16421, variable-length

p0f OS Version for the reverse flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

osFingerprint CERT (PEN 6871) IE 107, variable-length

p0f OS Fingerprint for the forward flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

reverseOsFingerprint CERT (PEN 6871) IE 16491, variable-length

p0f OS Fingerprint for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

firstPacketBanner CERT (PEN 6871) IE 38, variable-length

IP and transport headers for first packet in forward direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

reverseFirstPacketBanner CERT (PEN 6871) IE 16422, variable-length

IP and transport headers for first packet in reverse direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

secondPacketBanner CERT (PEN 6871) IE 39, variable-length

IP and transport headers for second packet in forward direction (third packet in sequence) to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

reverseSecondPacketBanner CERT (PEN 6871) IE 16423, variable-length

IP and transport headers for second packet in reverse direction (currently not used). Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

payload CERT (PEN 6871) IE 18, variable-length

Initial n bytes of forward direction of flow payload. Present if payload collection is enabled and payload is present in the forward direction of the flow. This element is contained in the yaf Payload template within the subTemplateMultiList.

reversePayload CERT (PEN 6871) IE 16402, variable-length

Initial n bytes of reverse direction of flow payload. Present if payload collection is enabled and payload is present in the reverse direction of the flow. This element is contained in the yaf Payload template within the subTemplateMultiList.

ingressInterface IE 10, 4 octets, unsigned

The index of the IP interface where packets of this flow are being received. Use --ingress, --napatech-interface, --dag-interface or configure yaf with bivio for this field to be present in the flow template. Use --ingress to manually set this field.

egressInterface IE 14, 4 octets, unsigned

The index of the IP interface where packets of this flow are being received. Use --egress, --napatech-interface, --dag-interface or configure yaf with bivio for this field to be present in the flow template. If using napatech, dag, or bivio, egressInterface will be the physical interface | 0x100. Use --egress to manually set this field.

yafLayer2SegmentId CERT (PEN 6871) IE 934, 4 octets, unsigned

Identifier of a layer 2 network segment in an overlay network. Present only if VXLAN decoding is enabled. The most significant byte identifies the layer 2 network overlay network encapsulation type:

0x00: Reserved
0x01: VxLAN
0x02: NVGRE
0x03: Geneve

The three lowest significant bytes hold the value of the layer 2 overlay network segment identifier.

dataByteCount CERT (PEN 6871) IE 502, 8 octets, unsigned

Total bytes transferred as payload.

averageInterarrivalTime CERT (PEN 6871) IE 503, 8 octets, unsigned

Average number of milliseconds between packets.

standardDeviationInterarrivalTime CERT (PEN 6871) IE 504, 8 octets, unsigned

Standard deviation of the interarrival time for up to the first ten packets.

tcpUrgTotalCount IE 223, 4 octets, unsigned

The number of TCP packets that have the URGENT Flag set.

smallPacketCount CERT (PEN 6871) IE 500, 4 octets, unsigned

The number of packets that contain less than 60 bytes of payload.

nonEmptyPacketCount CERT (PEN 6871) IE 501, 4 octets, unsigned

The number of packets that contain at least 1 byte of payload.

largePacketCount CERT (PEN 6871) IE 510, 4 octets, unsigned

The number of packets that contain more than 225 bytes of payload.

firstNonEmptyPacketSize CERT (PEN 6871) IE 505, 2 octets, unsigned

Payload length of the first non-empty packet.

maxPacketSize CERT (PEN 6871) IE 506, 2 octets, unsigned

The largest payload length transferred in the flow.

standardDeviationPayloadLength CERT (PEN 6871) IE 508, 2 octets, unsigned

The standard deviation of the payload length for up to the first 10 non empty packets.

firstEightNonEmptyPacketDirections CERT (PEN 6871) IE 507, 1 octet, unsigned

Represents directionality for the first 8 non-empty packets. 0 for forward direction, 1 for reverse direction.

reverseDataByteCount CERT (PEN 6871) IE 16886, 8 octets, unsigned

Total bytes transferred as payload in the reverse direction.

reverseAverageInterarrivalTime CERT (PEN 6871) IE 16887, 8 octets, unsigned

Average number of milliseconds between packets in reverse direction.

reverseStandardDeviationInterarrivalTime CERT (PEN 6871) IE 16888, 8 octets, unsigned

Standard deviation of the interarrival time for up to the first ten packets in the reverse direction.

reverseTcpUrgTotalCount Reverse (PEN 29305), IE 223, 4 octets, unsigned

The number of TCP packets that have the URGENT Flag set in the reverse direction.

reverseSmallPacketCount CERT (PEN 6871) IE 16884, 4 octets, unsigned

The number of packets that contain less than 60 bytes of payload in reverse direction.

reverseNonEmptyPacketCount CERT (PEN 6871) IE 16885, 4 octets, unsigned

The number of packets that contain at least 1 byte of payload in reverse direction.

reverseLargePacketCount CERT (PEN 6871) IE 16894, 4 octets, unsigned

The number of packets that contain more than 225 bytes of payload in the reverse direction.

reverseFirstNonEmptyPacketSize CERT (PEN 6871) IE 16889, 2 octets, unsigned

Payload length of the first non-empty packet in the reverse direction.

reverseMaxPacketSize CERT (PEN 6871) IE 16890, 2 octets, unsigned

The largest payload length transferred in the flow in the reverse direction.

reverseStandardDeviationPayloadLength CERT (PEN 6871) IE 16892, 2 octets, unsigned

The standard deviation of the payload length for up to the first 10 non empty packets in the reverse direction.

tcpSequenceNumber IE 184, 4 octets, unsigned

Initial sequence number of the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

reverseTcpSequenceNumber Reverse (PEN 29305) IE 184, 4 octets, unsigned

Initial sequence number of the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

initialTCPFlags CERT (PEN 6871) IE 14, 1 octet, unsigned

TCP flags of initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

unionTCPFlags CERT (PEN 6871) IE 15, 1 octet, unsigned

Union of TCP flags of all packets other than the initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

reverseInitialTCPFlags CERT (PEN 6871) IE 16398, 1 octet, unsigned

TCP flags of initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

reverseUnionTCPFlags CERT (PEN 6871) IE 16399, 1 octet, unsigned

Union of TCP flags of all packets other than the initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

mplsTopLabelStackSection IE 70, 3 octets

The MPLS Label from the top of the MPLS label stack entry. yaf does not include the Experimental bits and Bottom of the Stack bit in the export field. yaf must have been enabled with MPLS support for export of this field. Note that this field is defined as an octet array in the default libfixbuf Information Model. yaf uses the length override feature in libfixbuf to redefine it from variable length to 3 bytes.

mplsLabelStackSection2, IE 71, 3 octets

The MPLS Label from the MPLS label stack entry immediately before the top entry. yaf does not include the Experimental bits and Bottom of the Stack bit in the export field. yaf must have been enabled with MPLS support for export of this field. Note that this field is defined as an octet array in the default libfixbuf Information Model. yaf uses the length override feature in libfixbuf to redefine it from variable length to 3 bytes.

mplsLabelStackSection3, IE 72, 3 octets

The MPLS Label from the third entry in the MPLS label stack. yaf does not include the Experimental bits and Bottom of the Stack bit in the export field. yaf must have been enabled with MPLS support for export of this field. Note that this field is defined as an octet array in the default libfixbuf Information Model. yaf uses the length override feature in libfixbuf to redefine it from variable length to 3 bytes.

subTemplateMultiList IE 293, variable length

Represents a list of zero or more instances of a structured data type, where the data type of each list element can be different and corresponds with different template definitions. The Information Element Number will change upon updates to the IPFIX lists specification and libfixbuf releases.

Hooks Templates

yaf can export other templates within the subTemplateMultiList if plugins are enabled in yaf. See yafdpi(1) for descriptions of the yaf Deep Packet Inspection Information Elements. See yafdhcp(1) for descriptions of the DHCP Fingerprint Information Elements.

Flow Statistics Template

yaf can maintain and export more information about each flow than what is exported in the Basic Flow Template. If yaf is run with --flow-stats yaf will export the following attributes with every flow as long as one of the following characteristics is nonzero. The following flow attributes have been known to help in traffic classification.

Statistics Options Template

yaf will export information about its process periodically using IPFIX Options Template Record. This record gives information about the status of the flow and fragment table, as well as decoding information. This can be turned off using the --no-stats option. The following Information Elements will be exported:

systemInitTimeMilliseconds IE 161, 8 octets, unsigned: The time in milliseconds of the last (re-)initialization of yaf.
exportedFlowRecordTotalCount IE 42, 8 octets, unsigned: Total amount of exported flows from yaf start time.
packetTotalCount IE 86, 8 octets, unsigned: Total amount of packets processed by yaf from yaf start time.
droppedPacketTotalCount IE 135, 8 octets, unsigned: Total amount of dropped packets according to statistics given by libpcap, libdag, or the Napatech or Netronome APIs.
ignoredPacketTotalCount IE 164, 8 octets, unsigned: Total amount of packets ignored by the yaf packet decoder, such as unsupported packet types and incomplete headers, from yaf start time.
notSentPacketTotalCount IE 167, 8 octets, unsigned: Total amount of packets rejected by yaf because they were received out of sequence.
yafExpiredFragmentCount CERT (PEN 6871) IE 100, 4 octets, unsigned: Total amount of fragments that have been expired since yaf start time.
yafAssembledFragmentCount CERT (PEN 6871) IE 101, 4 octets, unsigned: Total number of packets that been assembled from a series of fragments since yaf start time.
yafFlowTableFlushEventCount CERT (PEN 6871) IE 104, 4 octets, unsigned: Total number of times the yaf flow table has been flushed since yaf start time.
yafFlowTablePeakCount CERT (PEN 6871) IE 105, 4 octets, unsigned: The maximum number of flows in the yaf flow table at any one time since yaf start time.
exporterIPv4Address IE 130, 4 octets, unsigned: The IPv4 Address of the yaf flow sensor.
exportingProcessId IE 144, 4 octets, unsigned: Set the ID of the yaf flow sensor by giving a value to --observation-domain. The default is 0.
yafMeanFlowRate CERT (PEN 6871) IE 102, 4 octets, unsigned: The mean flow rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.
yafMeanPacketRate CERT (PEN 6871) IE 103, 4 octets, unsigned: The mean packet rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.

Tombstone Options Template

yaf will export tombstone records periodically using IPFIX Options Template Records. These records are intended to allow the analysis of the time it takes for records to be processed by each tool (eg. YAF, Super Mediator, SiLK) in your environment. Each tombstone record generated by yaf consists of six information elements: the observation domain ID set by the --observation- domain argument, the exporting process ID which is the PID of the YAF process, a user- settable tombstone ID, a sequentially increasing "tombstoneID" for each record, the timestamp of the record's creation, and a subTemplateList of the time each program interacted with the tombstone record. With ideal randomness and/or proper user arguments, the 4 IDs taken together should uniquely specify a record. Tombstone records are only active when stats are active and can be individually turned off using the --no-tombstone option.

The following Information Elements will be exported:

observationDomainId IE 149, 4 octets, unsigned: The (user-set) observation domain of the YAF sensor.
exportingProcessId IE 144, 2 octets, unsigned: The PID of the YAF sensor.
certToolExporterConfiguredId CERT (PEN 6871) IE 551, 2 octets, unsigned: An identification number for the record that is user specifiable at runtime and shared across all records in a run of the given program.
certToolTombstoneId CERT (PEN 6871) IE 550, 4 octets, unsigned: A sequentially increasing identification number unique to each tombstone record in a run of a given program.
observationTimeSeconds IE 322, 4 octets, dateTimeSeconds: The UNIX timestamp of when the record was created.
certToolTombstoneAccessList CERT (PEN 6871) IE 554, variable length, subTemplateList: A subTemplateList consisting of Tombstone Access Templates (see below) that specify when each program that supports tombstone times-tamping interacted with the tombstone record.

Tombstone Access Template

The following two Information Elements will be exported as a template within the subTemplateList of a Tombstone Options Template:

certToolId CERT (PEN 6871) IE 553, 4 octets, unsigned: The identification number of the program that interacted with the record. yaf has and ID of 1.
observationTimeSeconds IE 322, 4 octets, dateTimeSeconds: The UNIX timestamp of when the program interacted with the record.

SIGNALS

yaf responds to SIGINT or SIGTERM by terminating input processing, flushing any pending flows to the current output, and exiting. If --verbose is given, yaf responds to SIGUSR1 by printing present flow and fragment table statistics to its log. All other signals are handled by the C runtimes in the default manner on the platform on which yaf is currently operating.

EXAMPLES

To generate flows from an pcap file into an IPFIX file:

yaf --in packets.pcap --out flows.yaf

To capture flows from a pcap interface and export them to files in the current directory rotated hourly:

yaf --live pcap --in en1 --out en1_capture --rotate 3600

To capture flows from an Endace DAG card and export them via IPFIX over TCP:

yaf --live dag --in dag0 --ipfix tcp --out my-collector.example.com

To capture flows from a Napatech Adapter card using stream ID 20 and export them via IPFIX over UDP:

yaf --live napatech --in nt3g20 --ipfix udp --out localhost --ipfix-port 18000

To capture flows from a Netronome NFE card and export to a file:

yaf --live netronome --in 0:0 --out /data/yaf/myipfix.yaf

To convert a pcap formatted packet capture into IPFIX:

yaf <packets.pcap >flows.yaf

To run yaf with application labeling enabled and export via IPFIX over TCP:

yaf --live pcap --in eth1 --out 127.0.0.1 --ipfix tcp --ipfix-port=18001 --applabel --dpi-rules-files=/usr/local/etc/yafDPIRules.conf --max-payload=300

To run yaf with BPF on UDP Port 53

yaf --live pcap --in en1 --out /path/to/dst/ --rotate 120 --filter="udp port 53"

To run yaf with Deep Packet Inspection enabled for HTTP, IMAP, and DNS:

yaf --in packets.pcap --out flows.yaf --applabel --max-payload=400 --dpi --dpi-select="80 143 53"

To run yaf with DHCP Fingerprinting:

yaf --in packets.pcap --out flows.yaf --applabel --max-payload=1000 --plugin-name=/usr/local/lib/dhcp_fp_plugin.la

To run yaf with pcap generation:

yaf --in eth0 --live pcap --out localhost --ipfix tcp --ipfix-port=18001 --pcap /data/pcap --pcap-meta-file=/data/pcap_info

To generate a pcap file for one particular flow in a pcap:

yaf --in packets.pcap --no-output --max-payload=2000 --pcap /data/oneflow.pcap --hash 2181525080 --stime 1407607897547

KNOWN ISSUES

YAF BPF Filtering is ignored when using --live dag, napatech, or netronome because libpcap is not used.

YAF PCAP Export options are ignored when using --live dag, napatech, or netronome.

YAF requires libfixbuf 3.0.0 or later.

YAF 2.0 must be used with an IPFIX Collecting Process that can handle IPFIX lists elements, especially the subTemplateMultiList Information Element in order to retrieve certain flow information. Older versions of YAF can read YAF 2.0 flow files, but will ignore anything contained in the subTemplateMultiList.

The plugin infrastructure has been modified in YAF 2.0 to export templates in YAF's subTemplateMultiList element.

YAF 2.0 will export statistics in an Options Template and Options Data Records unless the --no-stats switch is given. The IPFIX Collecting Process should be able to differentiate between options records and flow records in order to prevent incorrect transcoding of statistics records into flow records.

YAF will not rotate output files if it is not seeing any flow data. However, it will continue to write process statistics messages at the configured interval time to the most recent output file.

When using PF_RING ZC with yaf, a load balancing program is required. See yafzcbalance(1) for more information.

When running yaf with --live=pfring or --live=zc, the call to receive packets is blocking so yaf will not export statistics messages or respond to SIGUSR1 signals unless it is receiving data.

Presently, the destinationTransportPort information element contains ICMP type and code information for ICMP or ICMP6 flows; this is nonstandard and may not be interoperable with other IPFIX implementations.

Bug reports and feature requests may be sent via email to <netsa-help@cert.org>.

AUTHORS

Brian Trammell, Chris Inacio, Michael Duggan, Emily Sarneso, Dan Ruef, Matt Coates, Dillon Lareau and the CERT Network Situational Awareness Group Engineering Team, http://www.cert.org/netsa.