Overview

Why does the world need another network flow event generator? yaf was originally intended as an experimental implementation tracking developments in the IETF IPFIX working group, specifically bidirectional flow representation, archival storage formats, and structured data export with Deep Packet Inspection. It is designed to perform acceptably as a flow sensor on any network on which white-box flow collection with commodity hardware is appropriate. yaf can and should be used on specialty hardware when scalability and performance are of concern.

Tool Suite

The YAF toolchain presently consists of two primary tools, yaf itself, and yafscii. The YAF applications require the libairframe and libyaf libraries, which are included and installed as part of the YAF distribution. libairframe installs two additional tools, filedaemon and airdaemon. libyaf implements YAF file and network I/O, and contains YAF packet decoder, fragment assembler, and flow table. In addition, two tools to assist in PCAP analysis are also installed with YAF.

Core Tools

yaf

Yet Another Flowmeter. Processes packet data from pcap(3) dumpfiles as generated by tcpdump(1) or via live captures from an interface into bidirectional flows, then exports those flows to IPFIX Collecting Processes or in an IPFIX-based file format.

yafscii

YAF Flow printer. yafscii takes IPFIX flow data files generated by yaf and prints them in ASCII format loosely analogous to that produced by tcpdump(1), with one flow per line.

YAF PCAP tools

yafMeta2Pcap

yaf PCAP metadata file parser and PCAP file creator. yafMeta2Pcap takes the metadata files produced by yaf and some additional flow information provided by getFlowKeyHash to create PCAP files for a particular flow.

getFlowKeyHash

yaf flow key calculator. getFlowKeyHash takes information about a flow and calculates the yaf flow key hash and prints the flow records along with the hash to the screen. In addition, it will convert the flow's start time to milliseconds since Epoch time. Together, the flow key hash and the start time can be used as a unique identifier for a flow.

libairframe tools

airdaemon

Run a program as a daemon process, restarting the program if it dies.

filedaemon

filedaemon can invoke another program on files matching a glob pattern. It is often used to poll a directory and move files from one directory to another.

IPFIX tools

ipfixDump

Dump the contents of an IPFIX file to text.

PF_RING tools

yafzcbalance

Load balance the traffic from one or more zc interfaces to yaf processes.

Manuals

The following manuals provide general information about specific features of yaf. These features are not enabled by default, and require them to be enabled at compile time.

Configuration File

Information about the yaf configuration file which is an alternative to configuring yaf with command line options. The syntax of the configuration file is explained by examples.

Application labeling

Information about yaf application labeling, signature detection, and setting up the configuration file. Also provides a table of current application labels.

Deep Packet Inspection

Provides information about the deep packet inspection capabilities in yaf, including the available protocols, setting up the configuration file, and export fields.

DHCP Fingerprinting

yaf DHCP fingerprinting capability information. Provides information on how to use it, the configuration file, and the fields exported.

Dependencies

yaf requires glib 2.6.4 or later. Build and install glib before building YAF. Note that glib is also included in many operating environments or ports collections. If installing via rpm, please note that you must install the glib2-devel package as well.

yaf requires libpcap. Note that libpcap is included with many operating environments or ports collections. If installing via rpm, please note that you must install the libpcap-devel package as well.

yaf can process compressed PCAP files if the zlib library is installed and yaf is run with the --decompress option.

yaf requires libfixbuf 1.7.0 or later.

yaf provides support for the Endace/Emulex, Napatech, and Netronome capture cards. yaf can be configured to use the custom libpcap on these cards by using the --with-libpcap option or by setting CFLAGS and LDFLAGS when configuring yaf. However, if yaf is compiled with libdag, libntapi, or NFM and the appropriate name is given to --live, yaf, by default, will record the physical interface the packet was received on. To export these values, use the --export-interface option when running yaf. Interface values can be used to determine directionality of a flow in some cases. To disable interface collection, configure yaf with --enable-interface=no. To separate traffic received on separate ports into different flows, use the --enable-daginterface option when configuring yaf.

Endace DAG live input support requires libdag. Use the --with-dag option to ./configure to enable DAG support. Standard interface recording is enabled by default when running yaf with --live=dag.

Napatech live input support requires libntapi and the 3rd generation Napatech drivers. Use the --with-napatech option to ./configure to enable Napatech support. Standard interface recording is enabled by default when running yaf with --live=napatech.

Netronome live input support requires the Netronome Flow Manager (NFM) which includes the NFM PCAP library and NFM software. Use the --with-netronome option to ./configure to enable Netronome support. Standard interface recording is enabled by default when running yaf with --live=netronome.

Support for Bivio interface labeling requires yaf to be configured with --with-bivio.

Support for application labeling requires PCRE 7.3 or later. Build and install PCRE before building YAF. (Many Linux systems already have PCRE installed.) If installing via rpm, you must install the pcre-devel package. Support for application labeling requires giving the --enable-applabel option to ./configure.

Support for p0f requires libp0f. Build and install libp0f before building YAF. You may need to set the PKG_CONFIG_PATH environment variable if libp0f is not installed in the default location.

Spread support requires Spread 4.1 or later. Build and install Spread before building YAF.

yaf contains support for PF_RING and PF_RING ZC (ZERO COPY) if yaf is compiled with libpfring by giving the --with-pfring option to ./configure. PF_RING is available through ntop. Download and install PF_RING (v.6.2.0 or higher) before installing yaf. Install the PF_RING kernel modules, drivers, and library. PF_RING ZC requires a license purchase through ntop. To use PF_RING ZC, you are required to run yafzcbalance, a tool installed with yaf, or a similar application which will load balance the traffic on one or more interfaces to one or more yaf applications.

Building YAF

yaf uses a reasonably standard autotools-based build system. The customary build procedure (./configure && make && make install) should work in most environments. Note that yaf finds libfixbuf and libairframe using the pkg-config facility, so you may have to set the PKG_CONFIG_PATH variable on the configure command line if these libraries are installed in a nonstandard location, other than the prefix to which you are installing yaf itself.

Support for application labeling requires giving the --enable-applabel option to ./configure.

Support for p0f requires giving the --enable-p0fprinter and --enable-applabel options to ./configure.

Deep Packet Inspection (DPI) requires plugin support. Use the --enable-plugins option to ./configure.

yaf can generate Multiprotocol Label Switching (MPLS)-Aware flow data. yaf exports the first three MPLS labels from the label stack along with the traditional flow data. When this feature is enabled, yaf will also export non-IP flow data. To enable MPLS-aware flow, use the --enable-mpls to ./configure.

Tutorials

How-to guide on configuring yaf with SiLK.
How-to guide on using yaf to index large PCAP files. Basic flow analysis with SiLK and other tools will be discussed.
How-to guide on enabling rolling PCAP and metadata index generation in yaf. Analysis with SiLK and other tools will be discussed.
How-to guide on using yaf and super_mediator to collect DPI data and import that data into a MySQL database. SiLK flow collection will also be described.
How-to guide on using yaf and Orcus to collect DNS information and import that data into a PostgreSQL database for analysis. super_mediator DNS deduplication will also be described.

Papers and Presentations

Finding a Needle in a PCAP

This presentation from FloCon 2015 describes how to use yaf and super_mediator to index large PCAP files. A possible method for identifying and classifying malware is also presented.

YAF: Yet Another Flowmeter

yaf presented at LISA'10 Proceedings.

YAF: Yet Another Flowmeter

yaf presention slides from LISA'10 Proceedings.

Known Issues and Other Information

As of yaf 2.0.0, yaf uses a subTemplateMultiList to export certain flow elements. See yaf for more information. Older versions of yaf can read yaf 2.0 flow files, but will ignore anything contained in the subTemplateMultiList.

It is suggested to use --silk when running yaf with SiLK. If SiLK version 2.x is used, --silk and --no-stats should be used due to how yaf exports TCP flow information and yaf process statistics.

Presently, the destinationTransportPort information element contains ICMP type and code information for ICMP or ICMP6 flows; this is nonstandard and may not be interoperable with other IPFIX implementations.

By default YAF generates flows based on the standard 5-tuple and VLAN tag, if available. The 5-tuple consists of the source IP address, destination IP address, source port, destination port, and protocol. If YAF is configured with MPLS support --enable-mpls, YAF will use the top three MPLS labels from the MPLS label stack in addition to the 5-tuple and vlan to determine the flow. In MPLS mode, it will also export the top three MPLS labels in the IPFIX record. The exported fields will not include the experimental bits and the bottom of stack bit. In addition, if YAF is configured with --enable-nonip, YAF will accept non-IP data and generate flow data using just the MPLS labels. The 5-tuple and VLAN fields will be set to 0, and the exported flow will contain start and end times, packet counts, byte counts, and MPLS labels. Since the byte count is typically taken from the length in the IP header,YAF will use the length provided by libpcap. Non-IP data can only be exported if MPLS mode is enabled.

Contact

Please send bug reports, feature requests, and questions to contact_email. We welcome bug fixes and patches.