Contents
- Overview
- Tool Suite - overview of the tools
- Manuals - configuration, application labeling, DPI, DHCP Fingerprinting
- Dependencies
- Building
- Tutorials
- Related Tools and Documentation - IPFIX mediators
- Papers and Presentations
- Known Issues and Other Information
- Contact
Overview
Why does the world need another network flow event generator? yaf was originally intended as an experimental implementation tracking developments in the IETF IPFIX working group, specifically bidirectional flow representation, archival storage formats, and structured data export with Deep Packet Inspection. It is designed to perform acceptably as a flow sensor on any network on which white-box flow collection with commodity hardware is appropriate. yaf can and should be used on specialty hardware when scalability and performance are of concern.
Tool Suite
The YAF toolchain presently consists of two primary tools, yaf itself, and yafscii. The YAF applications require the libairframe and libyaf libraries, which are included and installed as part of the YAF distribution. libairframe installs two additional tools, filedaemon and airdaemon. libyaf implements YAF file and network I/O, and contains YAF packet decoder, fragment assembler, and flow table. In addition, two tools to assist in PCAP analysis are also installed with YAF.
Core Tools
|
Yet Another Flowmeter. Processes packet data from pcap(3) dumpfiles as generated by tcpdump(1) or via live captures from an interface into bidirectional flows, then exports those flows to IPFIX Collecting Processes or in an IPFIX-based file format. |
|
|
YAF Flow printer. yafscii takes IPFIX flow data files generated by yaf and prints them in ASCII format loosely analogous to that produced by tcpdump(1), with one flow per line. yafscii is unable to print deep packet inspection information generated by yaf. To view that, use the ipfixDump utility that is distributed with libfixbuf. |
YAF PCAP tools
|
yaf PCAP metadata file parser and PCAP file creator. yafMeta2Pcap takes the metadata files produced by yaf and some additional flow information provided by getFlowKeyHash to create PCAP files for a particular flow. |
|
|
yaf flow key calculator. getFlowKeyHash takes information about a flow and calculates the yaf flow key hash and prints the flow records along with the hash to the screen. In addition, it will convert the flow's start time to milliseconds since Epoch time. Together, the flow key hash and the start time can be used as a unique identifier for a flow. |
libairframe tools
|
airdaemon runs a program as a daemon process, restarting the program if it dies. |
|
|
filedaemon can invoke another program on files matching a glob pattern. It is often used to poll a directory and move files from one directory to another. |
PF_RING tools
|
Load balance the traffic from one or more zc interfaces to yaf processes. |
Manuals
The following manuals provide general information about specific features of yaf. These features are not enabled by default, and require them to be enabled at compile time.
|
Information about the yaf configuration file which is an alternative to configuring yaf with command line options. The syntax of the configuration file is explained by examples. |
|
|
Information about yaf application labeling, signature detection, and setting up the configuration file. Also provides a table of current application labels. |
|
|
Provides information about the deep packet inspection capabilities in yaf, including the available protocols, setting up the configuration file, and export fields. |
|
|
yaf DHCP fingerprinting capability information. Provides information on how to use it, the configuration file, and the fields exported. |
Dependencies
yaf requires glib 2.18.0 or later. Build and install glib before building YAF. Note that glib is also included in many operating environments or ports collections. If installing via rpm, please note that you must install the glib2-devel package as well.
yaf requires libpcap. Note that libpcap is included with many operating environments or ports collections. If installing via rpm, please note that you must install the libpcap-devel package as well.
yaf can process compressed PCAP files if the zlib library is installed.
yaf requires libfixbuf 2.3.0 or later.
yaf provides support for the Endace/Emulex, Napatech, and
Netronome capture cards. yaf can be configured to use the custom
libpcap on these cards by using the --with-libpcap option
or by setting CFLAGS and LDFLAGS when configuring yaf. However,
if yaf is compiled with libdag, libntapi, or NFM and the
appropriate name is given to --live, yaf, by default, will
record the physical interface the packet was received on. To export
these values, use the --export-interface option when
running yaf. Interface values can be used to determine
directionality of a flow in some cases. To disable interface
collection, configure yaf with
--enable-interface=no. To separate traffic received on
separate ports into different flows, use the
--enable-daginterface option when configuring yaf.
Endace DAG live input support requires libdag. Use the
--with-dag option to ./configure to enable DAG
support. Standard interface recording is enabled by default when
running yaf with --live=dag.
Napatech live input support requires libntapi and the 3rd generation
Napatech drivers. Use the --with-napatech option to
./configure to enable Napatech support. Standard interface
recording is enabled by default when running yaf with
--live=napatech.
Netronome live input support requires the Netronome Flow Manager (NFM)
which includes the NFM PCAP library and NFM software. Use the
--with-netronome option to ./configure to
enable Netronome support. Standard interface recording is enabled by
default when running yaf with --live=netronome.
Support for Bivio interface labeling requires yaf to be
configured with --with-bivio.
Support for application labeling requires PCRE 7.3 or later. Build and install
PCRE before building YAF. (Many Linux systems already have PCRE
installed.) If installing via rpm, you must install the pcre-devel
package. Support for application labeling requires giving the
--enable-applabel option to ./configure.
Support for p0f requires libp0f. Build and install libp0f before building YAF. You may need to set the PKG_CONFIG_PATH environment variable if libp0f is not installed in the default location.
Spread support requires Spread 4.1 or later. Build and install Spread before building YAF.
yaf contains support for PF_RING and PF_RING ZC (ZERO COPY) if
yaf is compiled with libpfring by giving the
--with-pfring option to ./configure. PF_RING
is available through ntop. Download and
install PF_RING (v.6.2.0 or higher) before installing yaf.
Install the PF_RING kernel modules, drivers, and library. PF_RING ZC
requires a license purchase through ntop. To use PF_RING ZC, you are
required to run yafzcbalance, a tool
installed with yaf, or a similar application which will load
balance the traffic on one or more interfaces to one or more yaf
applications.
Building YAF
yaf uses a reasonably standard autotools-based build system. The
customary build procedure
(./configure && make && make install)
should work in most environments. Note that yaf finds libfixbuf
and libairframe using the pkg-config facility,
so you may have to set the PKG_CONFIG_PATH variable on the
configure command line if these libraries are installed in a nonstandard
location, other than the prefix to which you are installing yaf
itself.
Support for application labeling requires giving the
--enable-applabel option to ./configure.
Support for p0f requires giving the --enable-p0fprinter and
--enable-applabel options to ./configure.
Deep Packet Inspection (DPI) requires plugin support. Use the
--enable-plugins option to ./configure.
yaf can generate Multiprotocol Label Switching (MPLS)-Aware flow
data. yaf exports the first three MPLS labels from the label
stack along with the traditional flow data. When this feature is
enabled, yaf will also export non-IP flow data. To enable
MPLS-aware flow, use the --enable-mpls to
./configure.
Tutorials
Papers and Presentations
This presentation from FloCon 2015 describes how to use yaf and super_mediator to index large PCAP files. A possible method for identifying and classifying malware is also presented.
yaf presented at LISA'10 Proceedings.
yaf presention slides from LISA'10 Proceedings.
Known Issues and Other Information
As of yaf 2.0.0, yaf uses a subTemplateMultiList to export certain flow elements. See yaf for more information. Older versions of yaf can read yaf 2.0 flow files, but will ignore anything contained in the subTemplateMultiList.
It is suggested to use --silk when running yaf with
SiLK. If SiLK version 2.x is used,
--silk and --no-stats should be used due to
how yaf exports TCP flow information and yaf process
statistics.
Presently, the destinationTransportPort information element contains ICMP type and code information for ICMP or ICMP6 flows; this is nonstandard and may not be interoperable with other IPFIX implementations.
By default YAF generates flows based on the standard 5-tuple and VLAN
tag, if available. The 5-tuple consists of the source IP address,
destination IP address, source port, destination port, and protocol. If
YAF is configured with MPLS support --enable-mpls, YAF will
use the top three MPLS labels from the MPLS label stack in addition to
the 5-tuple and vlan to determine the flow. In MPLS mode, it will also
export the top three MPLS labels in the IPFIX record. The exported
fields will not include the experimental bits and the bottom of stack
bit. In addition, if YAF is configured with --enable-nonip,
YAF will accept non-IP data and generate flow data using just the MPLS
labels. The 5-tuple and VLAN fields will be set to 0, and the exported
flow will contain start and end times, packet counts, byte counts, and
MPLS labels. Since the byte count is typically taken from the length in
the IP header,YAF will use the length provided by libpcap. Non-IP data
can only be exported if MPLS mode is enabled.
Contact
Please send bug reports, feature requests, and questions to 
. We welcome bug
fixes and patches.