The following program will read YAF IPFIX files or listen for connections from YAF on a given port using libfixbuf and writes the flow and Deep Packet Inspection (DPI) data to a MySQL database and/or export flow records in SiLK format to a SiLK collector (flowcap, rwflowpack). This program is intended to read IPFIX files generated from YAF 2.0 or later. It will read IPFIX files from earlier versions, however, it will not parse any flow data that is now contained in YAF's subTemplateMultiList. This program is able to read all DPI elements exported from YAF's DPI plugin-in and DHCP fingerprinting information. For DPI, YAF must have been configured with --enable-plugins and run with the DPI plugin (dpacketplugin.la) as the argument to --plugin-name. The following program requires glib 2.12 or later, libfixbuf 1.0.0 or later, and the mysqlclient libraries. In order to use the included CMake configuration file, CMake version 2.8 or later is required.
Updated for use with YAF 2.3 (Updated 5/21/13 with Bug Fix for importing DNS to MySQL) :
First, make sure you have libfixbuf 1.0.0 (or greater) and the mysqlcient libraries installed.
If you choose to use CMake and have CMake installed you can use the included configuration file. You may need to set PKG_CONFIG_PATH to the location of the libfixbuf.pc:
This will build both the mediator and the MySQL Table Builder (yafMySQL.c).
It may be necessary to set the CMAKE_LIBRARY_PATH environment variable to the location of the MySQL libraries.
Run yafMySQL first to build the necessary MySQL Tables:
The main table is the "flows" table. This will contain all flow data. Each flow is given a unique ID as the primary key. Most DPI tables will have 3 columns: id, listType, listTypeValue. Use the silkAppLabel field in the flow table with the ID to find which table contains the DPI information, if any, for that flow. For example, if a flow with id 555 has a silkAppLabel of 80, you will be able to find any available DPI data in the http table by querying "select * from http where id = 555;".
listType is an integer that corresponds with the Information Element ID. listTypeValue is the DPI text data that YAF has captured and exported. To view all listTypes (Information Element ID's) see here or run the following (optional):
This will create a "dpi_index" table from this index.txt file that contains all DPI Information Element ID's and their corresponding Information Element Names. However, MySQL must have the --local-infile option enabled to load this file into the database or you will get the error "Error Importing Index Rows. The used command is not allowed with this MySQL version."
Running the mediator:
File as input:
The above will take the yaf_ipfix.yaf IPFIX file and import all the flow and DPI data into the mysql database "eflows" running on localhost.
To also export the flow data to a SiLK flow collector running on localhost listening to port 18001, use instead:
Listen for connections from YAF:
The above will listen for TCP connections from YAF on port 18000 running on localhost and will export flow and DPI data to the "eflows" database and export IPFIX in SiLK format to flowcap or rwflowpack running on localhost listening for TCP connections on 18001.
Starting YAF with Deep Packet Inspection:
YAF must have been configured with --enable-applabel and --enable-plugins to use the above or below command line arguments.
Starting YAF with DPI for DNS only:
The --udp-uniflow option emits a flow for each DNS packet individually in order to perform DPI on all DNS records. The --filter is a BPF filter to only capture DNS traffic. Alternatively, you can use --plugin-opts=53 if you want DPI performed on DNS flows only and still create flow records for all other traffic.
Below are a few sample SQL queries:
To view 10 "HTTP" Flows:
To view all HTTP DPI Data that contain the word "Mozilla":
To view all HTTP User Agent Strings:
To view last 10 DNS Queries imported into the database:
To view all HTTP Data and corresponding Information Element Names:
To view all captured SSL Certificates:
Currently IPv6 is not supported. However, DNS AAAA records will be imported into the database.
Please email feedback, bug reports, and questions to firstname.lastname@example.org