CERT
Software Assurance Secure Systems Organizational Security Coordinating Response Training
Child pages
  • YAF 2.x Mediator (YAF to MySQL, SiLK)
Skip to end of metadata
Go to start of metadata

Description

The following program will read YAF IPFIX files or listen for connections from YAF on a given port using libfixbuf and writes the flow and Deep Packet Inspection (DPI) data to a MySQL database and/or export flow records in SiLK format to a SiLK collector (flowcap, rwflowpack).  This program is intended to read IPFIX files generated from YAF 2.0 or later.  It will read IPFIX files from earlier versions, however, it will not parse any flow data that is now contained in YAF's subTemplateMultiList.  This program is able to read all DPI elements exported from YAF's DPI plugin-in and DHCP fingerprinting information.  For DPI, YAF must have been configured with --enable-plugins and run with the DPI plugin (dpacketplugin.la) as the argument to --plugin-name.  The following program requires glib 2.12 or later, libfixbuf 1.0.0 or later, and the mysqlclient libraries.  In order to use the included CMake configuration file, CMake version 2.8 or later is required.

Source Code

Updated for use with YAF 2.3 (Updated 5/21/13 with Bug Fix for importing DNS to MySQL) :

yaf_silk_mysql_mediator-1.4.1.tar.gz

Building

First, make sure you have libfixbuf 1.0.0 (or greater) and the mysqlcient libraries installed.

./configure
make
make install

 

If you choose to use CMake and have CMake installed you can use the included configuration file.  You may need to set PKG_CONFIG_PATH to the location of the libfixbuf.pc:

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
cmake .
make

This will build both the mediator and the MySQL Table Builder (yafMySQL.c). 

It may be necessary to set the CMAKE_LIBRARY_PATH environment variable to the location of the MySQL libraries.

Running

Run yafMySQL first to build the necessary MySQL Tables:

./yafMySQL --out localhost --name root --pass password --database eflows

The main table is the "flows" table.  This will contain all flow data.  Each flow is given a unique ID as the primary key.  Most DPI tables will have 3 columns: id, listType, listTypeValue.  Use the silkAppLabel field in the flow table with the ID to find which table contains the DPI information, if any, for that flow.  For example, if a flow with id 555 has a silkAppLabel of 80, you will be able to find any available DPI data in the http table by querying "select * from http where id = 555;".  

listType is an integer that corresponds with the Information Element ID.  listTypeValue is the DPI text data that YAF has captured and exported.  To view all listTypes (Information Element ID's) see here or run the following (optional):

./yafMySQL --insert-index --out localhost --name root --pass password --database eflows

This will create a "dpi_index" table from this index.txt file that contains all DPI Information Element ID's and their corresponding Information Element Names.  However, MySQL must have the --local-infile option enabled to load this file into the database or you will get the error "Error Importing Index Rows.  The used command is not allowed with this MySQL version."

Running the mediator:

File as input:
./yaf_silk_mysql_mediator --in-file yaf_ipfix.yaf --mysql-host localhost --name root --pass password --database eflows

The above will take the yaf_ipfix.yaf IPFIX file and import all the flow and DPI data into the mysql database "eflows" running on localhost.

To also export the flow data to a SiLK flow collector running on localhost listening to port 18001, use instead:

./yaf_silk_mysql_mediator --in-file yaf_ipfix.yaf --mysql-host localhost --name root --pass password --database eflows --out-host localhost --out-port 18001
Listen for connections from YAF:
./yaf_silk_mysql_mediator --in-port 18000 --in-host localhost --mysql-host localhost --name root --pass password --database flows --out-host localhost --out-port 18001

The above will listen for TCP connections from YAF on port 18000 running on localhost and will export flow and DPI data to the "eflows" database and export IPFIX in SiLK format to flowcap or rwflowpack running on localhost listening for TCP connections on 18001.

Starting YAF with Deep Packet Inspection:

yaf --in eth0 --out localhost --ipfix tcp --ipfix-port 18000 --live pcap --applabel --max-payload 1500 --plugin-name=/usr/local/lib/yaf/dpacketplugin.la --verbose

YAF must have been configured with --enable-applabel and --enable-plugins to use the above or below command line arguments.

Starting YAF with DPI for DNS only:

yaf --in eth0 --out localhost --ipfix tcp --ipfix-port 18000 --live pcap --applabel --max-payload 1500 --plugin-name=/usr/local/lib/yaf/dpacketplugin.la --verbose --udp-uniflow=53 --filter="port 53"

The --udp-uniflow option emits a flow for each DNS packet individually in order to perform DPI on all DNS records.  The --filter is a BPF filter to only capture DNS traffic.  Alternatively, you can use --plugin-opts=53 if you want DPI performed on DNS flows only and still create flow records for all other traffic.

Database Examples:

Below are a few sample SQL queries:

To view 10 "HTTP" Flows:

SELECT inet_ntoa(srcip4), inet_ntoa(dstip4), srcport, dstport, protocol, octetTotalCount, packetTotalCount FROM flows WHERE silkAppLabel = 80 LIMIT 10;

To view all HTTP DPI Data that contain the word "Mozilla":

SELECT * FROM http WHERE listTypeValue LIKE "%Mozilla%";

To view all HTTP User Agent Strings:

SELECT listTypeValue FROM http WHERE listType=111;

To view last 10 DNS Queries imported into the database:

SELECT rrname FROM dns d, flows f WHERE f.id = d.id ORDER BY f.id DESC LIMIT 10;

To view all HTTP Data and corresponding Information Element Names:

SELECT d.name, h.listTypeValue FROM dpi_index d, http h WHERE h.listType = d.id;

To view all captured SSL Certificates:

SELECT * FROM tls;

Known Issues

Currently IPv6 is not supported.  However, DNS AAAA records will be imported into the database.

Contact

Please email feedback, bug reports, and questions to netsa-help@cert.org

  • No labels