The source code below provides an example of using the fixbuf library. The source code is for libfixbuf 1.x; it has not been modified to support libfixbuf 2.x.
Once the code is compiled, the yaf_silk_mysql_mediator program reads YAF IPFIX files or listens for connections from YAF on a given port and writes the flow and Deep Packet Inspection (DPI) data to a MySQL database and/or exports the flow records in SiLK format to a SiLK collector (flowcap, rwflowpack). The program is able to read DPI elements exported from YAF's DPI plugin-in and DHCP fingerprinting information.
For YAF to export DPI information, it must have been configured with --enable-plugins when it was built. You may check whether it was by running yaf --version. When running YAF, include the DPI plugin (dpacketplugin.la) as the argument to --plugin-name. See the YAF manual page for details.
The following program requires GLib 2.12 or later, libfixbuf 1.x, and the mysqlclient libraries. In order to use the included CMake configuration file, CMake version 2.8 or later is required.
Source Code
(SHA256=37aae59131aab20026a95f789017b765540c82cd466a4083e8239b82e79b2b84)
Building
First, make sure you have libfixbuf 1.x (not 2.x) and the mysqlcient
libraries installed. You may need to set the PKG_CONFIG_PATH
environment variable to the location of the libfixbuf.pc
file. (If you used a package manager to install libfixbuf or
mysqlcient, ensure that the development headers and libraries are
installed. These may be in a separate package with a
-devel or -dev suffix.)
To use the configure script:
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure make make install
If instead you choose to use CMake and have CMake installed you can use the included configuration file. It may be necessary to set the CMAKE_LIBRARY_PATH environment variable to the location of the MySQL libraries.
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig cmake . make
Either of the above build both the mediator and the MySQL Table Builder (yafMySQL.c).
Running
Creating the database tables:
Run yafMySQL first to build the necessary MySQL Tables:
./yafMySQL --out localhost --name root --pass password --database eflows
The main table is the "flows" table. This will contain all
flow data. Each flow is given a unique ID as the primary key. Most
DPI tables will have 3 columns: id, listType, listTypeValue. Use the
silkAppLabel field in the flow table with the ID to find which table
contains the DPI information, if any, for that flow. For example, if
a flow with id 555 has a silkAppLabel of 80, you will be able to find
any available DPI data in the http table by querying
select * from http where id = 555;
listType is an integer that corresponds with the Information Element ID. listTypeValue is the DPI text data that YAF has captured and exported. To view all listTypes (Information Element ID's) see here or run the following (optional):
./yafMySQL --insert-index --out localhost --name root
--pass password --database eflows
This will create a "dpi_index" table from this index.txt file that contains all DPI Information Element ID's and their corresponding Information Element Names. However, MySQL must have the --local-infile option enabled to load this file into the database or you will get the error "Error Importing Index Rows. The used command is not allowed with this MySQL version."
Running the mediator:
File as input:
./yaf_silk_mysql_mediator --in-file yaf_ipfix.yaf \
--mysql-host localhost --name root --pass password \
--database eflows
The above will take the yaf_ipfix.yaf IPFIX file and import all the flow and DPI data into the mysql database "eflows" running on localhost.
To also export the flow data to a SiLK flow collector running on localhost listening to port 18001, use instead:
./yaf_silk_mysql_mediator --in-file yaf_ipfix.yaf \
--mysql-host localhost --name root --pass password \
--database eflows --out-host localhost --out-port 18001
Listen for connections from YAF:
./yaf_silk_mysql_mediator --in-port 18000 --in-host localhost \
--mysql-host localhost --name root --pass password \
--database flows --out-host localhost --out-port 18001
The above will listen for TCP connections from YAF on port 18000 running on localhost and will export flow and DPI data to the "eflows" database and export IPFIX in SiLK format to flowcap or rwflowpack running on localhost listening for TCP connections on 18001.
Starting YAF:
Starting YAF with Deep Packet Inspection:
yaf --in eth0 --out localhost --ipfix tcp --ipfix-port 18000 \
--live pcap --applabel --max-payload 1500 \
--plugin-name=/usr/local/lib/yaf/dpacketplugin.la --verbose
YAF must have been configured with --enable-applabel and --enable-plugins to use the above or below command line arguments.
Starting YAF with DPI for DNS only:
yaf --in eth0 --out localhost --ipfix tcp --ipfix-port 18000 \
--live pcap --applabel --max-payload 1500 \
--plugin-name=/usr/local/lib/yaf/dpacketplugin.la --verbose \
--udp-uniflow=53 --filter="port 53"
The --udp-uniflow option emits a flow for each DNS packet individually in order to perform DPI on all DNS records. The --filter is a BPF filter to only capture DNS traffic. Alternatively, you can use --plugin-opts=53 if you want DPI performed on DNS flows only and still create flow records for all other traffic.
Database Examples
Below are a few sample SQL queries:
To view 10 "HTTP" Flows:
SELECT inet_ntoa(srcip4), inet_ntoa(dstip4),
srcport, dstport, protocol, octetTotalCount, packetTotalCount
FROM flows WHERE silkAppLabel = 80 LIMIT 10;
To view all HTTP DPI Data that contain the word "Mozilla":
SELECT * FROM http WHERE listTypeValue LIKE "%Mozilla%";
To view all HTTP User Agent Strings:
SELECT listTypeValue FROM http WHERE listType=111;
To view last 10 DNS Queries imported into the database:
SELECT rrname FROM dns d, flows f
WHERE f.id = d.id ORDER BY f.id DESC LIMIT 10;
To view all HTTP Data and corresponding Information Element Names:
SELECT d.name, h.listTypeValue FROM dpi_index d, http h
WHERE h.listType = d.id;
To view all captured SSL Certificates:
SELECT * FROM tls;
Known Issues
The source code has not been updated to support libfixbuf 2.x.
Currently IPv6 is not supported. However, DNS AAAA records will be imported into the database.
