Overview

super_mediator is an IPFIX mediator for use with the yaf and SiLK tools. What is an IPFIX mediator? An IPFIX mediator is an intermediate entity between IPFIX Exporters and Collectors that can potentially provide aggregation, filtering, and modification of IPFIX records. It may provide conversion to or from IPFIX or a conversion of IPFIX transport protocols. super_mediator collects and processes yaf output (IPFIX files or via TCP, UDP, or Spread) and exports that data in IPFIX, JSON, or CSV text format to one or more IPFIX collectors such as rwflowpack, flowcap, or to text files that may be bulk uploaded to a database. MySQL support is provided for automatic import.

super_mediator can provide simple filtering upon collection or at export time. Any traditional flow field can be used in a filter, including IP address or IPset (requires SiLK IPset library).

super_mediator can be configured to pull the Deep Packet Inspection (DPI) data from yaf and export that information to another IPFIX collector, or simply export the data to a CSV file or JSON file for bulk upload into a database of your choice. Given MySQL credentials, super_mediator will import the files into the given database.

super_mediator can also be configured to perform de-duplication of DNS resource records, DPI data, and SSL/TLS certificate data exported by YAF. It will export the de-duplicated records in IPFIX, CSV, or JSON format. See the man pages and tutorials for more information.

Tools

super_mediator

super_mediator is an IPFIX mediator that provides filtering, de-duplication, and modification of IPFIX records from yaf. It also provides conversion of transport protocols and configurable human-readable, text output.

super_table_creator

super_table_creator builds the MySQL database and tables used for auto insert with super_mediator.

Configuration

The following manual page provides information about how to configure super_mediator. super_mediator can be run from the command line with a limited set of capabilities. For advanced configuration, such as using more than one exporter, the super_mediator.conf is required.

super_mediator.conf

Configuration file for defining collectors and exporters. Filtering and advanced, custom output is also configured through this configuration file.

Dependencies

super_mediator requires glib 2.18.0 or later. Build and install glib before building super_mediator. Note that glib is also included in many operating environments or ports collections.

super_mediator requires libfixbuf 2.3.0 or later.

Spread support requires Spread 4.1 or later. Build and install Spread before building super_mediator.

super_mediator can process compressed IPFIX files if the zlib library is installed.

If MySQL libraries are available, the super_table_creator program will also be built. Compiling with MySQL, will also provide additional capabilities for automatic upload into a MySQL database. To disable these capabilities and building of the super_table_creator configure super_mediator with --with-mysql=no.

X.509 MD5/SHA1 Hashing support requires super_mediator to be built with OpenSSL support. Build and install OpenSSL before building super_mediator. Use the --with-openssl option to ./configure to enable hashing support.

Building super_mediator

super_mediator uses a reasonably standard autotools-based build system. The customary build procedure (./configure && make && make install) should work in most environments. Note that super_mediator finds libfixbuf using the pkg-config facility, so you may have to set the PKG_CONFIG_PATH variable on the configure command line if these libraries are installed in a nonstandard location, other than the prefix to which you are installing super_mediator itself.

super_mediator will look for MySQL and the SiLK IPSet library by default. However, if they are installed in a nonstandard location, providing the --with-mysql=MYSQL_CONFIG_PATH --with-skipset=SKIPSET_DIR options to ./configure will help in locating the libraries.

Tutorials

Quick and Easy super_mediator Setup (html)
A Quick and Easy guide for configuring super_mediator.
super_mediator Tutorial (html)
Tutorial for installing, configuring, and working with super_mediator. This tutorial will show examples of various outputs and how to configure super_mediator accordingly.
Configuring YAF and super_mediator (html)
How-to guide on using yaf and super_mediator to collect DPI data and import that data into a MySQL database. SiLK Flow collection will also be described.
Configuring de-duplication (html)
Configuration guide for data de-duplication
Configuring SSL de-duplication (html)
Configuration guide for SSL certificate de-duplication

Known Issues

Similar to yaf, the destinationTransportPort information element contains ICMP type and code information for ICMP or ICMP6 flows; this is nonstandard and may not be interoperable with other IPFIX implementations.

Contact

Please send bug reports, feature requests, and questions to contact_email. We welcome bug fixes and patches.