super_mediator

What's New in Super Mediator 2.0.

• Adaptability for Changes to YAF
• New Configuration File Syntax
• Removed Several Command Line Options
• Enhanced Statistics
• Record and Template IDs Have Changed
• Field Name Changes
• Limitations
  • Delimited Text Export Is A Work-in-Progress
  • SiLK IPset Support Temporarily Disabled
  • MySQL Support Temporarily Disabled

This page documents the new features and incompatible changes in Super Mediator 2.0.

Adaptability for Changes to YAF

Traditionally Super Mediator contained fixed template definitions that matched those in YAF. However, this meant changes to YAF also required changes to Super Mediator.

Super Mediator 2.0 largely uses the templates it receives as-is and passes them along to its export streams.

When looking for a particular templates (do to de-duplication, for example), Super Mediator 2.0 checks for particular elements instead of matching the entire template and makes use of the exhanced template metadata exported by YAF 3. This allows increased flexibility and should allow Super Mediator to handle moderate changes that YAF makes to its templates. (Major changes in YAF will still require changes to Super Mediator.)

New Configuration File Syntax

The configuration file has significant changes. Files from previous releases of Super Mediator must be updated to work with this release.

• SINGLE_FILE replaces previous keyword FILEANDLER

  In general, places that used FILEHANDLER should instead use SINGLE_FILE.

• COLLECTOR uses DIRECTORY_POLL in place of DIR.

  To have a COLLECTOR poll a directory, use the DIRECTORY_POLL keyword, and note that the PATH argument now takes directory name and not a file glob. There is no way to limit which files a DIRECTORY_POLL COLLECTOR processes.

• EXPORTER statement requires two arguments plus optional name.

  The EXPORTER keyword now takes two arguments: an output format (IPFIX, JSON, or TEXT) and a transport (SINGLE_FILE, TCP, UDP, or ROTATING_FILES).

• JSON is now an export format.

  The JSON keyword within an EXPORTER block is no longer supported. Instead, JSON should be used immediately after the EXPORTER keyword.

• Comparison filtering statements use the information element name, surrounded by double quotes.

  To select only DNS traffic, you should use "silkAppLabel" == 53 in place of APPLICATION == 53.

• The FIELDS statement now takes double-quoted information element name(s), with square brackets around the list.

  Change FIELDS hash, stime to FIELDS \[ "yafFlowKeyHash", "flowStartMilliseconds" \].

A new option --test-config has been added to allow testing the configuration file syntax.

Removed Several Command Line Options

Made changes to the command line parsing and eliminated several options. The configuration file is the preferred way to configure super_mediator.

• Option --in has been removed.

  Command line arguments that are not arguments to an option are treated as inputs with the types of inputs depending on the argument to --input-mode. For example, the inputs are treated as hosts if --input-mode is tcp or udp.

  In --input-mode is not given, the inputs are either file names to read or directory names to poll. All inputs must be of the same type.

• When using the configuration file, command line input files now cause an error. Inputs must be specified in the configuration file when it is in use.

• Glob handling has been removed from super_mediator. Instead, the shell should be used to expand the glob.

• Option --polling-interval replaces --watch.

• Option --move-dir replaces --move.

• Option --no-locked-files replaces --lock.

• Option --dns-dedup has been removed. This must be enabled within a configuration file.

In general, it is now an error to specify a value on the command line and in the configuration file.

Enhanced Statistics

Enhanced statistics for types of records read from a collector and written to an exporter.

Record and Template IDs Have Changed

Super Mediator processes data from YAF. Since YAF 3 has changed the structure and IDs of its templates, including the use of named lists, Super Mediator contains these changes when reading data from YAF 3.

Field Name Changes

Note that Super Mediator 2.0 has the same information element name changes as YAF 3.0.

The new names are used even if Super Mediator is processing IPFIX generated by YAF 2. The IPFIX input contains only numeric IDs for the elements. The mapping of IDs to names depends on the version of the software doing the mapping.

Limitations

Super Mediator currently has the following limitations.

Delimited Text Export Is A Work-in-Progress

Exporting as delimited TEXT is lightly tested and contains bugs.

SiLK IPset Support Temporarily Disabled

SiLK IPset support is not available in this release. This will be re-enabled in a later release.

MySQL Support Temporarily Disabled

MySQL and MariaDB support is not available currently. This will be re-enabled in a later release.