Super Mediator: Documentation

Super Mediator has the capability to route, aggregate, format, de-duplicate, enrich, filter, and coalesce data streams of IPFIX records from YAF or another Super Mediator instance. Super Mediator has three independent components: Collectors, Core, and Exporters. Collectors are responsible for ingesting IPFIX records from files or sockets, and making them available to the core processing unit. The core of Super Mediator handles every record, processes it based on the type of record, and passes it to the exporters. Exporters are responsible for writing records to output streams such as files or sockets, and generating specific derived records for that output stream, and curating outgoing records according to the configuration file.

Command Line Options

General Options

super_mediator has the following general options:

Input: Defining the Collector(s)

Super Mediator's input is called a collector. From the command line, one can create one or more collectors that listen on the network, one or more collectors that poll a directory, or one or more collectors that read from named files and the standard input.

Network Collector

To create network collectors, the --ipfix-input switch must be given. Its takes an argument of either tcp or udp specifying the type of network stream. The --ipfix-port switch specifies the port to listen on; it is optional and if not given port 18000 is used. The host(s) to listen on are given as arguments.

When acting as a network collector, super_mediator runs until it is signaled to stop.

The following examaples create a network collector. Output is written as IPFIX to the standard output.

super_mediator --ipfix-input=tcp --ipfix-port=7777 localhost

Bind to TCP port 7777 on localhost to listen for connections.

super_mediator --ipfix-input=tcp ::1

Bind to TCP port 18000 on IP addresses and ::1 to listen for connections.

Directory Polling Collector

One can configure super_mediator to periodically poll directories for files. The files found in the directory are read and then either moved to another directory or deleted. To create a polling collector, either --move-dir and/or --polling-interval must be given. The argument to --move-dir is the directory where files are moved after being processed; if not specified, the files are deleted. The period for polling the directory is specified by --polling-interval, and defaults to 30 seconds if not given. The directory(s) to poll are given as arguments.

super_mediator periodically polls the directory until it is signaled to terminate.

For example, the following create a directory polling collector. Output is written as IPFIX to the standard output.

super_mediator --move=/var/sm/complete /var/sm/incoming

Poll the directory /var/sm/incoming every 30 seconds; after processing each file, move it to /var/sm/complete.

super_mediator --move=/var/sm/complete --polling-interval=5 /var/sm/incoming

Similar to the previous example, except /var/sm/incoming is polled every 5 seconds.

super_mediator --polling-interval=30 /var/sm/incoming /var/sm/incoming2

Poll the directories /var/sm/incoming and /var/sm/incoming2 every 30 seconds; delete each file after processing it.

Note: If you are coming from v1.8.0 or earlier, note that the argument in v2.0.0 is a directory name. Previously, super_mediator took a glob pattern as an argument, but that is no longer its behavior.

File Collector

To have super_mediator run as a file collector, name the files on the command line. If no file names are given, super_mediator reads from its standard input. To have super_mediator read from files and from the standard input, use "-" as a file name. Each filename argument is treated as a separate collector.

When running as a file collector, super_mediator exits once all inputs have been read.

As an example, the following reads files a.yaf, b.yaf, and c.yaf and writes the result as IPFIX to the standard output.

super_mediator a.yaf b.yaf c.yaf

Input Options

These command line arguments are used to define the collectors when the configuration file is not used.

super_mediator treats any command line arguments that are not associated with an option as an INPUT_SPECIFIER. Zero or more INPUT_SPECIFIERs may be specified; each is treated as an IPFIX collector, and all specifiers must be the same type.

If super_mediator is listening on the network or polling a directory, it runs until it is signaled to stop (killed). When given a file list, super_mediator exits after processing the files.

Output: Defining the Exporter

The output from Super Mediator is called an exporter. A single exporter is created when not using the configuration file. Super Mediator may export IPFIX to a network host and port, or it may export IPFIX, JSON, or delimited text to a single file or a series of files where the output file is closed and a new file opened periodically (a rotating output file).

Network Exporter

To create an IPFIX network exporter, the --output-mode switch must be given and have an argument of either tcp or udp. The --out option names the host or IP address where the records are sent. By default, super_mediator attempts to contact that host on port 18001, but that may be changed by using the --export-port switch.

The --sleep option may be specified to introduce a delay in how often records are written to the output. This can be used to help reduce data loss when transmitting IPFIX over UDP. Its argument is the number of microseconds to delay between adding records to the output buffer.

The following examples read IPFIX from the standard input and write it over the network.

super_mediator --output-mode=tcp --out=localhost

Write data over TCP to port 18001 on localhost.

super_mediator --output-mode=tcp --export-port=7788 --out=

Write data over TCP to port 7788 on

Rotating File Exporter

When the --rotate option is given, super_mediator writes its output to a series of files. The argument to --out is a directory and filename prefix to use for the output files, and the suffix depends on the format of the output being written.

If the --output-mode option is not given, the output is IPFIX, and the suffix of the output files is based on the current time. The suffix is "" where YYYYmmdd is the current year, month, and day, HHMMSS is the current UTC time, and NNNNN is a serial number that is incremented for each file created.

To have super_mediator write JSON, specify --output-mode=json. In this case, the suffix is based on the time within the flow record itself, and the suffix is ".YYYYmmddHHMMSS.json", where YYYYmmdd and HHMMSS is based on the year, month, day, and time of the flow record's endFlowMilliseconds.

To have super_mediator write delimited-separated-value text (with the vertical bar, |, also called pipe, as the delimiter), specify --output-mode=text. In this case, as with JSON, the suffix is based on the time within the flow record itself, and the suffix is ".YYYYmmddHHMMSS.txt".

The following examples read IPFIX from the standard input.

super_mediator --rotate=30 --out=/data/fccx

Write binary IPFIX data to files in /data. When processing the Flaming Cupcake Challenge (FCC) sample data from 2015, one of the output files is named "/data/".

super_mediator --rotate=30 --output-mode=json --out=/data/fccx

Write the flows in the JSON format to files in /data. When processing the FCC sample data, one of the output files is named "/data/fccx.20150914235417.json"

super_mediator --rotate=30 --output-mode=text --out=/data/fccx

Write the flows in a pipe-delimited text format to files in /data. When processing the FCC sample data, one of the output files is named "/data/fccx.20150914235648.txt"

Single File Exporter

To write to a single file do not specify --rotate. The argument to --out is the destination file. Use --output-mode to produce JSON or delimited-text output, or do not specify the option to produce IPFIX. If --out is not given, output is written to the standard output.

The following examples read IPFIX from the standard input.

super_mediator --out=/data/my-file.ipfix

Writes IPFIX to "/data/my-file.ipfix".

super_mediator --output-mode=json --out=/data/my-file.json

Writes JSON to "/data/my-file.json".

super_mediator --output-mode=text --out=/data/my-file.txt

Writes delimited text to "/data/my-file.txt".

Output Options

These options control where super_mediator sends its output and the type of output it writes. super_mediator can write flows to an IPFIX file, text file, or to an IPFIX collector over TCP or UDP. By default, if no options are given, super_mediator writes IPFIX to standard out.

Modifying the Exported Records

These options control how the exported data appears or what types of records are exported.

Logging Options

By default, super_mediator writes log messages at levels WARNING and ERROR to the standard error. These options change that behavior.

The command-line logging switches override the log settings set in the configuration file.

Daemon Options

To run super_mediator as a daemon, specify the --daemonize option. The configuration file does not offer a way to enable this. super_mediator refuses to run as a daemon if the log output is being written to the standard error.

Even without the --daemonize option, super_mediator runs until killed when using a network or directory polling collector.

Privilege Options

There is no need to start super_mediator as the root user as it does not require access to any privileged resources. However, if it is started as root, use of the following are highly recommended for security purposes.

Configuration File

For details of the configuration file, see the super_mediator.conf manual page.