super_mediator

Super Mediator: Documentation

• Command Line Options
  • General Options
  • Input: Defining the Collector(s)
    • Network Collector
    • Directory Polling Collector
    • File Collector
    • Input Options
  • Output: Defining the Exporter
    • Network Exporter
    • Rotating File Exporter
    • Single File Exporter
    • Output Options
  • Modifying the Exported Records
  • Logging Options
  • Daemon Options
  • Privilege Options
• Configuration File

Super Mediator has the capability to route, aggregate, format, de-duplicate, enrich, filter, and coalesce data streams of IPFIX records from YAF or another Super Mediator instance. Super Mediator has three independent components: Collectors, Core, and Exporters. Collectors are responsible for ingesting IPFIX records from files or sockets, and making them available to the core processing unit. The core of Super Mediator handles every record, processes it based on the type of record, and passes it to the exporters. Exporters are responsible for writing records to output streams such as files or sockets, and generating specific derived records for that output stream, and curating outgoing records according to the configuration file.

Command Line Options

General Options

super_mediator has the following general options:

• --config CONFIG_FILE

  Specifies the name of the configuration file defining the collectors, exporters, and desired functionality. Using a configuration file is preferred over using command-line options. If this option is given, most other command line options are not allowed.

• --test-config

  Causes the program to exit after parsing the configuration file.

• --version

  Tells super_mediator to print its version number and the set of features that were enabled when it was built, and then exit.

• --help

  Causes super_mediator to print a brief description of the command line arguments it accepts and exit.

Input: Defining the Collector(s)

Super Mediator's input is called a collector. From the command line, one can create one or more collectors that listen on the network, one or more collectors that poll a directory, or one or more collectors that read from named files and the standard input.

Network Collector

To create network collectors, the --ipfix-input switch must be given. Its takes an argument of either tcp or udp specifying the type of network stream. The --ipfix-port switch specifies the port to listen on; it is optional and if not given port 18000 is used. The host(s) to listen on are given as arguments.

When acting as a network collector, super_mediator runs until it is signaled to stop.

The following examaples create a network collector. Output is written as IPFIX to the standard output.

super_mediator --ipfix-input=tcp --ipfix-port=7777 localhost

Bind to TCP port 7777 on localhost to listen for connections.

super_mediator --ipfix-input=tcp 127.0.0.1 ::1

Bind to TCP port 18000 on IP addresses 127.0.0.1 and ::1 to listen for connections.

Directory Polling Collector

One can configure super_mediator to periodically poll directories for files. The files found in the directory are read and then either moved to another directory or deleted. To create a polling collector, either --move-dir and/or --polling-interval must be given. The argument to --move-dir is the directory where files are moved after being processed; if not specified, the files are deleted. The period for polling the directory is specified by --polling-interval, and defaults to 30 seconds if not given. The directory(s) to poll are given as arguments.

super_mediator periodically polls the directory until it is signaled to terminate.

For example, the following create a directory polling collector. Output is written as IPFIX to the standard output.

super_mediator --move=/var/sm/complete /var/sm/incoming

Poll the directory /var/sm/incoming every 30 seconds; after processing each file, move it to /var/sm/complete.

super_mediator --move=/var/sm/complete --polling-interval=5 /var/sm/incoming

Similar to the previous example, except /var/sm/incoming is polled every 5 seconds.

super_mediator --polling-interval=30 /var/sm/incoming /var/sm/incoming2

Poll the directories /var/sm/incoming and /var/sm/incoming2 every 30 seconds; delete each file after processing it.

Note: If you are coming from v1.8.0 or earlier, note that the argument in v2.0.0 is a directory name. Previously, super_mediator took a glob pattern as an argument, but that is no longer its behavior.

File Collector

To have super_mediator run as a file collector, name the files on the command line. If no file names are given, super_mediator reads from its standard input. To have super_mediator read from files and from the standard input, use "-" as a file name. Each filename argument is treated as a separate collector.

When running as a file collector, super_mediator exits once all inputs have been read.

As an example, the following reads files a.yaf, b.yaf, and c.yaf and writes the result as IPFIX to the standard output.

super_mediator a.yaf b.yaf c.yaf

Input Options

These command line arguments are used to define the collectors when the configuration file is not used.

super_mediator treats any command line arguments that are not associated with an option as an INPUT_SPECIFIER. Zero or more INPUT_SPECIFIERs may be specified; each is treated as an IPFIX collector, and all specifiers must be the same type.

If super_mediator is listening on the network or polling a directory, it runs until it is signaled to stop (killed). When given a file list, super_mediator exits after processing the files.

• --ipfix-input TRANSPORT_PROTOCOL

  Causes super_mediator to operate as an IPFIX network collector, listening for connections via the specified protocol TRANSPORT_PROTOCOL, which must be either tcp or udp. UDP is not recommended as it is not a reliable transport protocol and cannot guarantee delivery of messages. The port to listen on is specified by the --ipfix-port option and defaults to 18000; the hostname to listen on must be specified in the INPUT_SPECIFIER.

• --ipfix-port PORT

  Requires the presence of --ipfix-input and specifies the TCP or UDP port where super_mediator listens for incoming connections. If not present, the default port 18000 is used. The hostname to listen on is specified in the INPUT_SPECIFIER.

• --polling-interval POLL_TIME

  Causes the INPUT_SPECIFIER(s) to be treated as directory name(s) and tells super_mediator to process files in the directory(s) every POLL_TIME seconds. super_mediator runs forever waiting for files to appear in the directory. After processing the incoming files, super_mediator deletes the files unless a --move-dir is specified.

• --move-dir PROCESSED_INPUT_DIRECTORY

  Causes the INPUT_SPECIFIER(s) to be treated as directory name(s) and tells super_mediator to process files in the directory(s) periodically. The default period is 30 seconds, but may be changed with the --polling-interval option. The incoming files are moved to PROCESSED_INPUT_DIRECTORY after processing. If --polling-interval is specified and --move-dir is not, the incoming files are deleted after they are processed.

• --no-locked-files

  Currently unimplemented. Tells super_mediator to ignore the presence of lock files and process all files in the incoming directory when either --polling-interval or --move is specified. By default, super_mediator does not read files that are locked, which means they have the extension ".lock" appended to the end of the filename. This can be used if super_mediator is reading from a yaf(1) export directory and yaf(1) is run with --lock. This will prevent super_mediator from removing the files out from under yaf(1). This does not lock files that the super_mediator is writing to. Use the super_mediator configuration file to enable locking of output files.

Output: Defining the Exporter

The output from Super Mediator is called an exporter. A single exporter is created when not using the configuration file. Super Mediator may export IPFIX to a network host and port, or it may export IPFIX, JSON, or delimited text to a single file or a series of files where the output file is closed and a new file opened periodically (a rotating output file).

Network Exporter

To create an IPFIX network exporter, the --output-mode switch must be given and have an argument of either tcp or udp. The --out option names the host or IP address where the records are sent. By default, super_mediator attempts to contact that host on port 18001, but that may be changed by using the --export-port switch.

The --sleep option may be specified to introduce a delay in how often records are written to the output. This can be used to help reduce data loss when transmitting IPFIX over UDP. Its argument is the number of microseconds to delay between adding records to the output buffer.

The following examples read IPFIX from the standard input and write it over the network.

super_mediator --output-mode=tcp --out=localhost

Write data over TCP to port 18001 on localhost.

super_mediator --output-mode=tcp --export-port=7788 --out=127.0.0.1

Write data over TCP to port 7788 on 127.0.0.1.

Rotating File Exporter

When the --rotate option is given, super_mediator writes its output to a series of files. The argument to --out is a directory and filename prefix to use for the output files, and the suffix depends on the format of the output being written.

If the --output-mode option is not given, the output is IPFIX, and the suffix of the output files is based on the current time. The suffix is "-YYYYmmddHHMMSS-NNNNN.med" where YYYYmmdd is the current year, month, and day, HHMMSS is the current UTC time, and NNNNN is a serial number that is incremented for each file created.

To have super_mediator write JSON, specify --output-mode=json. In this case, the suffix is based on the time within the flow record itself, and the suffix is ".YYYYmmddHHMMSS.json", where YYYYmmdd and HHMMSS is based on the year, month, day, and time of the flow record's endFlowMilliseconds.

To have super_mediator write delimited-separated-value text (with the vertical bar, |, also called pipe, as the delimiter), specify --output-mode=text. In this case, as with JSON, the suffix is based on the time within the flow record itself, and the suffix is ".YYYYmmddHHMMSS.txt".

The following examples read IPFIX from the standard input.

super_mediator --rotate=30 --out=/data/fccx

Write binary IPFIX data to files in /data. When processing the Flaming Cupcake Challenge (FCC) sample data from 2015, one of the output files is named "/data/fccx-20220222220222-02003.med".

super_mediator --rotate=30 --output-mode=json --out=/data/fccx

Write the flows in the JSON format to files in /data. When processing the FCC sample data, one of the output files is named "/data/fccx.20150914235417.json"

super_mediator --rotate=30 --output-mode=text --out=/data/fccx

Write the flows in a pipe-delimited text format to files in /data. When processing the FCC sample data, one of the output files is named "/data/fccx.20150914235648.txt"

Single File Exporter

To write to a single file do not specify --rotate. The argument to --out is the destination file. Use --output-mode to produce JSON or delimited-text output, or do not specify the option to produce IPFIX. If --out is not given, output is written to the standard output.

The following examples read IPFIX from the standard input.

super_mediator --out=/data/my-file.ipfix

Writes IPFIX to "/data/my-file.ipfix".

super_mediator --output-mode=json --out=/data/my-file.json

Writes JSON to "/data/my-file.json".

super_mediator --output-mode=text --out=/data/my-file.txt

Writes delimited text to "/data/my-file.txt".

Output Options

These options control where super_mediator sends its output and the type of output it writes. super_mediator can write flows to an IPFIX file, text file, or to an IPFIX collector over TCP or UDP. By default, if no options are given, super_mediator writes IPFIX to standard out.

• --out OUTPUT_SPECIFIER

  OUTPUT_SPECIFIER is an output specifier. If --output-mode is present, and set to TCP or UDP, the OUTPUT_SPECIFIER specifies the hostname or IP address of the collector to which the flows will be exported. If --output-mode is set to TEXT, OUTPUT_SPECIFIER is a filename in which the flows will be written in pipe-delimited (C<|>) format. If --output-mode is set to JSON, OUTPUT_SPECIFIER is a filename in which the flows will be written in pipe-delimited (C<|>) format. Otherwise, OUTPUT_SPECIFIER is a filename in which flows will be written in IPFIX Format. The string - may be used to write to standard output (the default). If --rotate is present, OUTPUT_SPECIFIER is the prefix name of each output file to write to. When writing to a network socket, super_mediator must be able to make an initial connection to the OUTPUT_SPECIFIER for super_mediator to start. If the connection is lost after the initial connection, super_mediator will immediately retry the connection after reporting a warning message to the log. If the retry is unsuccessful, super_mediator will retry the connection every 15 seconds until the connection is successful. Flows will be lost while the connection is down.

• --output-mode TRANSPORT_PROTOCOL

  If present, causes super_mediator to operate as an IPFIX, TEXT, or JSON exporter, exporting via the specified protocol TRANSPORT_PROTOCOL to a collector (e.g rwflowpack, flowcap) named in the OUTPUT_SPECIFIER. Valid TRANSPORT_PROTOCOL values are tcp, udp, text, and json. UDP is not recommended, as it is not a reliable transport protocol and cannot guarantee delivery of messages.

• --export-port PORT

  If --output-mode is present and set to TCP or UDP, export flows to port PORT. If not present, the default port 18001 will be used. The host to export to is specified with the --out option.

• --rotate ROTATE_SECONDS

  Causes the OUTPUT_SPECIFIER to be treated as the prefix of an output file name. super_mediator appends a timestamp to the prefix and periodically closes the output file and opens a new open. This switch determines how often that occurs.

• --sleep MICROSECONDS

  If present, super_mediator sleeps for MICROSECONDS microseconds between each record it appends to an IPFIX message. This is useful if super_mediator is reading an IPFIX file and transmitting IPFIX over UDP. super_mediator may send the messages too quickly for the IPFIX Collector to receive them (possibly dropping messages.) This option is only available with one collector and one exporter when executing super_mediator from the command line.

Modifying the Exported Records

These options control how the exported data appears or what types of records are exported.

• --fields FIELD_LIST

  If present and --output-mode=TEXT is also present, writes only the fields given in FIELD_LIST. FIELD_LIST is a list of IPFIX element names, separated by a comma. The list of acceptable fields is nearly any IPFIX element.

• --print-headers

  If present for TEXT Exporters, super_mediator writes a header for delimited flow data. If files rotate, it writes one header at the top of each flow data file.

• --no-stats

  If present, super_mediator does not forward YAF process statistics records or log statistics. It is possible to configure certain exporters to process stats while others ignore stats messages. This must be done with through the super_mediator configuration file.

• --preserve-obdomain

  If present, super_mediator will not overwrite the observation domain in the incoming IPFIX records. If given and the incoming records do not have an observationDomainId element, the exported records will have a domain of zero. super_mediator's default behavior is to copy the observation domain ID from the incoming IPFIX messages' headers to the records it exports, overwriting any previous observationDomainId value in the records.

• --rewrite-ssl-certs

  If specified, super_mediator will, for IPFIX exporters, rewrite the incoming TLS/SSL certificate records to have explicit information elements for parts of the certificate's issuer, subject, and extensions instead of having data stored in a subTemplateList of key-value (sslObjectType,sslObjectValue) pairs.

• --disable-metadata-export

  If present, super_mediator does not include information element and template metadata in IPFIX output.

• --ipsetfile IPSET_FILE

  Exits the program due to missing IPset support.

• --udp-temp-timeout TIMEOUT_SECS

  Currently unimplemented. Set UDP template timeout in seconds if --ipfix-mode is set to UDP. As per RFC 5101 recommendations, super_mediator will attempt to export templates three times within TEMPLATE_SECS. The default template timeout period is 600 seconds (10 minutes).

Logging Options

By default, super_mediator writes log messages at levels WARNING and ERROR to the standard error. These options change that behavior.

The command-line logging switches override the log settings set in the configuration file.

• --log LOG_SPECIFIER

  Specifies the destination for log messages. LOG_SPECIFIER can be a syslog(3) facility name, the special value stderr for standard error, or the absolute path to a file for file logging. To write messages to rotating files in a directory, use the --log-dir option. The default log specifier is stderr. The log level can be specified by the LOGLEVEL keyword in the super_mediator configuration file or by using either --verbose or --quiet. The default level is WARNING.

• --log-dir LOG_PATH

  Tells super_mediator to write log messages to files in LOG_PATH. LOG_PATH must be a complete directory path. The log files have the form LOG_PATH/sm-YYYYMMDD.log where YYYYMMDD is the current date. The log files are rotated at midnight local time. When the log files are rotated a new log is opened, the previous file is closed, and gzip(1) is invoked on the previous day's log file. (Old log files will not be removed by super_mediator.)

• --verbose

  Enables logging of all messages. The default log level is WARNING. This option changes the log level to DEBUG and logs all yaf(1) and super_mediator process statistics, along with any IO operations.

• --quiet

  Turns off logging completely. super_mediator will not log errors.

Daemon Options

To run super_mediator as a daemon, specify the --daemonize option. The configuration file does not offer a way to enable this. super_mediator refuses to run as a daemon if the log output is being written to the standard error.

Even without the --daemonize option, super_mediator runs until killed when using a network or directory polling collector.

• --daemonize

  Causes super_mediator to become a daemon.

• --pidfile PIDFILE_NAME

  Sets the complete path to the file in which super_mediator writes its process ID (pid) when running as a daemon. --pid-file is ignored if --daemon is not present.

Privilege Options

There is no need to start super_mediator as the root user as it does not require access to any privileged resources. However, if it is started as root, use of the following are highly recommended for security purposes.

• --become-user UNPRIVILEGED_USER

  Tells super_mediator to drop its privileges to UNPRIVILEGED_USER after starting. Using --become-user requires super_mediator to be run as root or setuid root. This option will cause all files written by super_mediator to be owned by the user UNPRIVILEGED_USER and the user's primary group; use --become-group as well to change the group super_mediator runs as for output purposes.

• --become-group UNPRIVILEGED_GROUP

  Tells super_mediator to change its privileges to UNPRIVILEGED_GROUP after starting. The --become-group option allows changing the group from the default of the user given in --become-user. This option has no effect if given without the --become-user option as well.

Configuration File

For details of the configuration file, see the super_mediator.conf manual page.