Monitoring for Large-Scale Networks

super_mediator

super_mediator.conf

NAME

super_mediator.conf - Configuration file for super_mediator

DESCRIPTION

As part of the collecting and exporting of flow data, super_mediator(1) needs to know what type of data it is collecting and how to collect it (e.g. listen on 18000/udp or export to 18001/tcp or collect only flow data with source port = 80). The Super Mediator Configuration File, super_mediator.conf, contains this information, and this manual page describes the syntax of the file.

The super_mediator.conf file may have any name, and it does not have to reside in a particular location. The location of the file is specified by the --config switch to super_mediator.

Some command line arguments will override settings declared in the configuration file. In particular, any collector information provided on the command line will override collectors defined in the configuration file. Command line arguments for exporters will be ignored if a configuration file is also present.

The Configuration file for super_mediator defines the following concepts:

collector

A collector specifies a source for flow data from yaf(1). The source could be a transport protocol such as TCP or UDP or Spread. The source could also be a directory that is periodically polled for IPFIX files, or it could simply be a single file to process. When defining the collector, you must specify the collector type. Valid types are TCP, UDP, SPREAD, FILEHANDLER, and DIR. super_mediator expects IPFIX data created by yaf or super_mediator. It may collect other valid IPFIX data but expects certain information elements to be present to use particular features. The syntax of collectors is described in "Collector Block" below.

filter

If a filter is specified, super_mediator passes each flow record through each filter to determine whether the record should be passed to the exporters. A filter block in the configuration file defines filters for the collector processes only. Filters may be specified for each exporter, but should exist in the appropriate exporter block. A filter that is defined in a collector block will only apply to that particular collector. A filter that is defined outside of a collector or exporter block will apply to ALL collectors. See "Filter Block".

exporter

An exporter specifies a destination for the flow data it processes. The super_mediator can have multiple exporters. The destination point can be a transport protocol such as TCP, UDP, or Spread. It can be a single IPFIX or TEXT file, or otherwise it could be a directory that super_mediator will rotate output files every so many seconds. When defining the exporter, you must specify the exporter type. Valid types are TCP, UDP, SPREAD, FILEHANDLER, or TEXT. Only one SPREAD Exporter is permitted, due to the nature of Spread. A Spread Exporter can export to multiple Spread Groups. A TEXT exporter can be configured to import CSV files to a MySQL database if MySQL libraries are available. Exporters may have filters associated with them. When super_mediator loses connection with one of the configured exporters, it will report an initial warning message to the log and retry the connection immediately. If the retry is unsuccessful, it will retry the connection every 15 seconds until successful. To verify the connection attempts, use the DEBUG loglevel. Flows will be lost while the connection is down. Exporters are described in "Exporter Block" below.

group

A Spread exporter is a special type of exporter, since one Spread Exporter can export to multiple points using defined groups. Spread is a pub/sub high performance message service. Spread functions as a unified message bus for distributed applications. The Spread exporter specifies the Spread daemon name to connect and one or more Spread groups in the EXPORTER block. Then in the GROUP block, the group name is specified along with one or more filters to use when deciding what flow records to send to the group. For each defined group name in the EXPORTER block, one GROUP block can be defined. See "Group Block"

dns-dedup

super_mediator will perform dns de-duplication for each exporter that enables dns de-duplication. It will take any resource record information collected from yaf(1) and cache rrname, rrtype, and rrval tuples for each resource record. If a new record arrives that does not exist in the cache, the record is exported to the appropriate exporters. Otherwise, the hitcount is incremented and flushed based on the default values or the values specified in the DNS_DEDUP block. See "DNS De-duplication Block"

dpi-config

The DPI_CONFIG block is used for advanced configuration of the Deep Packet Inspection (DPI) data export of an EXPORTER in TEXT mode. The super_mediator can be configured to insert labels into the CSV output of DPI metadata. This can be used to assist a database loading process to determine which table in the database a particular line should be inserted. The label is effectively the name of the database table the line should be loaded in to. The DPI_CONFIG block allows the user to choose which DPI fields (user agent strings, urls, etc.) are exported and how they should be labeled according to their personal database schema. This block is optional; its syntax is described in "DPI Configuration Block". It is only necessary if you want a subset of the available DPI fields or the default table names are not sufficient. The DPI_CONFIG block can also be used in conjunction with the Exporter command, MULTI_FILES, to configure the names of the output files. See "MULTI_FILES" below for more information.

ssl-config

THE SSL_CONFIG block is used to configure which SSL X.509 certificate fields an EXPORTER in TEXT mode will write. Each TEXT EXPORTER may have one SSL_CONFIG block associated with it. Within the SSL_CONFIG block, you can specify ISSUER fields, SUBJECT fields, and OTHER fields. This block is similar to a DPI_FIELD_LIST but is the only way to configure which fields from the ISSUER and SUBJECT of an X.509 Certificate will be exported. If this block is present for an EXPORTER, all SSL/TLS fields will be disabled unless explicitly specified within one of the lists. See "SSL_CONFIG Block".

This block is also used for enabling SSL Certificate Deduplication. super_mediator will perform SSL de-duplication if SSL_DEDUP is present in this block or SSL_DEDUP_ONLY is present in any EXPORTER block. It will take SSL Certificate information collected by yaf(1) and output the certificate information once and cache the serial number and issuer name for each certificate. Certificates matching the serial number and issuer tuple will simply increment the internal counters. Certificates in the cache are flushed based on the default values or the configurable values specified in this block.

dedup-config

super_mediator will perform de-duplication of most deep packet inspection (DPI) information element values if enabled. To enable de-duplication, this block must be present and "assigned" to one EXPORTER using the exporter name. Using the provided information element ID, super_mediator will cache the value of the information element ID and the source IP address OR destination IP address. Any record that contains the same tuple will increment the hitcount in the cache. The record will be flushed when either the max hit count is reached or no records with the same tuple have been seen in the configurable timeout period. See "DEDUP_CONFIG Block".

maps

super_mediator can add an additional key for deduplication with a VLAN_MAP or OBID_MAP. Without the use of a MAP, super_mediator will use the SIP, DIP, or FLOW KEY HASH to aggregate DPI information elements. If you additionally want to aggregate information elements per VLAN (or group of VLANs) or per Observation ID (or group of observation IDs) then use a VLAN_MAP or OBID_MAP to define the mapping of IDs to String. The String provided will be included in the output record. The syntax is described in "MAP Blocks".

SYNTAX

When parsing the super_mediator configuration file, blank lines are ignored. At any location in a line, the character # indicates the beginning of a comment, which continues to the end of the line. These comments are ignored.

All other lines begin with optional leading whitespace, a command name, and one or more arguments to the command. Command names are a sequence of non-whitespace characters. Arguments are textual atoms: any sequence of non-whitespace, non-# characters, including numerals and punctuation.

There are nine main contexts for commands: top-level, collector block, filter block, exporter block, group block, dns-dedup, ssl-config, dedup-config, and dpi-config block. The collector block, filter block, exporter block, group block, dns-dedup, dpi-config, ssl-config, and dedup-config block contexts are used to describe individual features of collectors, filters, exporters, spread groups, DNS de-duplication, DPI export, SSL configuration and de-duplication, and general de-duplication respectively.

The valid commands for each context are described below.

Top-Level Commands

In addition to the commands to begin a collector, filter, exporter, group, dns-dedup, dpi-config, ssl-config, dedup-config, or map block, the top-level context supports the following commands:

NO_STATS

If the NO_STATS keyword is present anywhere in the configuration file outside of a collector, filter, exporter, or group block, any stats messages received from yaf(1) will be dropped. They will not be logged to the super_mediator log file (if specified) nor will they be exported to any of the defined exporters. The super_mediator also keeps process statistics about how many flows it receives, filters, and exports. This stats message is logged every 5 minutes. If NO_STATS is present, this stats message will not be logged.

PRESERVE_OBDOMAIN

By default, super_mediator copies the observation domain identifier from the incoming IPFIX messages' headers to the records it exports, overwriting any previous observationDomainId value in the records. If the PRESERVE_OBDOMAIN keyword is present anywhere in the configuration file outside of a collector, filter, exporter, or group block, super_mediator preserves the incoming records' value of the observationDomainId. If the incoming records do not contain that element, the records exported by super_mediator contain an observation domain of zero.

REWRITE_SSL_CERTS

If specified, super_mediator will, for all IPFIX exporters, rewrite the SSL certificate records that are represented in a subTemplateList on the SSL DPI record (template 51722 (0xca0a)). The rewriting changes the structure of template 51723 (0xca0b) and eliminates template 52756 (0xce14). The new SSL certificate record has explicit information elements for parts of the certificate's issuer and subject instead of having data stored in a list of key-value pairs. This setting must be used when super_mediator is reading IPFIX records from an upstream super_mediator instance that rewrote its SSL records.

LOG_FILE LOG_SPECIFIER

Specifies the destination for log messages. LOG_FILE can be a syslog(3) facility name, the special value stderr for standard error, or the absolute path to a file for file logging. The default log specifier is stderr. The log level can be specified by the LOGLEVEL keyword. Default level is WARNING. The log file contains process statistics such as uptime, total flows received, DNS flows received, and deduplicated flows. It also contains information about each of the collectors and exporters. In addition, it will record any statistics messages received from YAF. Below are example log messages. Lines that begin with "SM" are statistics about super_mediator. The "SM" log messages contain the uptime, total number of flows received by all Collectors, any flows that were filtered by all collectors, and the total number of statistics (IPFIX options records) received. If DNS deduplication is enabled, super_mediator will also report the number of flows that contain DNS data (dns), the number of individual resource records (RRrecords), and the number of DNS records "exported" after deduplication (dedup). yaf statistics contain "YAF" and the Collector name that received the stats record. The Exporter line contains the number of flows exported, statistics exported, and information about bandwidth. If DNS deduplication is enabled, the Exporter will also report the number of DNS records exported. The Collector log message reports the number of flows received by that collector, the total number of statistics (IPFIX options) records, the total number of flows filtered, and the number of connections accepted.

SM: Uptime: 0d:0h:4m:23s, Total Flows: 93, Filtered: 0, Stats: 0

SM: dns: 10, RRrecords: 41 dedup: 0

C2: YAF ID: 10 IP: 10.20.11.51 Uptime: 0d:0h:5m:1s

C2: YAF Flows: 104 Packets: 13230 Dropped: 0 Ignored: 188 Out of Sequence: 0 Expired Frags: 0 Assembled Frags: 0

Exporter E4: 96 flows, 0 stats, 0.0005 Mbps, 192.00 bytes per record

Collector C2: 93 flows, 0 stats, 0 filtered, 1 connection

LOG_DIR LOG_DIRECTORY_PATH

If present, super_mediator will write log files to LOG_DIRECTORY_PATH. LOG_DIRECTORY_PATH must be a complete directory path. The log files have the form

LOG_DIRECTORY_PATH/sm-YYYYMMDD.log

where YYYYMMDD is the current date. The log files are rotated at midnight local time. When the log files are rotated a new log is opened, the previous file is closed, and gzip(1) is invoked on the previous day's log file. (Old log files will not be removed by super_mediator.)

LOGLEVEL LOG_LEVEL

Specify minimum level for logged messages. In increasing levels of verbosity, the supported log levels are QUIET, ERROR, WARNING, MESSAGE, and DEBUG. The default logging level is WARNING. This level only logs critical errors or potential problems. The MESSAGE level will log all yaf(1) process statistics it receives, along with periodic process statistics about itself. Setting the LOG_LEVEL to QUIET, ERROR, or WARNING will prevent these messages from being logged. The DEBUG level will report any I/O operations, such as opening, closing, moving, and deleting of input and output files in addition to all yaf and super_mediator process statistics.

PIDFILE PIDFILE_NAME

If present, and super_mediator is run in daemon mode, super_mediator will write the process ID (pid) to the file specified by PIDFILE_NAME. PIDFILE_NAME should be the complete path to the file.

STATS_TIMEOUT TIMEOUT_SECOND

If present, log super_mediator process statistics every TIMEOUT_SECOND seconds. The default is 5 minutes [600 seconds]. If set to 0, super_mediator will not log statistics.

TIME_UNITS UNITS

Specifies the time units (for example milliseconds or nanoseconds) to use for the flow record timestamps in Exporters that write TEXT or JSON. This setting affects all exporters that follow it. An exporter may set its own TIME_UNITS to override the setting at top-level. This command is described in the "Exporter Block" section below. TIME_UNITS was added in super_mediator-1.11.0.

USER_IE INFO_ELEMENT_ID INFO_ELEMENT_NAME *APPLICATION_LABEL*

If present, add the Information Element with ID INFO_ELEMENT_ID and INFO_ELEMENT_NAME name to the Information Model. The Information Element will have the CERT Private Enterprise Number. If user-defined Information Elements are not added prior to collection, super_mediator will ignore the information element and emit a warning similar to:

BasicList Decode Error: No Information Element with ID 254 defined

Any user-defined information element defined in the yafDPIRules.conf file, should also be added to the super_mediator.conf in order for super_mediator to collect the element. If APPLICATION_LABEL is present, then this element will be added to the appropriate file if using MULTI_FILES. For example, if APPLICATION_LABEL is set to 80, then the information element will be added to the http.txt file by default. If you are using custom lists or a DPI_FIELD_LIST, setting APPLICATION_LABEL is not necessary. If DPI_CONFIG is set for custom tables, do not set APPLICATION_LABEL.

USER_IE 999 my_info_element
USER_IE 1002 http_other_field 80

Collector Block

The first command below is used at the top-level to begin a collector definition block, and the remaining commands are accepted within the collector context.

COLLECTOR COLLECTOR-TYPE COLLECTOR-NAME

The COLLECTOR command begins a new collector block and it continues to the COLLECTOR END command. The argument to the COLLECTOR command is the type of collector to be defined and an OPTIONAL COLLECTOR-NAME. The COLLECTOR-NAME will be used in the log file and in the default flow and stats pipe-delimited text output. If COLLECTOR-NAME is not provided, super_mediator uses C1, C2, C3, etc. The COLLECTOR-TYPE must be one of the following:

TCP: This collector processes IPFIX, listening for connections from yaf(1) on a TCP port.
UDP: This collector processes IPFIX, listening for connections from yaf(1) on a UDP port. Note that UDP is not recommended, as it is not a reliable transport protocol, and thus cannot guarantee delivery of messages. Also note that unless the super_mediator is started prior to starting yaf, it will not receive the necessary templates until yaf periodically transmits them as specified by the command line --udp-temp-timeout given to yaf at startup time. libfixbuf will display warning messages until the templates are received.
SPREAD: This collector subscribes to the given group names through the Spread daemon, also supplied in the COLLECTOR block.
FILEHANDLER: This collector reads from a single IPFIX file.
DIR: This collector polls the given directory waiting for files that match a given glob pattern. It will either delete files (default) after they have been processed and transmitted to the appropriate exporters, or move the files to the given directory (specified in the COLLECTOR block).

COLLECTOR END

The COLLECTOR END command ends the definition of a collector. Following a COLLECTOR END command, top-level commands are again accepted.

PORT PORT

This command specifies the network port the collector should collect flow data. The command may only be present when the COLLECTOR-TYPE is TCP or UDP. If not specified, the default is 18000.

HOST hostname

This optional command specifies the IP or name of the host the collector should listen on (bind(2) to). Its value is the name of the host or its IP address. The command may only be present when the COLLECTOR-TYPE is TCP, UDP, or SPREAD. If SPREAD, then HOST is the hostname that the Spread Daemon is running on. The default is to listen on localhost.

PATH file path

This command specifies the file or directory path the collector should read from. If the COLLECTOR-TYPE is FILEHANDLER this should be the name of the IPFIX file to read and process. Otherwise, if the type is DIR or FILEHANDLER and the POLL keyword is present, PATH should be the directory path in which to poll files.

POLL POLL-TIME

This command specifies the time (seconds) between directory polls if the collector is defined as a DIR type. Also, if the collector is a FILEHANDLER and this keyword is present, the PATH keyword will be translated as directory path, and the super_mediator will run forever. The default is 30 seconds.

DAEMON

This specifies the name of the Spread Daemon to connect to. This keyword is only valid if COLLECTOR_TYPE is SPREAD.

GROUP

This specifies one and only one Spread Group Name to subscribe to. This keyword is only valid if COLLECTOR_TYPE is SPREAD. It is acceptable to have multiple GROUP keywords in the Collector block.

LOCK

When this command is given, super_mediator will not read files that have ".lock" appended to the filename. This keyword is only valid if the collector is setup to poll a directory. It is useful if yaf(1) is writing to rotating IPFIX files and the super_mediator is reading from that same directory. This prevents the super_mediator from pulling the file out from under yaf while it is still being written to. Note this is different from how the super_mediator will lock export files. See LOCK under the EXPORTER concept.

MOVE FILE_PATH

When this command is given and super_mediator is configured to poll a directory for IPFIX files, it will move the processed files to the FILE_PATH. You must specify either MOVE or DELETE for a collector if it is polling from a directory.

DELETE

When this command is given, super_mediator will delete the IPFIX files after they have been processed. If you don't want to delete the files, use the MOVE keyword.

DECOMPRESS_DIRECTORY FILE PATH

If present and the input file(s) are compressed (gzip'd), decompress the file to a temporary file within FILE_PATH. If If this option is not present, super_mediator will decompress files to the variable specified by the TMPDIR environment variable or /tmp if TMPDIR is not set. The zlib library must be installed to use this feature.

SILK_PROBE PROBE_NAME VLAN|INGRESS

This command was added in super_mediator 1.9.0 and it is only allowed when super_mediator has been compiled with SiLK Type and Sensor Labeling support.

This command assigns values to this COLLECTOR for use by a SILK_CONFIG block. If using a SILK_CONFIG block and this statement is not present, records from this COLLECTOR are not labeled. See the description of the "SILK_CONFIG" below for the meaning of the arguments.

AND_FILTER

If present, require a record to pass all filters in the COLLECTOR block to process the record. In effect, join the filters with a logical AND.

Collector Filters

Each collector can contain one or more filters to define what data should be collected. The syntax for the filters is the same as defined below in the "Filter Block". The filters defined in the collector block are only for the collector that contains them. The filters are by default are treated as OR filters: a record is processed if it passes ANY filter. Use AND_FILTER to make the filters AND filters. If a filter block is defined outside a COLLECTOR block, the filter will apply to all defined COLLECTORS. A filter statement should be not be listed directly before the "COLLECTOR END" statement. See "EXAMPLES".

The use of filter blocks is optional. They are used to filter out certain flows on collection. One and only filter block can be specified in the configuration file. However, more than one filter statement can be defined in the filter block.

The first command below is used at the top-level to begin a filter block, and the remaining commands are accepted within the filter block.

FILTER

A filter block starts with the FILTER keyword on a single line, and it continues to the FILTER END command. There is no argument to the FILTER command.

Filters are composed of comparisons. In each filter block, each comparison appears on a line by itself. By default, if any comparison in a filter returns a match or success, the flow record is sent through to the exporters. If none of the comparisons match, the flow record is dropped by the super_mediator. By default, all filters in super_mediator are "OR" filters; they only have to pass one comparison to succeed. If the user wants to make the filters "AND" filters, they can use the AND_FILTER keyword in the FILTER block or EXPORTER block. When the AND_FILTER keyword is present, all filters have to pass to succeed.

Each comparison is made up of three elements: a flow record field, an operator, and a compare value. A comparison is considered a match for a record if the expression created by replacing the field name with the field's value is true.

Available Fields

All the following fields can be used to filter data.

ANY_IP: Either the source IPv4 Address or destination IPv4 Address.
ANY_PORT: Either the source port or destination port.
SIP_V4: The source IPv4 Address.
DIP_V4: The destination IPv4 Address.
SPORT: The source port.
DPORT: The destination port.
PROTOCOL: The IP Protocol. This is an integer, where 6 is TCP, 17 is UDP.
APPLICATION: The service port of the record as set by yaf(1)'s silkAppLabel field. For example, this would be 80 if yaf recognizes the packets as being part of an HTTP session. See applabel(1).
SIP_V6: The source IPv6 Address.
DIP_V6: The destination IPv6 Address.
ANY_IP6: Either the source or destination IPv6 Address.
OBDOMAIN: The observation domain of the yaf(1) process as specified by --observation-domain on the yaf command line. If not specified, the observationDomainId defaults to 0. This could be used to distinguish between multiple yaf processes.
VLAN: The VLAN tag of the flow.
VERSION: The IP Version of the Flow. Valid values are 6 and 4. If sourceIPv6Address or destinationIPv6Address exists, version is 6.
COLLECTOR: This is only valid for EXPORTER filters. It is invalid within a COLLECTOR block and ignored in a FILTER block. It can be used with == or != operators and set to a COLLECTOR_NAME. If present, it will only export flows that were collected by that COLLECTOR.

Operators and Compare Values

There are eight operators that are supported. The operator determines the form that the compare value takes.

==: Succeeds when the value from the record is equal to the compare value.
!=: Succeeds when the value from the record is not equal to the compare value.
<: Succeeds when the value from the record is strictly less than the compare value.
<=: Succeeds when the value from the record is less than or equal to the compare value.
>: Succeeds when the value from the record is strictly greater than the compare value.
>=: Succeeds when the value from the record is greater than or equal to the compare value.
IN_LIST: Succeeds when the value from the record belongs to the given IPset. This operator is only valid for IP Addresses and IPSets. This operator is only valid of super_mediator is compiled with SiLK IPset support. The IPset must be a valid IPset. To compare any IP address (v4 or v6) use ANY_IP, SIP, or DIP.
NOT_IN_LIST: Succeeds when the value from the record does not belong to the given IPset. This operator is only valid for IP Addresses and IPSets. This operator is only valid if super_mediator is compiled with SiLK IPset support. The IPset must be a valid IPset. To compare any IP address (v4 or v6) use ANY_IP, SIP, or DIP.

AND_FILTER

If present, all filters listed in the FILTER block must pass in order to succeed. By default, all filters in super_mediator are "OR" filters.

FILTER END

The FILTER END command ends the definition of a group. Following a FILTER END command, top-level commands are again accepted.

Filter Example: Filter on DNS Traffic:

FILTER
    APPLICATION == 53
FILTER END

Filter Example: Filter on Ports 80 OR 53:

FILTER
    ANY_PORT == 80
    ANY_PORT == 53
FILTER END

Filter Example: Filter for IP(s) in IPset:

FILTER
    SIP IN_LIST "/data/sets/mysample.set"
FILTER END

Filter Example: Filter for IPv6 Address(es) in IPset:

FILTER
    SIP_V6 IN_LIST "/data/sets/mysamplev6.set"
FILTER END

Filter Example: Filter for DNS labeled traffic not on port 53:

FILTER
    APPLICATION == 53
    DPORT != 53
    AND_FILTER
FILTER END

Exporter Block

The configuration file must contain at least one exporter. However, it can contain multiple exporters. The exception is that it can only contain one Spread Exporter.

EXPORTER EXPORTER_TYPE EXPORTER_NAME

The EXPORTER command begins a new Exporter block and it continues to the EXPORTER END command. The EXPORTER_NAME is optional and will be used in the log if it is provided. The required argument to the EXPORTER command is the EXPORTER_TYPE. It must be one of the following:

TCP

The exporter will send IPFIX via TCP to the specified hostname or IP address and port.

UDP

The exporter will send IPFIX via UDP to the specified hostname or IP address and port. Note that UDP is not recommended, as it is not a reliable transport protocol, and thus cannot guarantee delivery of messages. The super_mediator will periodically send out templates as specified by the UDP_TEMP_TIMEOUT keyword.

SPREAD

The exporter will send IPFIX via Spread to the groups defined in the Exporter and Group blocks.

FILEHANDLER

The exporter will write IPFIX to a file. If the ROTATE keyword is present, the exporter will rotate output files every so many seconds. Rotated IPFIX files will have the file suffix, ".med".

TEXT

The exporter will write delimited text to a file or JSON if the keyword JSON is present within the EXPORTER block. Options present within the EXPORTER block will vary the format of the output. If the ROTATE keyword is present, the exporter will rotate output files every ROTATE_SECONDS seconds. Text flow data files, by default, will be in the form (text wrapped for readability):

start-time | end-time | dur | rtt | protocol | srcip | \
srcport | pkt | oct | attributes | mac | dstip | dstport | \
rpkt | roct | rev-attributes | dstmac | iflags | uflags | \
riflags | ruflags | tcpseq | revtcpseq | ingress| egress |
vlan | app | tos | end-reason | collector | payload | revpayload

start-time and end-time are in the form 2012-01-28 13:12:32.786. Using the included program, super_table_creator(1), which is available if mysql is installed, you can create a MySQL table for the full flow, by running super_table_creator with --flow-only.

Unless FLOW_ONLY is present, a TEXT Exporter will write every IPFIX field it decodes into the file given to PATH, as well as yaf process statistics records. Statistics records will be labeled with the word stats and are in the following form (text wrapped for readability):

stats | total_flows | total_packets | dropped_packets | \
ignored_packets | expired_fragments | assembled_fragments |\
flush_events | flow_table_peak_count | sensor_IP | \
process_id | mean_flow_rate | mean_pkt_rate

Example:

stats|2|56|0|0|0|0|2|1|127.0.0.1|0|131|3685

If DPI data exists with the flow, super_mediator will write one line of flow data in the above format followed by a flow index line and one or more lines containing the DPI meta data.

Example EXPORTER TEXT configuration:

EXPORTER TEXT
   PATH "/data/flow.txt"
EXPORTER END

Example output with above configuration:

2012-04-03 04:42:55.606|2012-04-03 04:45:13.738| 138.132|   0.088|  6| 10.10.1.102| 2592| 30| 1591|00|00:00:00:00:00| 10.10.34.130| 7000| 24| 8001|00|00:00:00:00:00|  S| APRF|  AS| AP|b3332bea|ead9dce8|000| 194|000|000|
irc|125|NICK nickname
irc|125|USERHOST Omega
irc|125|VERSION mIRC v6
irc|125|NAMESX
stats|2|56|0|0|0|0|2|1|10.20.11.51|0|131|3685

If DPI_ONLY is present, the line of flow data will condensed to the following format and labeled with the word flow by default (text wrapped for readability):

flow | flow_key_hash | start-time-ms | srcip | dstip | \
protocol | srcport | dstport | vlan | obid

Example:

flow|109074684|1207197775606|10.10.1.102|10.10.34.130|6|2592|7000|0|0

flow_key_hash is a 32-bit hash of the 5-tuple + vlan.
start-time-ms is the milliseconds since Epoch time.

For all protocols except DNS, SSL/TLS, and DNP3 the output follows the following format. Each line contains the default table name, information element id, flow key hash, flow start time in milliseconds, observation domain id, and the data. There will be one line for each data field associated with the flow (text wrapped for readability):.

table_name | element_id | data

If DEDUP_PER_FLOW is present for a TEXT EXPORTER, the format for each DPI line will be (see below for a description of DEDUP_PER_FLOW):

table_name | element_id | hitcount | data

If DPI_ONLY is present for a TEXT EXPORTER, any DPI data associated with the flow will have the following form (note that DPI_ONLY adds the flow_key_hash, start_time_ms, and observation domain id to the DPI data):

table-name | flow_key_hash | start_time_ms | obid | elem_id | data

If DPI_ONLY and DEDUP_PER_FLOW are both present, the format will have the following form:

table-name | flow_key_hash | start_time_ms | obid | elem_id | hitcount | data

Example EXPORTER TEXT config with DPI_ONLY:

EXPORTER TEXT
    PATH "/data/flow.txt"
    DPI_ONLY
EXPORTER END

Example Output with above config:

http|1441601726|1207802496583|115|http://en.wikipedia.org/wiki/Http
http|1441601726|1207802496583|114|HTTP/1.0
http|1441601726|1207802496583|114|HTTP/1.0
http|1441601726|1207802496583|117|en.wikipedia.org
flow|1441601726|1207802496583|128.237.224.172|208.80.152.2|6|1360|80|0

Example EXPORTER TEXT config with DPI_ONLY and DEDUP_PER_FLOW:

EXPORTER TEXT
    PATH "/data/flow.txt"
    DPI_ONLY
    DEDUP_PER_FLOW
EXPORTER END

Example Output with above config:

http|1441601726|1207802496583|115|1|http://en.wikipedia.org/wiki/Http
http|1441601726|1207802496583|114|2|HTTP/1.0
http|1441601726|1207802496583|117|1|en.wikipedia.org
flow|1441601726|1207802496583|128.237.224.172|208.80.152.2|6|1360|80|0

See below for a list of information element ids and the default table names. See yafdpi(1) for descriptions of each of the information elements. The default flow index and table names can be configured in the DPI_CONFIG block. The DPI_CONFIG block will also configure the super_mediator to write only particular information elements.

The format of the DNS CSV output is as follows (Note: This is different from the de-duplicated DNS output):

table_name | QR | dnsID | section | nxdomain | authoritative | \
response_type | ttl | name | value

QR: whether the record is a Query(Q) or a Response(R)
dnsID: the transaction ID from the DNS record
section: the section of the packet the resource record was extracted from (0-Query, 1-Answer, 2-Name Server, 3-Additional)
nxdomain: whether the record was an NXDomain(1) or not(0)
authoritative: whether the response is from an authoritative name server(1) or not(0)
response_type: the TYPE field of the DNS resource record
ttl: the time to live from the resource record
name: the Query or Response Name
value: the RDATA field from the resource record

DNS Example with DPI_ONLY:

flow|114422227|1207802496560|128.237.224.172|128.2.1.10|17|1599|53|0
dns|114422227|1207802496560|0|Q|14728|0|0|0|1|0|meta.wikimedia.org.

yaf version 2.3.0 changed the format of X.509 Certificate export. If using version 2.3.0 or later, the format of TLS/SSL CSV will be as follows:

table_name | elem_id | [I|S|E] | cert seq no. | data

Note that SSL Certificate Extension fields are only exported if specifically set in the SSL_CONFIG block.

elem_id: is the object identifier as given in the X.509 ASN.1 RelativeDistinguishedName sequence. A list of common identifiers are listed below with the element ID numbers.
ISE: denotes if the data came from an Issuer Field(I), Subject Field(S), or Extension Field (E). For fields that are not associated with the issuer, subject, or extension but describe other characteristics of the certificate, an I will be used (for example, not-before or not-after timestamps).
cert seq no: signifies which certificate the data came from in the certificate chain. Usually, this field will contain a 0, 1, or 2.

DNP3.0 also has a different format. DNP3.0 will be written in the following form:

table_name | elem_id [284] | dnp src addr | dnp dst addr | \
dnp function | dnp data

elem_id: always 284
dnp src addr: the source address found in the packet payload
dnp dst addr: the destination address found in the packet payload
dnp function: the function code describing the function of the following dnp data
dnp data: the bytes captured by the regular expression executed by yaf written in hexadecimal

modbus and ethernet/IP data will also be written in hexadecimal.

As of yaf version 2.3.0, yaf can export enhanced flow metrics when running yaf with --flow-stats. By default, super_mediator will print the flow-stats to the TEXT file given to PATH. flow-stats will be written in the following form (text wrapped for readability) and will directly follow the flow they refer to:

flowstats | tcpUrgTotalCount | smallPacketCount | nonEmptyPacketCount | \
dataByteCount | averageInterarrivalTime | \
firstNonEmptyPacketSize | largePacketCount | maxPacketSize |\
firstEightNonEmptyPacketDirections | \
standardDeviationPayloadLength | \
standardDeviationInterarrivalTime | \
averagePacketSize | reverseTcpUrgTotalCount | \
reverseSmallPacketCount | reverseNonEmptyPacketCount | \
reverseDataByteCount | reverseAverageInterarrivalTime | \
reverseFirstNonEmptyPacketSize | reverseLargePacketCount | \
reverseMaxPacketSize | reverseStandardDeviationPayloadLength |\
reverseStandardDeviationInterarrivalTime | reverseAveragePayloadLength

For descriptions of these information elements, see the yaf man page.

FlowStats Example:

flowstats|0|1|1|49|0|49|0|49|00|0|0|49|0|0|0|0|0|0|0|0|0|0|0|0

For a more custom TEXT output, use the FIELDS keyword. See the documentation below.

EXPORTER END

The EXPORTER END command ends the definition of an exporter. Following an EXPORTER END command, top-level commands are again accepted.

PORT PORT

Specifies the port the exporter should write to. This command may only be present when the EXPORTER_TYPE is TCP or UDP.

HOST HOST

Specifies the hostname or IP address of the collector to which the flows should be exported. If the EXPORTER_TYPE is SPREAD and the Spread daemon is running on a remote host, HOST should specify the host name or IP address the Spread daemon is running on.

PATH PATH

Specifies the name of the file to write to, or the directory to write rolling IPFIX or TEXT Files if ROTATE is defined.

DAEMON SPREAD_DAEMON_NAME

Specifies the name of the Spread Daemon the exporter should connect to.

GROUP GROUP_NAME

The Spread Group name the exporter should publish messages to. Only one group name per line. Each exporter can have multiple GROUP commands. To set a filter for each Spread Group, use the Group Block.

LOCK

If specified, super_mediator will prepend "." to a file that it is currently writing to. This is can be used with rwsender(8), so that rwsender doesn't move the file out from under the super_mediator. Once the file is closed, the dot will be removed from the filename.

DELIMITER DELIMITER

If specified, super_mediator will use the single character DELIMITER to separate flow fields when writing to a text file. The default is |. Only valid for TEXT Exporters.

DPI_DELIMITER DELIMITER

If specified, super_mediator will use the single character DELIMITER to separate DPI fields when writing to a text file. If not specified, the DELIMITER is the same as DELIMITER above, which by default is |. Using a different delimiter than above will potentially cause two different delimiters to be used on the same line. This may be useful when uploading text files to a database with the desire to use one column for DPI fields. Only valid for TEXT Exporters.

ROTATE ROTATE_SECONDS

If specified, rotate output files every ROTATE_SECONDS. Only valid for FILEHANDLER Exporters. If the super_mediator is not receiving any flow data, files will not be rotated. super_mediator uses flow end time in the incoming flow records to determine the current time and when to rotate files. Text filenames use the flow end time when rotating files for indexing purposes. If MULTI_FILES is present, by default super_mediator will rotate files using a serial number, not a timestamp in the filename. If TIMESTAMP_FILES is present, it will timestamp the files instead of using the serial number. IPFIX Exporters use system time when rotating files.

GZIP_FILES

When this command is present, super_mediator will compress the output files after it is done writing to them, if gzip is available.

MOVE FILE_DIRECTORY

When this command is present, super_mediator will move the output files to FILE_DIRECTORY, which must be a valid file directory, after it is done writing to them.

UDP_TEMP_TIMEOUT TIMEOUT_MINS

If specified, send templates out 3 times in TIMEOUT_MINS. By default, as per the recommendations in RFC 5101, super_mediator will retransmit templates three times within 10 minutes.

NO_FLOW

If present, do not forward basic flow records to this EXPORTER. This is typically used if other de-duplication features are enabled. Flow export is enabled by default unless SSL_DEDUP_ONLY, DEDUP_ONLY, STATS_ONLY, DNS_DEDUP_ONLY, DNS_RR_ONLY or NO_FLOW is present.

FLOW_ONLY

If specified, only forward basic flow information to the exporter. This should be used with SiLK collectors, such as rwflowpack(8) or flowcap(8), as they do not collect Deep Packet Inspection data. If present, super_mediator will not forward or write stats messages.

DPI_ONLY

If specified, only export flows that have some Deep Packet Inspection data associated with it. If the exporter has EXPORTER_TYPE of TEXT, the super_mediator will write a flow index line and associated DPI data to the output file. (See above TEXT for format and examples). super_mediator will not write stats messages. For advanced configuration of the DPI to CSV export use the DPI_CONFIG block.

DNS_RESPONSE_ONLY

If present, only export DNS responses. This will ignore all DNS queries. This option is ignored if no DNS DPI data is present in the flow.

MULTI_FILES

Only valid if DPI_ONLY is also present. Only valid for TEXT Exporters. If present, the super_mediator will separate DPI data based on application protocol into separate files in the file directory given to PATH, which must exist prior to starting the super_mediator. For advanced configuration of the filenames, edit the DPI_CONFIG block. This is useful if the mysqlimport tool will be loading the CSV output from the super_mediator. The mysqlimport tool loads tables from text files. The base name of the text file must be the name of the table that should be used. For a list of the default table names and information elements they contain, see below. If MULTI_FILES is present, the CSV does not contain the table name. The EXPORTER will write flow index lines in the following form to a separate file flow.txt0:

flow_key_hash | start_time_ms | srcip | dstip | \
protocol | srcport | dstport | vlan | obid

DPI data (all protocols except DNS, SSL, and DNP) will be written in the following form:

flow_key_hash | start_time_ms | ob-id | elem_id | data

If DEDUP_PER_FLOW is also present, the format will be:

flow_key_hash | start_time_ms | ob-id | elem_id | count | data

Exceptions:

DNS will be written in the following form (different from DNS_DEDUP form. DNS_DEDUP is not permitted if MULTI_FILES is present). See above under TEXT for a description of each field:

flow_key_hash | start_time_ms | obid | [Q|R] | dnsID | \
section | nxdomain | authoritative | response_type | \
ttl | name | value

SSL/TLS will be written in the following form (See above under TEXT for a description of each field):

elem_id | flow_key_hash | start_time_ms | obid | [I|S|E] | \
cert_no_seq | data

DNP3.0 will be written in the following form:

elem_id | flow_key_hash | start_time_ms | obid | \
dnpsrcaddress | dnpdstaddress | dnpfunction | dnpdata

RTP will be written in the following form:

elem_id | flow_key_hash | start_time_ms | obid |
payloadType | reversePayloadType

Flow-stats will be written as described above, except the line will not include the table name [flowstats].

Example EXPORTER MULTI_FILES Configuration:

EXPORTER TEXT
    PATH "/data/dpi"
    DPI_ONLY
    MULTI_FILES
    ROTATE 600
    LOCK
EXPORTER END

Example Data in /data/dpi/flow.txt0:

109074684|1207197775606|10.10.1.102|10.10.34.130|6|2898|7000|0|0

Example Data in /data/dpi/irc.txt0:

109074684|1207197775606|0|125|NICK OmegaT
109074684|1207197775606|0|125|USERHOST OmegaT
109074684|1207197775606|0|125|VERSION mIRC v6

In the above example, the three lines were written to irc.txt0 because information element id 125 has the default label irc. The default labels are listed below. They can be modified in the DPI_CONFIG block. The files will rotate if ROTATE is present in the EXPORTER Block. The files will "lock" if LOCK is present in the EXPORTER Block. Enclosed with the super_mediator distribution is super_table_creator, a program that creates the default MySQL tables for default super_mediator CSV output. The following is an example of using the super_table_creator and mysqlimport tool with super_mediator output:

super_table_creator -n username -p password -d super_db

mysqlimport -u user -p --fields-terminated-by="|" super_db irc.txt0

Alternatively, you can use the MySQL LOAD DATA INFILE command to load the first three columns from a file into the "super" database:

mysql -u user -p -e "LOAD DATA INFILE '/data/dpi/flow.txt0' into table super_flows FIELDS TERMINATED by '|' (column1, column2, column3);" super

super_mediator can also be configured to do the importing if the MySQL client libraries are installed and if the MySQL credentials are listed in the EXPORTER block. The EXPORTER block must have MYSQL_USER, MYSQL_PASSWORD, and MYSQL_DATABASE all present for super_mediator to attempt the import. super_mediator uses the default database schemas used by the super_table_creator. super_table_creator contains schemas for all DPI protocols, DNS-deduplication, and flow.

If NO_INDEX is also present in the EXPORTER BLOCK, the super_mediator will not write to a flow index file. For each line in the DPI files, the flow key hash will be expanded into its components and the timestamp will be printed in human-readable format (text wrapped for readability):

start-time | srcip | dstip | protocol | srcport | \
dstport | vlan | obid | elem_id | data

Example:

2008-04-10 04:41:36.583|1.2.3.4|5.6.7.8|6|1360|80|0|0|114|HTTP/1.1

The follow example shows an example configuration file and sample data using the NO_INDEX and TIMESTAMP_FILES options:

Example NO_INDEX Exporter Configuration:

EXPORTER TEXT
    PATH "/data/dpi"
    DPI_ONLY
    MULTI_FILES
    ROTATE 600
    LOCK
    TIMESTAMP_FILES
    NO_INDEX
EXPORTER END

Example Data in /data/dpi/http.txt20080410044142:

2008-04-10 04:41:36.582|1.2.3.4|5.6.7.8|6|1359|80|0|0|111|Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
2008-04-10 04:41:36.582|1.2.3.4|5.6.7.8|6|1359|80|0|0|112|/w/index.php?
2008-04-10 04:41:36.582|1.2.3.4|5.6.7.8|6|1359|80|0|0|114|HTTP/1.1

Example Data in /data/dpi/dns.txt20110128215727:

2011-01-28 21:52:23.473|1.2.3.4|6.7.8.9|17|49664|53|905|0|Q|525|0|0|0|1|0|www.dropbox.com

NO_STATS

If preset, the exporter will not write or forward stats messages from yaf(1).

STATS_ONLY

If present, the exporter will write or forward only stats messages from yaf(1). If this is present after the DPI_ONLY or FLOW_ONLY keywords, it will turn on stats messages in the exporter. The stats messages will not be prefaced "stats|". Instead they will be prefaced with "\N|" so that when they are imported into a database, the timestamp field will be updated to the current time.

DNS_DEDUP

If present, the super_mediator will perform DNS de-duplication of yaf(1) IPFIX flow records with DNS information and write the aggregated DNS records to the exporter. Super_mediator will also write previously de-duplicated DNS records that are received in the following format. DNS de-duplicated records for TEXT exporters have the format:

first_seen | rrtype | rrname | rrval

first_seen is a timestamp in the form 2012-01-23 04:45:13.897. DNS_DEDUP is not valid if MULTI_FILES is also present. It is recommended to separate the DNS_DEDUP and MULTI_FILES into separate exporters. See super_mediator(1) for a complete description of the information elements exported for IPFIX Exporters.

DNS_DEDUP_ONLY

If present, super_mediator will only write DNS de-duplicated records to this exporter. It will not write any flow or other Deep Packet Inspection data that it collects. Not valid if MULTI_FILES, SSL_DEDUP_ONLY, DEDUP_ONLY, <DNS_RR_ONLY>, or <FLOW_ONLY> is also present.

DNS_RR_ONLY FULL

If present, super_mediator will only write DNS resource records to this exporter. It will not write any flow or other Deep Packet Inspection data that it collects. These records are not de-duplicated. Every DNS resource record that super_mediator receives will be transmitted. Not valid if MULTI_FILES, SSL_DEDUP_ONLY, DEDUP_ONLY, or FLOW_ONLY is also present. The IPFIX Information Elements that are exported are as follows (in order). The exporter will only export the following fields labeled with FULL if the word FULL is present in the configuration file:

flowStartMilliseconds IE 152, 8 octets, unsigned: Flow start time in milliseconds since 1970-01-01 00:00:00 UTC. Always present.
sourceIPv6Address IE 27, 16 octets, unsigned, FULL: IPv6 address of flow source or biflow initiator. Present for IPv6 flows or IPv6-mapped IPv4 flows only.
destinationIPv6Address IE 28, 16 octets, unsigned, FULL: IPv6 address of flow source or biflow responder. Present for IPv6 flows or IPv6-mapped IPv4 flows only.
sourceIPv4Address IE 8, 4 octets, unsigned, FULL: IPv4 address of flow source or biflow initiator. Present for IPv4 flows without IPv6-mapped addresses only.
destinationIPv4Address IE 12, 4 octets, unsigned, FULL: IPv4 address of flow source or biflow responder. Present for IPv4 flows without IPv6-mapped addresses only.
dnsTTL CERT (PEN 6871) IE 199, 4 octets, unsigned: DNS Time To Live. This is an unsigned integer that specifies the time interval, in seconds, that the resource record may be cached for. This will contain a value of zero for DNS Queries
observationDomainId IE 149, 4 octets, unsigned: An identifier of an Observation Domain that is locally unique to an Exporting Process. This is typically set on the yaf(1) command line.
yafFlowKeyHash CERT (PEN 6871) IE 106, 4 octets, unsigned: The hash of the 5-tuple (sourceIPAddress, destinationIPAddress, sourcePort, destinationPort, protocol) and vlanId.
dnsQRType CERT (PEN 6871) IE 175, 2 octets, unsigned: DNS Query/Response Type. This corresponds with the QTYPE field in the DNS Question Section or the TYPE field in the DNS Resource Record Section. This field determines the type of data found in the dnsRName field.
sourceTransportPort IE 7, 2 octets, unsigned, FULL: TCP or UDP port on the flow source or biflow initiator endpoint. Always present.
destinationTransportPort IE 11, 2 octets, unsigned, FULL: TCP or UDP port on the flow destination or biflow responder endpoint. Always present. For ICMP flows, contains ICMP type * 256 + ICMP code. This is non-standard, and an open issue in yaf.
vlanId IE 58, 2 octets, unsigned, FULL: 802.1q VLAN tag of the first packet in the forward direction of the flow.
dnsID CERT (PEN 6871) IE 226, 2 octets, unsigned: DNS Transaction ID. This identifier is used by the requester to match up replies to outstanding queries
protocolIdentifier IE 4, 1 octet, unsigned, FULL: IP protocol of the flow.
dnsQueryResponse CERT (PEN 6871) IE 174, 1 octet, unsigned: DNS Query/Response header field. This corresponds with the DNS header one bit field, QR. If the message is a query (0), or a response (1).
dnsAuthoritative CERT (PEN 6871) IE 176, 1 octet, unsigned: DNS Authoritative header field. This corresponds with the DNS header one bit field, AA. This bit is only valid in responses (when dnsQueryResponse is 1), and specifies that the responding name server is an authority for the domain name in the question section.
dnsNXDomain CERT (PEN 6871) IE 177, 1 octet, unsigned: DNS NXDomain or Response Code (RCODE). This corresponds with the DNS RCODE header field. This field will be set to 3 for a Name Error, 2 for a Server Failure, 1 for a Format Error, and 0 for No Error. See http://www.iana.org/assignments/dns-parameters for other valid values.
dnsRRSection CERT (PEN 6871) IE 178, 1 octet, unsigned: DNS Resource Record Section Field. This field will be set to 0 if the information is from the Question Section, 1 for the Answer Section, 2 for the Name Server Section, and 3 for the Additional Section.
dnsQName CERT (PEN 6871) IE 179, variable length: A DNS Query or Response Name. This field corresponds with the QNAME field in the DNS Question Section or the NAME field in the DNS Resource Record Section.
dnsRName CERT (PEN 6871) IE 927, variable length: The DNS Resource Record Data field. The information contained in this field depends on the type of resource record. For an A record, this will be the resolving IPv4 Address. For an AAAA record, this will be the resolving IPv6 Address. For a NS record, it will be the NSDNAME. For a CNAME Record, this will contain a CNAME. For a SOA Record, this will contain the SOA MNAME field. For a PTR Record, this will contain the PTRDNAME. For a MX Record, this will contain the MX Exchange field. For a TXT Record, this will contain the TXT-DATA field. For a SRV Record, this will contain the Target field.

DNS_RR FULL

If present, super_mediator will write DNS resource records to this exporter. These records are not de-duplicated. Every DNS resource record that super_mediator receives will be exported. The IPFIX Information Elements that are exported as described above under DNS_RR_ONLY. The option FULL keyword will export the additional 5-tuple. Only valid for IPFIX Exporters.

TIME_UNITS UNITS

This setting specifies the time units to use when exporting flow records as TEXT or JSON, such as millisecons or nanoseconds. The setting also enables export of multiple timestamps. The setting affects the FIELDS STIME (start time), ETIME (end time), DURATION, and RTT (round-trip-time).

TIME_UNITS may also be set at top-level. When set there, it affects all exporters that follow the command, though exporters may set their own TIME_UNITS to override the top-level setting.

If TIME_UNITS is not specified on a TEXT exporter, the default is to use milliseconds if FIELDS is not specified or the timestamp with the greatest precision otherwise. If not specified, a JSON exporter exports all timestmaps that were present on the incoming record. TIME_UNITS was added in super_mediator-1.11.0.

The forms of the command are:

TIME_UNITS BEST: Use the time units that are present on the incoming record. If timestamps appear multiple times on the record, use the one with the greatest precision. This is the default for TEXT exports when FIELDS is specified.
TIME_UNITS INCOMING: Use the time units that are present on the incoming record. If timestamps appear multiple times on the record, include all timestamps in the output. This is the default for JSON exports.
TIME_UNITS NOFRAC: Export the times as a whole number of seconds, e.g., flowStartSeconds.
TIME_UNITS MILLI: Export the times to millisecond (precision, e.g., flowStartMilliseconds. This is the default for TEXT exporters when FIELDS is not specified.
TIME_UNITS MICRO: Export the times to microsecond precision, e.g., flowStartMicroseconds.
TIME_UNITS NANO: Export the times to nanosecond precision, e.g., flowStartNanoseconds.
TIME_UNITS [ {NOFRAC | MILLI | MICRO | NANO} ,... ]: The TIME_UNITS command also accepts a comma-separated list of values surrounded by square brackets, which causes the timestamps to appear multiple times in the output. The list may include any value except BEST and INCOMING.
TIME_UNITS [ ]: An empty list reverts the time-units back to the default value. When set within an exporter block, it overrides any global setting.

FIELDS FIELDS

If present for TEXT Exporters, the super_mediator will write only the fields contained in FIELDS. FIELDS contains the list of flow attributes (a.k.a. fields or columns) to print. The columns will be displayed in the order the fields are specified (DPI is the exception). FIELDS is a comma or space separated list of field-names or field-integers, and a field identifier may be repeated. Example:

FIELDS stime,ETIME,0,1,SPORT,DPORT,dpi

YAF process statistics are not enabled by default when a FIELD list is specified. Use STATS_ONLY to enable stats with custom field lists.

The complete list of built-in fields that super_mediator supports follows:

SIP,sip,0

Source IP address expressed in its canonical format. See also SIP_INT.

DIP,dip,1

Destination IP address expressed in its canonical format. See also DIP_INT.

SPORT,sport,4

Source port for TCP and UDP. For ICMP flow, ICMP type.

DPORT,dport,5

Destination port for TCP and UDP. For ICMP flows, ICMP code.

PROTOCOL,protocol,6

IP protocol.

APPLICATION,APP,application,app,7

Application label as reported by yaf.

OBDOMAIN,domain,13

Observation domain from IPFIX header or from the incoming IPFIX record if --preserve-obdomain was specified.

VLAN,vlan,15

VLAN ID, expressed in hexadecimal. See VLANINT for integer version.

FLOWKEYHASH,HASH,flowkeyhash,hash,16

Flow key hash of the 5 tuple. Used to correlate flows.

DURATION,duration,dur,17

Flow duration in floating point seconds. The number of decimal places matches the precision of the flow start and end times, and in JSON output the key reflects the precision of time interval. The format is affected by the TIME_UNITS value and the field may produce multiple output fields.

STIME,stime,18

Flow start time in ISO 8601 format with fractional seconds; for example YYYY-MM-DD hh:mm:ss.sss. The fractional seconds are determined by the TIME_UNITS setting, which may also cause this field to produce multiple output fields. See also STIME_EPOCH and STIMEMS.

ETIME,etime,19

Flow end time in ISO 8601 format with fractional seconds; for example YYYY-MM-DD hh:mm:ss.sss. The fractional seconds are determined by the TIME_UNITS setting, which may also cause this field to produce multiple output fields. See also ETIME_EPOCH and ETIMEMS.

STIMEMS,stimems,20

Flow start time as an integer representing the number of milliseconds since 1970-01-01 00:00:00 UTC. See also STIME_EPOCH and STIME.

ETIMEMS,etimems,21

Flow end time as an integer representing the number of milliseconds since 1970-01-01 00:00:00 UTC. See also ETIME_EPOCH and ETIME.

SIP_INT,SIPINT,sipint,22

Source IPv4 address as a 32-bit integer.

DIP_INT,DIPINT,dipint,23

Destination IPv4 address as a 32-bit integer.

RTT,rtt,24

Difference between the timestamps of the first packet in the forward direction and the first packet in the reverse direction. This can be an estimate of the round-trip time. The difference is expressed in floating point seconds, and the number of decimal places matches the precision of the flow start and end times. In JSON output, the key reflects the precision of time interval. The format is affected by the TIME_UNITS value and the field may produce multiple output fields.

PACKETS,pkts,25

Forward packet count.

RPACKETS,rpkts,26

Reverse packet count.

BYTES,bytes,27

Forward octet count.

RBYTES,rbytes,28

Reverse octet count.

IFLAGS,iflags,29

TCP flags on the first packet in the forward direction, where each flags bit is represented by the first character in the flag's name: FIN, SYN, RST, PSH, ACK, URG, ECE, CWR.

RIFLAGS,riflags,30

TCP flags on the first packet in the reverse direction, where each flags bit is represented by the first character in the flag's name: FIN, SYN, RST, PSH, ACK, URG, ECE, CWR.

UFLAGS,uflags,31

TCP flags union across all forward packets except the first, where each flags bit is represented by the first character in the flag's name: FIN, SYN, RST, PSH, ACK, URG, ECE, CWR.

RUFLAGS,ruflags,32

TCP flags union across all reverse packets except the first, where each flags bit is represented by the first character in the flag's name: FIN, SYN, RST, PSH, ACK, URG, ECE, CWR.

ATTRIBUTES,attributes,33

Flow attributes in hexadecimal format.

RATTRIBUTES,rattributes,34

Reverse flow attributes in hexadecimal format.

MAC,mac,35

Source MAC address.

DSTMAC,dstmac,36

Destination MAC address.

TCPSEQ,tcpseq,37

Forward initial TCP sequence number in hexadecimal format.

RTCPSEQ,rtcpseq,38

Reverse initial TCP sequence number in hexadecimal format.

ENTROPY,entropy,39

Shannon-Fano Entropy for the forward flow.

RENTROPY,rentropy,40

Shannon-Fano Entropy for the reverse flow.

ENDREASON,endreason,41

If not present, the flow ended normally (i.e., by TCP RST or FIN). Otherwise it is one of the following:

idle: Flow was expired by idle timeout. No packets were received for IDLE_TIMEOUT seconds (see yaf(1)) and the flow was presumed closed.
active: Flow was expired by active timeout. The flow's duration was longer than ACTIVE_TIMEOUT seconds (see yaf(1)) and the flow was flushed from the flow table.
eof: Flow was still active in the flow table at the end of the dumpfile or at yaf shutdown time; it was flushed as the flow table was cleared.
rsrc: Flow was prematurely flushed as idle because more than FLOW_TABLE_MAX flows (see yaf(1)) were active in the flow table.
force: yaf forced a write of the flow, but the flow remained open.

OSNAME,osname,42

p0f OS Name for the forward flow.

OSVERSION,osversion,43

p0f OS Version for the forward flow.

ROSNAME,rosname,44

p0f OS Name for the reverse flow.

ROSVERSION,rosversion,45

p0f OS Version for the reverse flow.

OSFINGERPRINT,FINGERPRINT,osfingerprint,fingerprint,46

p0f OS Fingerprint for the forward flow.

ROSFINGERPRINT,RFINGERPRINT,rosfingerprint,rfingerprint,47

p0f OS Fingerprint for the reverse flow.

DHCPFP,dhcpfp,48

DHCP Fingerprint, usually OS or Hardware name.

DHCPVC,dhcpvc,49

DHCP Vendor class ID found in Option 60 of the DHCP packet.

RDHCPFP,rdhcpfp,50

DHCP Fingerprint for reverse flow.

RDHCPVC,rdhcpvc,51

DHCP Vendor class ID found in Option 60 of the DHCP packet for the reverse flow.

INGRESS,ingress,52

Index of the IP interface where packets of the flow were received.

EGRESS,egress,53

Index of the IP interface where packets in the reverse direction of the flow were received.

DATABYTES,databytes,54

dataByteCount field in flow statistics. See yaf(1).

RDATABYTES,rdatabytes,55

reverseDataByteCount field in flow statistics. See yaf(1).

ITIME,itime,56

averageInterarrivalTime field in flow statistics in fractional seconds. See yaf(1).

RITIME,ritime,57

reverseAverageInterarrivalTime field in flow statistics in fractional seconds. See yaf(1).

STDITIME,stditime,58

standardDeviationInterarrivalTime field in flow statistics. See yaf(1).

RSTDITIME,rstditime,59

reverseStandardDeviationInterarrivalTime field in flow statistics. See yaf(1).

TCPURG,tcpurg,60

tcpUrgTotalCount field in flow statistics. See yaf(1).

RTCPURG,rtcpurg,61

reverseTcpUrgTotalCount field in flow statistics. See yaf(1).

SMALLPKTS,smallpkts,62

smallPacketTotalCount field in flow statistics. See yaf(1).

RSMALLPKTS,rsmallpkts,63

reverseSmallPacketTotalCount field in flow statistics. See yaf(1).

LARGEPKTS,largepkts,64

largePacketCount field in flow statistics. See yaf(1).

RLARGEPKTS,rlargepkts,65

reverseLargePacketCount field in flow statistics. See yaf(1).

NONEMPTYPKTS,nonemptypkts,66

nonEmptyPacketCount field in flow statistics. See yaf(1).

RNONEMPTYPKTS,rnonemptypkts,67

reverseNonEmptyPacketCount field in flow statistics. See yaf(1).

MAXSIZE,maxsize,68

maxPacketSize field in flow statistics. See yaf(1).

RMAXSIZE,rmaxsize,69

reverseMaxPacketSize field in flow statistics. See yaf(1).

STDPAYLEN,stdpaylen,70

standardDeviationPayloadLength field in flow statistics. See yaf(1).

RSTDPAYLEN,rstdpaylen,71

reverseStandardDeviationPayloadLength field in flow statistics. See yaf(1).

FIRSTEIGHT,firsteight,72

firstEightNonEmptyPacketDirections field in flow statistics in hexadecimal format. See yaf(1).

DPI,dpi,73

Deep Packet Inspection information. For all protocols, except DNS and TLS/SSL, adding DPI to the field list will add the information element id and data value to the end of the line regardless of what order DPI is in the list. DPI information will always be at the end of the line. For each DPI field captured by YAF, there will be one line in the output text file. To configure the DPI fields super_mediator exports, use the DPI_FIELD_LIST or the DPI_CONFIG block options. If both DPI_FIELD_LIST and DPI_CONFIG are present in the configuration file, the elements listed in the DPI_FIELD_LIST will take priority for that particular exporter. All protocols except DNS and SSL will add the following columns to the end of the line:

elem_id | data

For DNS, the following fields will be added to the end of the line:

QR | dnsID | section | nxdomain | authoritative | \
response_type | ttl | name | value

For SSL, the following fields will be added to the end of the line:

elem_id | IS | cert seq no. | data

See above (under TEXT) for explanations of the fields.

VLANINT,vlanint,74

The VLAN tag of the flow exported as an integer.

TOS,tos,75

Type of Service field from the IP Header.

RTOS,rtos,76

Type of Service field from the IP header of the reverse flow.

MPLS1,mpls1,77

The top of stack MPLS label.

MPLS2,mpls2,78

The second MPLS label in the stack.

MPLS3,mpls3,79

The third MPLS label in the stack.

COLLECTOR,collector,80

Name of the collector that received the flow.

FIRSTNONEMPTY,firstnonempty,81

The firstNonEmptyPacketSize field in flow statistics. See yaf(1).

RFIRSTNONEMPTY,rfirstnonempty,82

The reverseFirstNonEmptyPacketSize field in flow statistics. See yaf(1).

MPTCPSEQ,mptcpseq,83

The initial data sequence number found in the MPTCP Data Sequence Signal (DSS) option. See yaf(1).

MPTCPTOKEN,mptcptoken,84

The token used to identify an MPTCP connection over multiple subflows. This value is found in the MP_JOIN TCP Option for the initial SYN of a subflow. See yaf(1).

MPTCPMSS,mptcpmss,85

The maximum segment size reported in the Maximum Segment Size TCP Option. This should be consistent over all subflows. See yaf(1).

MPTCPID,mptcpid,86

The address ID of the subflow found in the SYN/ACK of an MP_JOIN operation. See yaf(1).

MPTCPFLAGS,mptcpflags,87

Various MPTCP Values. See yaf(1).

PAYLOAD,payload,88

The initial n bytes of forward direction of the flow's payload. Present if yaf payload export is enabled. payload is in hexdump format and are prefixed with the string ->. For JSON export mode, the payload is base64-encoded.

RPAYLOAD,rpayload,89

The initial n bytes of reverse direction of the flow's payload. Present if yaf payload export is enabled. rpayload is in hexdump format and are prefixed with the string <-. For JSON export mode, the rpayload is base64-encoded.

DHCPOPTIONS,dhcpoptions,90

A comma separated list of DHCP Options in the order they were requested. See yafdhcp(1).

RDHCPOPTIONS,rdhcpoptions,91

A comma separated list of DHCP Options in the reverse flow as they were requested. See yafdhcp(1).

NDPI_MASTER,ndpimaster,92

NDPI Layer 7 Master protocol identified by the nDPI library via yaf(1).

NDPI_SUB,ndpisub,93

NDPI Layer 7 Sub/secondary protocol identified by the nDPI library via yaf(1).

STIME_EPOCH,stime_epoch,94

Flow start time expressed as floating point seconds since the UNIX epoch, 1970-01-01 00:00:00. The precision reflects that of the record, and the TIME_UNITS setting has no effect on this field. See also STIMEMS and STIME. Since super_mediator-1.11.0.

ETIME_EPOCH,etime_epoch,95

Flow end time expressed as floating point seconds since the UNIX epoch, 1970-01-01 00:00:00. The precision reflects that of the record, and the TIME_UNITS setting has no effect on this field. See also ETIMEMS and ETIME. Since super_mediator-1.11.0.

DPI_FIELD_LIST DPI_IE_LIST

If present for TEXT exporters, super_mediator will only export DPI information elements contained in <DPI_IE_LIST>. The DPI_IE_LIST is a list of information element ids from the below list (see DPI_CONFIG), separated by a comma, and surrounded by square brackets, [ and ]. For example, the following line will direct super_mediator to only export HTTP user agent strings and get requests.

DPI_FIELD_LIST [111, 112]

PRINT_HEADER

If present for TEXT Exporters, the super_mediator will write a header for delimited flow data. If files rotate, it will write one header at the top of each flow data file. Ignored for custom field lists.

METADATA_EXPORT

If present, super_mediator will include information element and template metadata in ipfix output. This option is not available to TEXT exporters.

REMOVE_EMPTY_FILES

If present for TEXT or FILEHANDLER Exporters, the super_mediator will remove output files that have a file size of 0. This is enabled by default.

NO_INDEX

If present for TEXT Exporters, the super_mediator will not write separate lines with flow information. It will include the flow information in the following form at the beginning of each DPI data line. The start time will be in human-readable format. The flow key hash will not be printed. This will make the DPI files substantially larger in size. See the above sample configuration and example (under MULTI_FILES).

start-time | srcip | dstip | protocol | srcport | \
dstport | vlan | obid

TIMESTAMP_FILES

By default, the super_mediator includes the timestamp (flow end time) of the first flow in the filename of the TEXT file, except if MULTI_FILES is present. If, TIMESTAMP_FILES is present, super_mediator will include the timestamp in the DPI files after the file extension (e.g. http.txt20120606123430). To search for a particular flow, use the flow's end time to determine which file contains the flow.

NO_FLOW_STATS

If present for TEXT Exporters, the super_mediator will not write yaf flow-stats. Only valid for TEXT exporters. Ignored for any other exporter type. See yaf(1) Flow Statistics Template for more information.

MYSQL_USER USER_NAME

If present for TEXT Exporters, the super_mediator will import the CSV file(s) to a MySQL database. It will use USER_NAME when connecting to the server. MYSQL_DATABASE and MYSQL_PASSWORD must also be set for super_mediator to complete uploads.

MYSQL_PASSWORD PASSWORD

If present for TEXT Exporters, the super_mediator will import the CSV file(s) to a MySQL database. It will use PASSWORD when connecting to the server. MYSQL_USER and MYSQL_DATABASE must also be set for super_mediator to complete uploads.

MYSQL_DATABASE DATABASE_NAME

If present for TEXT exporters, the super_mediator will import the CSV file(s) to the MySQL database DATABASE_NAME. It will use MYSQL_USER and MYSQL_PASSWORD when connecting to the server. All three values must be set or super_mediator will not try to import the files.

MYSQL_HOST HOSTNAME

If present for TEXT Exporters, the super_mediator will import the CSV file(s) to the MySQL database on the server located at HOSTNAME. If not present, and MYSQL_DATABASE, MYSQL_USER, and MYSQL_PASSWORD are present, super_mediator will try to import the file to the MySQL server running locally.

MYSQL_TABLE TABLE_NAME

If present for TEXT Exporters, the super_mediator will import the CSV file(s) to the table TABLE_NAME. This is only used if FLOW_ONLY or DNS_DEDUP_ONLY is also present. The default table name for FLOW_ONLY data is flow. The default table name for the dns-dedup data is dns. In order to set the table names for the DPI files, use the DPI_CONFIG block. Use the super_table_creator to create the database schemas for the DNS deduplication and flow tables.

Example of DPI Import:

EXPORTER TEXT
    PATH "/data/dpi"
    DPI_ONLY
    MULTI_FILES
    ROTATE 600
    LOCK
    TIMESTAMP_FILES
    NO_INDEX
    MYSQL_DATABASE super_db
    MYSQL_USER root
    MYSQL_PASSWORD password
EXPORTER END

Example of DNS_DEDUP Import:

EXPORTER TEXT
    PATH "/data/dns/dns_dedup"
    DNS_DEDUP_ONLY
    ROTATE 600
    LOCK
    MYSQL_DATABASE dns_dedup_db
    MYSQL_USER root
    MYSQL_PASSWORD password
    MYSQL_TABLE dns
EXPORTER END

Example of User-defined table names:

EXPORTER TEXT
    PATH "/data/dpi"
    DPI_ONLY
    MULTI_FILES
    ROTATE 600
    LOCK
    MYSQL_DATABASE my_super_db
    MYSQL_USER bob
    MYSQL_PASSWORD password
EXPORTER END

DPI_CONFIG
    TABLE my_http_table [111, 112, 113, 114, 115]
    TABLE my_dns_table [1, 2, 6, 28]
    TABLE my_dhcp_table [242, 243]
DPI_CONFIG END

REMOVE_UPLOADED

If present and the MySQL import was successful, super_mediator will remove the file it uploaded to the database. This only removes files that were successfully uploaded. super_mediator does not consider MySQL warnings as unsuccessful. Please be certain that the database is set up correctly, and the data is imported successfully before using this option.

AND_FILTER

If present, AND all filters in the EXPORTER block. All filters must pass for super_mediator to export the record.

ESCAPE_CHARS

If present, super_mediator will escape any control characters by replacing them with a four character sequence of a reverse solidus (backslash, "\") and the three digit octal code for the control character. Backslash characters that are present in the string will also be escaped by using a double backslash. super_mediator will also escape the delimiter character that is used for the EXPORTER, "|" by default. This option is only available for TEXT EXPORTERS.

DEDUP_PER_FLOW

If present, super_mediator will deduplicate DPI fields within a flow. Often, yaf will export multiple values for an information element that are the same. With this option, super_mediator will only export unique values for an information element along with a count of how many times the value was present with the flow. This only affects certain protocols such as HTTP, FTP, IMAP, RTSP, SIP, SMTP, SSH, IRC, POP3, MODBUS, ENIP, SLP. The hit count for each value will be written in the column before the value. For JSON exporters, DPI values will be de-duplicated, but super_mediator will not export a hit count.

EXPORTER TEXT
    PATH "/data/flow.txt"
    DEDUP_PER_FLOW
EXPORTER END

Example DPI Output with above config (third column is hit count):

http|115|1|http://en.wikipedia.org/wiki/Http
http|114|2|HTTP/1.0
http|117|1|en.wikipedia.org

JSON

If present, super_mediator will write flow and DPI data in JSON format. super_mediator will use the standard IPFIX information element as the key for each value present in the flow. JSON cannot be combined with MULTI_FILES. If no other keywords are present, JSON exporters will export the same information elements as the standard TEXT exporter including every DPI information element present in the flow. JSON can also be combined with FIELDS to choose which fields to export in JSON format. The standard root element for a flow is "flows". The root elements will be different for DNS_DEDUP, SSL_DEDUP, SSL CERTIFICATE, DEDUP, or DNS_RR records.

DEDUP_ONLY

If present, super_mediator will only write de-duplicated DPI data. See the DEDUP_CONFIG block for more information on configuring data de-duplication.

SSL_DEDUP_ONLY

If present, super_mediator will perform SSL certificate de-duplication on SSL certificates received from yaf(1). See the SSL_CONFIG block for more information on SSL certificate de-duplication. If this keyword is present for an exporter, super_mediator will only export SSL Dedup records and SSL certificate records. Not valid if DEDUP_ONLY, DNS_DEDUP_ONLY, DNS_RR_ONLY, or FLOW_ONLY is also present.

SSL_DEDUP

If present, <super_mediator> will perform SSL certificate de-duplication on SSL certificates received from yaf(1). It will also write previously de-duplicated SSL records from super_mediator to the EXPORTER. See the SSL_CONFIG block for more information on SSL certificate de-duplication.

SSL_CERT_HASH_SHA1

If present, super_mediator will perform a SHA1 hash against the entire X.509 certificate, if available. This option is only available if super_mediator was built with OpenSSL support. SHA1 hashing can also be enabled by listing 298 in the OTHER FIELD_LIST in the SSL_CONFIG block. If using the MULTI_FILES option, add 298 to the TABLE_LIST in the DPI_CONFIG block.

SSL_CERT_HASH_MD5

If present, super_mediator will perform an MD5 hash against the entire X.509 certificate, if available. This option is only available if super_mediator was built with OpenSSL support. MD5 hashing can also be enabled by listing 299 in the OTHER FIELD_LIST in the SSL_CONFIG block. If using the MULTI_FILES option, add 299 to the TABLE_LIST in the DPI_CONFIG block

MATCHES_IPSET SILK_IPSET_FILE_PATH

MATCHES_IPSET IPSET_OUT_NAME SILK_IPSET_FILE_PATH

If present and super_mediator is built with SiLK or SiLK IPSet support, super_mediator checks whether the IP addresses in each flow record (sourceIPv4Address and destinationIPv4Address or, if both of those are zero, sourceIPv6Address and destinationIPv6Address) are present in the SiLK IPSet file loaded from SILK_IPSET_FILE_PATH and adds an entry to the record's subTemplateMultiList that reflects the result of the check.

If the record contains IPv6 addresses but SiLK was not compiled with IPv6 support, the sub-record is not added to the subTemplateMultiList.

The record added to the multi-list uses this template:

smIPSetName CERT (PEN 6871) IE 933, variable length, string: A name for the IPSet. If IPSET_OUT_NAME is specified, it is written to this field. Otherwise the final component of SILK_IPSET_FILE_PATH is written here.
smIPSetMatchesSource CERT (PEN 6871) IE 931, 1 octet, unsigned: Whether the source address was present in SILK_IPSET_FILE_PATH: 1 means it was, 0 means it was not.
smIPSetMatchesDestination CERT (PEN 6871) IE 932, 1 octet, unsigned: Whether the destinationIPv4Address address was present in SILK_IPSET_FILE_PATH: 1 means it was, 0 means it was not.

When exporting as JSON, the output depends the number of IPSet records present in the record. When only one is present, the values are shown at the top-level:

{
  "flows": {
    "flowStartMilliseconds": "2012-06-30 14:07:30.085",
    "flowEndMilliseconds": "2012-06-30 14:07:31.226",
    ...
    "smIPSetName": "this is my name",
    "smIPSetMatchesSource": 0,
    "smIPSetMatchesDestination": 0
  }
}

When multiple IPSet records are present, the values are presented in an array:

{
  "flows": {
    "flowStartMilliseconds": "2012-06-30 14:07:30.085",
    "flowEndMilliseconds": "2012-06-30 14:07:31.226",
    ...
    "smIPSetMatchingList": [
      {
        "smIPSetName": "this is my name",
        "smIPSetMatchesSource": 0,
        "smIPSetMatchesDestination": 0
      },
      {
        "smIPSetName": "all.set",
        "smIPSetMatchesSource": 1,
        "smIPSetMatchesDestination": 1
      }
    ]
  }
}

The MATCHES_IPSET command was added in super_mediator 1.9.0 and it is only allowed when super_mediator has been compiled with SiLK IPSet support.

PREFIX_MAP SILK_PMAP_FILE_PATH

PREFIX_MAP PMAP_OUT_NAME SILK_PMAP_FILE_PATH

If present and super_mediator is built with SiLK Prefix Map support, super_mediator adds an entry to the record's subTemplateMultiList that contains the labels found within the SiLK Prefix Map (PMAP) file for the record's source and destination fields.

If the Prefix Map at SILK_PMAP_FILE_PATH maps IP addresses to labels, super_mediator gets the labels for the record's sourceIPv4Address and destinationIPv4Address or, if both of those are zero, the record's sourceIPv6Address and destinationIPv6Address. If the record contains IPv6 addresses but SiLK was not compiled with IPv6 support, the sub-record is not added to the subTemplateMultiList.

If the Prefix Map maps protocol-port pairs to labels, super_mediator gets the labels for the (protocolIdentifier, sourceTransportPort) and (protocolIdentifier, destinationTransportPort) pairs.

The record added to the multi-list uses this template:

smPrefixMapName CERT (PEN 6871) IE 937, variable length, string: A name for the Prefix Map. If PMAP_OUT_NAME is specified, it is written to this field. Otherwise the final component of SILK_PMAP_FILE_PATH is written here.
smPrefixMapLabelSource CERT (PEN 6871) IE 934, 1 octet, unsigned: The label that was found for the source address or the (protocolIdentifier, sourceTransportPort) pair.
smPrefixMapLabelDestination CERT (PEN 6871) IE 935, 1 octet, unsigned: The label that was found for the destination address or the (protocolIdentifier, destinationTransportPort) pair.
smPrefixMapTypeId CERT (PEN 6871) IE 936, 1 octet, unsigned: The type of Prefix Map that assigned the labels, where 0 indicates a map from IPv4 addresses to labels, 1 from protocol/port pairs, and 2 from IPv6 addresses.

When exporting as JSON from super_mediator, the values are written at top-level if only one Prefix Map label exists:

{
  "flows": {
    "flowStartMilliseconds": "2012-06-30 14:07:30.085",
    "flowEndMilliseconds": "2012-06-30 14:07:31.226",
    ...
    "smPrefixMapName": "ip-map.pmap",
    "smPrefixMapLabelSource": "internal",
    "smPrefixMapLabelDestination": "external",
    "smPrefixMapTypeId": 0
  }
}

When multiple are present, the values are presented in an array:

{
  "flows": {
    "flowStartMilliseconds": "2012-06-30 14:07:30.085",
    "flowEndMilliseconds": "2012-06-30 14:07:31.226",
    ...
    "smPrefixMapLabelList": [
      {
        "smPrefixMapName": "ip-map.pmap",
        "smPrefixMapLabelSource": "internal",
        "smPrefixMapLabelDestination": "external",
        "smPrefixMapTypeId": 0
      },
      {
        "smPrefixMapName": "proto-port-map.pmap",
        "smPrefixMapLabelSource": "TCP",
        "smPrefixMapLabelDestination": "TCP/HTTPS",
        "smPrefixMapTypeId": 1
      }
    ]
  }
}

The PREFIX_MAP command was added in super_mediator 1.9.0 and it is only allowed when super_mediator has been compiled with SiLK Prefix Map support.

Exporter Filters

Each exporter can contain one or more OR filters to define what data should be exported to the exporter. The syntax for the filters is the same as defined above in the Filter Block. The filters defined in the exporter block are only for the exporter that contains them. The filters are by default OR filters. Use AND_FILTER to make the filters AND filters. See Examples.

Group Block

The information from the group block is used by the super_mediator to determine what flow data to send each Spread Group defined in the exporter block. The group blocks are optional. If they don't exist, every Spread Group named in the exporter block will receive every and all flows. If a filter is defined in the exporter block, every Spread Group defined in the exporter block will receive the flows that pass the exporter filter. In the case that each Spread Group should receive some subset of the flows, a filter can be defined for the group in this group block.

GROUP GROUP_NAME: The GROUP command begins a new group block and it continues to the GROUP END command. The argument to the GROUP command is the name of the group for which the filter is being defined. The GROUP_NAME must already exist in the exporter block. Group names should not contain spaces or special characters.
GROUP END: The GROUP END command ends the definition of a group. Following a GROUP END command, top-level commands are again accepted. Each Group Block should contain at least 1 filter.
Group Filters: The Group block contains a series of Filters that should be in the same form as described above in the Filter block. See Examples.

DNS De-duplication Block

The first command below is used at the top-level to begin a DNS de-duplication block, and the remaining commands are accepted within the dns de-duplication context. The information from the dns de-duplication block is used by the super_mediator to determine how records will be flushed from the cache. See super_mediator(1) for a complete description of the IPFIX information elements exported for DNS_DEDUP records.

DNS_DEDUP

The DNS_DEDUP command begins a new DNS de-duplication block and it continues to the DNS_DEDUP END command. There are no arguments to the DNS_DEDUP command.

DNS_DEDUP END

The DNS_DEDUP END command ends the definition of a DNS de-duplication block. Following a DNS_DEDUP END command, top-level commands are again accepted.

MAX_HIT_COUNT COUNT

If present, the super_mediator will flush a DNS record from the cache when the internal hit count reaches count. If the keyword LAST_SEEN is also present, the super_mediator will write the record. Otherwise, the record will be silently flushed from the cache. The default COUNT is 500. The max COUNT is 65535.

FLUSH_TIME FLUSH_SECONDS

If present, the super_mediator will flush a DNS record from the cache when a new record has not been seen for over FLUSH_SECONDS. If the keyword LAST_SEEN is also present in the DNS De-duplication block, the super_mediator will write the record. Otherwise, the record will be silently flushed from the cache. The default FLUSH_SECONDS is 300 seconds, or 5 minutes. The max FLUSH_SECONDS is 65535, or approx. 18 hours.

LAST_SEEN

If present, the super_mediator will write records when they are flushed, rather than when they are first seen. The records will also contain a last_seen time and a hit count. If LAST_SEEN is present, the output will be in the following format:

first_seen | last_seen | rrtype | rrname | hitcount | rrval

first_seen and last_seen are timestamps in the form 2012-01-23 04:45:13.897.

BASE64_ENCODE

If present, the super_mediator will BASE 64 Encode the domain names in the DNS records. It will only BASE 64 Encode names for TEXT Exporters.

RECORDS DNS_RESOURCE_RECORD_TYPE_LIST

If present, super_mediator will dedup on resource record types contained contained in DNS_RESOURCE_RECORD_TYPE_LIST. The DNS_RESOURCE_RECORD_TYPE_LIST is a list of resource record types, separated by a comma, and surrounded by square brackets, [ and ]. For example, the following line will direct super_mediator to only dedup on A Records and NS Records. Resource record types available are 0 for NXDomains and 1, 2, 5, 6, 12, 15, 16, 28, 33.

RECORDS [1, 2]

MAP(MAPNAME)

By default, super_mediator hashes the DNS resource record name to store and aggregate DNS resource records. In addition to the name, super_mediator can aggregate the data per VLAN ID (or group of VLAN IDs) or per Observation Domain ID (or group of Observation domains) with a VLAN_MAP or OBID_MAP. To use a VLAN_MAP or OBID_MAP in addition to the resource record name key use MAP(MAPNAME). The VLAN_MAP or OBID_MAP with MAPNAME must be defined before the DNS_DEDUP block in the configuration file. If a MAP is used, then a TEXT record will include an extra column at the end of the line with the MAPNAME and an IPFIX record will contain an additional information element observationDomainName with the name of the MAP.

VLAN_MAP x
   S1 [1, 2, 3]
   S2 [6]
   DISCARD_OTHER
VLAN_MAP END

DNS_DEDUP test
    MAP(x)
DNS_DEDUP END

ADD_EXPORTER_NAME

If present, super_mediator will add an extra column onto the TEXT record with the name of the EXPORTER associated with DNS_DEDUP block. If the EXPORTER is an IPFIX EXPORTER, the observationDomainName IE field will be populated with the exporter name. This option is ignored if a MAP is also defined for this EXPORTER.

DPI Configuration Block

The first command below is used at the top-level to begin a DPI configuration block, and the remaining commands are accepted within the DPI config context. The information from the dpi config block is used by the super_mediator to determine how to write the DPI data in the text files, or if MULTI_FILES is present in the EXPORTER block, it determines the filenames of the CSV files that contain the DPI data. If a DPI_CONFIG block is present, it will be used for all exporters that are exporting DPI Information (it does not effect DNS deduplication exporters). For example, if an exporter is using a custom field list FIELDS, and DPI is an item in that list, it will only write flows that contain a DPI element that is listed in the DPI_CONFIG block. However, if the DPI_FIELD_LIST keyword is also present in the exporter, the DPI_FIELD_LIST takes precedence over the DPI_CONFIG items.

DPI_CONFIG INDEX_NAME

The DPI_CONFIG command begins a new DPI config block and it continues to the DPI_CONFIG END command. The INDEX_NAME is an optional argument to the DPI_CONFIG command. If INDEX_NAME is present, the table name or filename for the flow index information will have the name INDEX_NAME. If not present, the default flow will be used.

DPI_CONFIG_END

The DPI_CONFIG_END command ends the definition of DPI_CONFIG block. Following the DPI_CONFIG_END command, top-level commands are accepted. Only one DPI_CONFIG block is permitted in a configuration file.

TABLE TABLE_NAME TABLE_LIST

If TABLE is present, label the lines that contain information element ids in the TABLE_LIST with the word TABLE_NAME. TABLE_NAME should not contain spaces. The TABLE_LIST is a list of information element ids from the below list, separated by a comma, and surrounded by square brackets, [ and ]. The following example will label the lines that contain user agent strings (id 111), with "http_ua":

TABLE http_ua [111]

The following list contains the default information element ids and their respective table name label (see yafdpi(1) for a description of each information element):

information element name    | id     | table label
----------------------------+--------+--------------
osName                      | 36     | p0f
osVersion                   | 37     | p0f
osFingerPrint               | 107    | p0f
httpServerString            | 110    | http
httpUserAgent               | 111    | http
                            |        |
httpGet                     | 112    | http
httpConnection              | 113    | http
httpVersion                 | 114    | http
httpReferer                 | 115    | http
httpLocation                | 116    | http
                            |        |
httpHost                    | 117    | http
httpContentLength           | 118    | http
httpAge                     | 119    | http
httpAccept                  | 120    | http
httpAcceptLanguage          | 121    | http
                            |        |
httpContentType             | 122    | http
httpResponse                | 123    | http
httpCookie                  | 220    | http
httpSetCookie               | 221    | http
pop3TextMessage             | 124    | pop3
                            |        |
ircTextMessage              | 125    | irc
tftpFilename                | 126    | tftp
tftpMode                    | 127    | tftp
slpVersion                  | 128    | slp
slpMessageType              | 129    | slp
                            |        |
slpString                   | 130    | slp
ftpReturn                   | 131    | ftp
ftpUser                     | 132    | ftp
ftpPass                     | 133    | ftp
ftpType                     | 134    | ftp
                            |        |
ftpRespCode                 | 135    | ftp
imapCapability              | 136    | imap
imapLogin                   | 137    | imap
imapStartTLS                | 138    | imap
imapAuthenticate            | 139    | imap
                            |        |
imapCommand                 | 140    | imap
imapExists                  | 141    | imap
imapRecent                  | 142    | imap
rtspURL                     | 143    | rtsp
rtspVersion                 | 144    | rtsp
                            |        |
rtspReturnCode              | 145    | rtsp
rtspContentLength           | 146    | rtsp
rtspCommand                 | 147    | rtsp
rtspContentType             | 148    | rtsp
rtspTransport               | 149    | rtsp
                            |        |
rtspCSeq                    | 150    | rtsp
rtspLocation                | 151    | rtsp
rtspPacketsReceived         | 152    | rtsp
rtspUserAgent               | 153    | rtsp
rtspJitter                  | 154    | rtsp
                            |        |
sipInvite                   | 155    | sip
sipCommand                  | 156    | sip
sipVia                      | 157    | sip
sipMaxForwards              | 158    | sip
sipAddress                  | 159    | sip
                            |        |
sipContentLength            | 160    | sip
sipUserAgent                | 161    | sip
smtpHello                   | 162    | smtp
smtpFrom                    | 163    | smtp
smtpTo                      | 164    | smtp
                            |        |
smtpContentType             | 165    | smtp
smtpSubject                 | 166    | smtp
smtpFilename                | 167    | smtp
smtpContentDisposition      | 168    | smtp
smtpResponse                | 169    | smtp
                            |        |
smtpEnhanced                | 170    | smtp
smtpSize                    | 222    | smtp
sshVersion                  | 171    | ssh
nntpResponse                | 172    | nntp
nntpCommand                 | 173    | nntp
                            |        |
sslCipher                   | 185    | tls
sslClientVersion            | 186    | tls
sslServerCipher             | 187    | tls
sslCompressionMethod        | 188    | tls
sslCertVersion              | 189    | tls
                            |        |
sslCertSignature            | 190    | tls
sslCertIssuerCountryName*   | 191    | tls
sslCertIssuerOrgName*       | 192    | tls
sslCertIssuerOrgUnitName*   | 193    | tls
sslCertIssuerZipCode*       | 194    | tls
                            |        |
sslCertIssuerState*         | 195    | tls
sslCertIssuerCommonName*    | 196    | tls
sslCertIssuerLocalityName   | 197    | tls
sslCertIssuerStreetAddres*  | 198    | tls
sslCertSubCountryName*      | 200    | tls
                            |        |
sslCertSubOrgName*          | 201    | tls
sslCertSubOrgUnitName*      | 202    | tls
sslCertSubZipCode*          | 203    | tls
sslCertSubState*            | 204    | tls
sslCertSubCommonName*       | 205    | tls
                            |        |
sslCertSubLocalityName*     | 206    | tls
sslCertSubStreetAddress*    | 207    | tls
sslCertSerialNumber         | 244    | tls (in hexadecimal)
sslCertValidityNotBefore    | 247    | tls
sslCertValidityNotAfter     | 248    | tls
                            |        |
sslPublicKeyAlgorithm       | 249    | tls
sslPublicKeyLength          | 250    | tls
sslRecordVersion            | 288    | tls
sslServerName               | 294    | tls
sslCertificateHash          | 295    | tls (in hexadecimal)
                            |        |
sslCertificate              | 296    | tls (Base64 encoded)
sslCertificateMD5           | 299    | tls (in hexadecimal)
sslCertificateSHA1          | 298    | tls (in hexadecimal)
mysqlUsername               | 223    | mysql
mysqlCommandText            | 225    | mysql
                            |        |
dhcpFingerPrint             | 242    | dhcp
dhcpVendorCode              | 243    | dhcp
httpAuthorization           | 252    | http
httpVia                     | 253    | http
httpX-Forwarded-For         | 254    | http
                            |        |
httpExpires                 | 255    | http
httpRefresh                 | 256    | http
httpIMEI                    | 257    | http
httpIMSI                    | 258    | http
httpMSISDN                  | 259    | http
                            |        |
httpSubscriber              | 260    | http
httpAcceptCharset           | 261    | http
httpAcceptEncoding          | 262    | http
httpAllow                   | 263    | http
httpDate                    | 264    | http
                            |        |
httpExpect                  | 265    | http
httpFrom                    | 266    | http
httpProxyAuthentication     | 267    | http
httpUpgrade                 | 268    | http
httpWarning                 | 269    | http
                            |        |
httpDNT                     | 270    | http
httpX-Forwarded-Proto       | 271    | http
httpX-Forwarded-Host        | 272    | http
httpX-Forwarded-Server      | 273    | http
httpX-DeviceID              | 274    | http
                            |        |
httpX-Profile               | 275    | http
httpLastModified            | 276    | http
httpContentEncoding         | 277    | http
httpContentLanguage         | 278    | http
httpContentLocation         | 279    | http
                            |        |
httpX-UA-Compatible         | 280    | http
dnp3ObjectData              | 284    | dnp (in hexadecimal)
modbusData                  | 285    | modbus (in hexadecimal)
ethernetIPData              | 286    | enip (in hexadecimal)
rtpPayloadData              | 287    | rtp

Not all fields are turned on by default in YAF. See the yafDPIRules.conf to turn on/off certain fields.

*These items were removed as of YAF 2.3.0. SSL Certificate information elements were replaced by the X.509 object identifier value. Use id 443 in TABLE_LIST or DPI_FIELD_LIST to include all of the following elements. To specify individual fields, use the SSL_CONFIG block Below is a list of common objects in an X.509 RelativeDistinguishedName Sequence (not all possibilities are listed):

object identifier name    | id     | table label
--------------------------+--------+--------------
common name               | 3      | tls
countryName               | 6      | tls
localityName              | 7      | tls
stateOrProvinceName       | 8      | tls
streetAddress             | 9      | tls
                          |        |
organization              | 10     | tls
organizational unit       | 11     | tls
title                     | 12     | tls
postalCode                | 17     | tls
name                      | 41     | tls

DNS has a different format. TABLE_LIST should contain the DNS Resource Record Types in the below list:

Q/R Record Type             | id     | default table name
----------------------------+--------+--------------
A Record                    | 1      | dns
NS Record                   | 2      | dns
CNAME Record                | 5      | dns
SOA Record                  | 6      | dns
MX Record                   | 12     | dns
                            |        |
PTR Record                  | 15     | dns
TXT Record                  | 16     | dns
AAAA Record                 | 28     | dns
SRV Record                  | 33     | dns
DS Record*                  | 43     | dns
                            |        |
RRSIG Record**              | 46     | dns
NSEC Record**               | 47     | dns
DNSKEY Record*              | 48     | dns
NSEC3 Record*               | 50     | dns
NSEC3PARAM Record*          | 51     | dns
                            |        |
All Others***               | 53     | dns

*Records that do not contain data for the rdata field.

**RRSIG Type records contain the signer name in the rdata field. NSEC Type records contains the next domain name in the rdata field.

***Any nonstandard query response type, such as 251 - Incremental Transfers, can be filtered by using 53.

SSL_CONFIG Block

The first command below is used at the top-level to begin a SSL_CONFIG block, and the remaining commands are accepted within the SSL DPI context. The information from the SSL_CONFIG block is used by super_mediator to determine which SSL/TLS X.509 certificates to export (for TEXT exporters). This block can be used in conjunction with the DPI_FIELD_LIST. If this block is present, all other SSL/TLS fields will be disabled and only elements present in one of the ISSUER, SUBJECT, OTHER, and EXTENSIONS lists will be exported. See below for an example of use. To export SSL de-deduplication information in IPFIX, use the SSL_DEDUP_ONLY in the EXPORTER block. The SSL_CONFIG block only applies to TEXT Exporters. This block can also be used to configure incoming de-duplicated SSL records from another super_mediator.

SSL_CONFIG EXPORTER_NAME

The SSL_CONFIG command begins a new SSL_CONFIG block and it continues to the SSL_CONFIG END command. The only required argument to the SSL_CONFIG command is the EXPORTER_NAME for which this SSL configuration applies. This should match the name of one and only one TEXT EXPORTER from the configuration file. The SSL_CONFIG block should be placed after the EXPORTER block for which it refers to.

SSL_CONFIG END

The SSL_CONFIG END command ends the definition of a SSL_CONFIG block. Following a SSL_CONFIG END command, top-level commands are again accepted.

ISSUER FIELD_LIST

If present, super_mediator will only write certain X.509 object identifier values specified in FIELD_LIST from the ISSUER x.509 RelativeDistinguishedName Sequence. The FIELD_LIST is a list of X.509 RelativeDistinguishedName Sequence object IDs, separated by a comma, and surrounded by square brackets, [ and ]. A list of common object IDs is listed above under the DPI_CONFIG block. The default behavior is to print all issuer fields, however if any list (ISSUER, SUBJECT, OTHER, EXTENSIONS) is present in the SSL_CONFIG block, super_mediator will only print the elements specifically denoted in the FIELD_LIST. To print all elements use [*] to denote all elements.

SUBJECT FIELD_LIST

If present, super_mediator will only write certain X.509 object identifier values specified in FIELD_LIST from the SUBJECT x.509 RelativeDistinguishedName Sequence. The FIELD_LIST is a list of X.509 RelativeDistinguishedName Sequence object IDs, separated by a comma, and surrounded by square brackets, [ and ]. A list of common object IDs is listed above under the DPI_CONFIG block. The default behavior is to print all subject fields, however if any list (ISSUER, SUBJECT, OTHER, EXTENSIONS) is present in the SSL_CONFIG block, super_mediator will only print the elements specifically denoted in the FIELD_LIST. To print all elements use [*] to denote all elements.

OTHER FIELD_LIST

If present, super_mediator will only write certain SSL/TLS information elements specified in FIELD_LIST. The FIELD_LIST is a list of information element IDs, separated by a comma, and surrounded by square brackets, [ and ]. A list of SSL/TLS Information Element IDs is listed above under the DPI_CONFIG block (valid elements have a label of tls). The default behavior is to print all "other" fields, however if any list (ISSUER, SUBJECT, OTHER, EXTENSIONS) is present in the SSL_CONFIG block, super_mediator will only print the elements specifically denoted in the FIELD_LIST. To print all elements use [*] to denote all elements. If the full X.509 certificate is available and if openSSL is available, super_mediator has the ability to perform a MD5 or SHA1 hash of the certificate. To print these values, list 299 (MD5) or 298 (SHA1) in the FIELD_LIST. super_mediator can also Base64 encode the full X.509 certificate, if available. To export the Base64 encoded certificate, add 296 to the OTHER FIELD_LIST.

EXTENSIONS FIELD_LIST

If present, super_mediator will write particular extension types from the SSL/TLS certificate specified in FIELD_LIST. The FIELD_LIST is a list of X.509 RelativeDistinguishedName Sequence object IDs, separated by a comma, and surrounded by square brackets, [ and ]. A list of acceptable object IDs is listed below. super_mediator will only export extension types if they are specifically listed within the SSL_CONFIG block. Some extension types contain a sequence of values. Each element in the sequence will be exported on a separate line. The default behavior is to print all extension fields, however if any list (ISSUER, SUBJECT, OTHER, EXTENSIONS) is present in the SSL_CONFIG block, super_mediator will only print the elements specifically denoted in the FIELD_LIST. To print all elements use [*] to denote all elements. The accepted extension types are:

extension type                 | id
-------------------------------+-----
subjectKeyIdentifier           | 14*
keyUsage                       | 15*
privateKeyUsagePeriod          | 16*
subjectAltName                 | 17
issuerAltName                  | 18
                               |
certificateIssuer              | 29
cRLDistributionPoints          | 31
certificatePolicies            | 32*

* denotes when super_mediator writes the value in hex.

For rewritten SSL certificates, the following EXTENSION values are supported:

extension type                 | id
-------------------------------+-----
sslCertExtSubjectKeyIdent      | 316
sslCertExtKeyUsage             | 317
sslCertExtPrivKeyUsagePeriod   | 318
sslCertExtSubjectAltName       | 319
sslCertExtIssuerAltName        | 320
                               |
sslCertExtCertIssuer           | 321
sslCertExtCrlDistribution      | 322
sslCertExtCertPolicies         | 323
sslCertExtAuthorityKeyIdent    | 324
sslCertExtExtendedKeyUsage     | 325

SSL_DEDUP

If present, super_mediator will perform de-duplication of SSL certificates received from yaf(1). It will take SSL Certificate information collected by yaf and output the certificate information once and cache the serial number and issuer name for each certificate. Certificates received matching the serial number and issuer tuple after initial export will simply increment the internal counters. Certificates in the cache are flushed based on the default values or the values configured in the SSL_CONFIG block. A SSL de-duplication record will be exported either when the internal hit count reaches 5000 or when the record has been in the cache for 5 minutes. These parameters are configurable by setting MAX_HIT_COUNT or FLUSH_TIME within the SSL_CONFIG block. The SSL de-duplication record has the following format (in IPFIX):

flowStartMilliseconds IE 152, 8 octets, unsigned: The first time this certificate was seen (start time of the flow that contained this certificate).
flowEndMilliseconds IE 153, 8 octets, unsigned: The last time this certificate was seen (start time of the flow that contained this certificate).
observedDataTotalCount CERT (PEN 6871) IE 929, 8 octets, unsigned: The number of times this certificate was seen in the time period.
sslCertSerialNumber CERT (PEN 6871) IE 244, variable length: The serial number of the X.509 Certificate.
sslCertIssuerCommonName CERT (PEN 6871) IE 196, variable length: The common name of the Issuer (Certificate Authority) in the X.509 Certificate.

The CSV format of the SSL dedup record is as follows:

fseen | lseen | serial | hitcount | issuer

The SSL certificates exported will follow the same IPFIX format as described in yafdpi(1). The text format for SSL certificates is as follows:

serial | issuer | stime | id | ISE | cert_no | data

Serial is the serial number in hexadecimal. Issuer is the common name (id-at 3) of the X.509 Issuer (Certificate Authority). If no common name is present, the organizational unit name is used (id-at 11). stime is the time in milliseconds that the certificate was first seen. id is the object/member ID for the X.509 RelativeDistinguishedName Sequence. A list of common objects can be found above. If ISSUER, SUBJECT, OTHER, or EXTENSION field lists are present within the SSL_CONFIG block, super_mediator will only print objects that are present within the field lists. ISE denotes if the data came from an Issuer Field(I), Subject Field(S), or Extension Field (E). For fields that are not associated with the issuer, subject, or extension but describe other characteristics of the certificate, an I will be used (for example, not-before or not-after timestamps). cert seq no signifies which certificate the data came from in the certificate chain. Usually, this field will contain a 0, 1, or 2.

MAX_HIT_COUNT COUNT

If present, the super_mediator will flush a SSL record from the cache when the internal hit count reaches count. The default COUNT is 5000.

FLUSH_TIME FLUSH_SECONDS

If present, the super_mediator will flush a SSL record from the cache when a new record has not been seen for over FLUSH_SECONDS. super_mediator will write the record when it is flushed. The default FLUSH_SECONDS is 300 seconds, or 5 minutes.

CERT_FILE FILENAME

If present, super_mediator will write SSL certificates in the above format to the file path provided by FILENAME. This option is only valid for TEXT Exporters (not JSON). The file will be rotated or locked if those options are configured in the EXPORTER block associated with the SSL_CONFIG block.

MAP(MAPNAME)

By default, super_mediator hashes the SSL Certificate Serial Number and Issuer Name to store and aggregate SSL Certificates. In addition to the serial number, super_mediator can aggregate the data per VLAN ID (or group of VLAN IDs) or per Observation Domain ID (or group of Observation domains) with a VLAN_MAP or OBID_MAP. To use a VLAN_MAP or OBID_MAP in addition to the serial number/issuer name key use MAP(MAPNAME). The VLAN_MAP or OBID_MAP with MAPNAME must be defined before the SSL_CONFIG block in the configuration file. If a MAP is used, then a TEXT record will include an extra column at the end of the line with the MAPNAME and an IPFIX record will contain an additional information element observationDomainName with the name of the MAP.

OBID_MAP x
   S1 [1, 2, 3]
   S2 [6]
   DISCARD_OTHER
OBID_MAP END

SSL_CONFIG test
    MAP(x)
SSL_CONFIG END

ADD_EXPORTER_NAME

If present, super_mediator will add an extra column onto the TEXT record with the name of the EXPORTER associated with SSL_CONFIG block. If the EXPORTER is an IPFIX EXPORTER, the observationDomainName IE field will be populated with the exporter name. This option is ignored if a MAP is also defined for this EXPORTER.

DEDUP_CONFIG Block

The first command below is used at the top-level to begin a DEDUP_CONFIG block, and the remaining commands are accepted within the DEDUP_CONFIG context. The information from the DEDUP_CONFIG block is used by super_mediator to determine which fields to perform de-deduplication on. De-duplication can be used with any type of EXPORTER. This block can also be used to configure incoming de-duplicated records from another super_mediator. The pipe-delimited format for any TEXT file (except SSL) is as follows (line wrapped for readability):

first_seen | last_seen | sourceIP or dstIP | flow stime | \
flowkeyhash | hitcount | value | map or exporter name

where first_seen is the time of the first record, last_seen is the time of the last record seen with this tuple. SourceIP or dstIP is the IP address for the flow which contained this value. By default, super_mediator stores values with the source IP address, but this behavior can be changed with the PREFIX command. flow stime is the start time of the last flow to have this data present within the payload of the flow. flowkeyhash is the hash of the last flow's 5-tuple to have this data present within the payload of the flow. The flow stime and the flowkeyhash can provide a unique key for tracing the record back to flow data. The hitcount is the number of times the tuple was seen within first_seen and last_seen. The value is the value of the information element. The map or exporter name is optional and will be present if ADD_EXPORTER_NAME is present in the DEDUP_CONFIG block or a PREFIX line is configured with a MAP.

For SSL, the format is slightly different (line wrapped for readability):

first_seen | last_seen | sourceIP or dstIP | flow stime | \
flowkeyhash | hitcount | serial1 | issuer1 | serial2 | issuer2 \
| map or exporter name

where serial1 is the serial number (in hex) of the first certificate in the SSL certificate chain. issuer1 is the issuer's common name (id 3) of the first certificate in the SSL certificate chain. serial2 is the serial number (in hex) of the second certificate in the SSL certificate chain and issuer2 is the the issuer's common name (id 3) of the second certificate in the chain.

For IPFIX exporters, the following information elements will be exported:

monitoringIntervalStartMilliSeconds IE 359, 8 octets, unsigned: The first time this data was seen (start time of the flow that contained this data).
monitoringIntervalEndMilliSeconds IE 360, 8 octets, unsigned: The last time this data was seen (start time of the flow that contained this data).
flowStartMilliseconds IE 152, 8 octets, unsigned: The start time of the last flow that contained the data being aggregated. The flowStartMilliseconds, with the yafFlowKeyHash, can create a unique key that can be used to correlate with a SiLK flow or PCAP repository.
observedDataTotalCount CERT (PEN 6871) IE 929, 8 octets, unsigned: The number of times this data was seen in the time period.
sourceIPv6Address IE 27, 16 octets, unsigned, FULL: IPv6 address of flow source or biflow initiator. Present for IPv6 flows or IPv6-mapped IPv4 flows only.
sourceIPv4Address IE 8, 4 octets, unsigned, FULL: IPv4 address of flow source or biflow initiator. Present for IPv4 flows without IPv6-mapped addresses only.
yafFlowKeyHash CERT (PEN 6871) IE 106, 4 octets, unsigned: The hash of the 5-tuple (sourceIPAddress, destinationIPAddress, sourcePort, Destinationport, protocol) and vlanId.
observationDomainName IE 300, variable length: This field is populated if the DEDUP_CONFIG block was configured with ADD_EXPORTER_NAME or the PREFIX line contained a VLAN_MAP or OBID_MAP.
Information element depends on FIELD_LIST defined in DEDUP_CONFIG block.: This information element will depend on how super_mediator is configured and which information elements are present in the FIELD_LIST provided to PREFIX.
sslCertSerialNumber CERT (PEN 6871) IE 244, variable length SSL_ONLY: The serial number of the first X.509 Certificate in the chain (local). Only present for SSL de-duplicated data (FIELD_LIST = [244]).
sslCertIssuerCommonName CERT (PEN 6871) IE 196, variable length SSL_ONLY: The common name of the Issuer (Certificate Authority) in the X.509 Certificate (first certificate in the chain). Only present for SSL de-duplicated data.
sslCertSerialNumber CERT (PEN 6871) IE 244, variable length SSL_ONLY: The serial number of the second X.509 Certificate in the chain. Only present for SSL de-duplicated data.
sslCertIssuerCommonName CERT (PEN 6871) IE 196, variable length SSL_ONLY: The common name of the Issuer (Certificate Authority) in the X.509 Certificate. Only present for SSL de-duplicated data.

DEDUP_CONFIG EXPORTER_NAME

The DEDUP_CONFIG command begins a new DEDUP_CONFIG block and it continues to the DEDUP_CONFIG END command. The only required argument to the DEDUP_CONFIG command is the EXPORTER_NAME for which the deduplication configuration applies. This should match the name of one and only one TEXT EXPORTER from the configuration file. The DEDUP_CONFIG block should be placed after the EXPORTER block for which it refers to.

DEDUP_CONFIG END

The DEDUP_CONFIG END command ends the definition of a DEDUP_CONFIG block. Following a DEDUP_CONFIG END command, top-level commands are again accepted.

PREFIX FILE_PREFIX SIP|DIP|FLOWKEYHASH MAP(MAPNAME) FIELD_LIST

At least one PREFIX command must be present within a DEDUP_CONFIG block. The PREFIX command defines which information elements will be deduplicated and the filename prefix in which the records will be written to. If the EXPORTER for which this DEDUP_CONFIG applies is a TEXT exporter, it should have specified a PATH that is a file directory. FILE_PREFIX will be the prefix of the filename that deduplicated records will be written to. The timestamp will be appended to the FILE_PREFIX. If the EXPORTER is a JSON EXPORTER or an IPFIX File EXPORTER, all deduplicated information will be written to the same file. The PATH keyword for the associated EXPORTER should be a valid file path with file prefix. For JSON exporters, the deduplication information will be identified by the key FILE_PREFIX. Optionally, you can specify if the values should be cached with the source or destination IP address. By default, super_mediator uses the source IP address (SIP). You may decide to use the destination IP address for fields that are traditionally found in the reverse direction of the flow, such as "httpResponse" or "httpHost". Or you can use FLOWKEYHASH to deduplicate flows that have the same 5-tuple and data field. The FIELD_LIST is a list of information element IDs, separated by a comma, and surrounded by square brackets, [ and ]. To use a VLAN_MAP or OBID_MAP in addition to the SIP, DIP, FLOWKEYHASH key use MAP(MAPNAME). The VLAN_MAP or OBID_MAP with MAPNAME must be defined before the DEDUP_CONFIG block in the configuration file. If a MAP is used, then a TEXT record will include an extra column at the end of the line with the MAPNAME and an IPFIX record will contain an additional information element observationDomainName with the name of the MAP. A list of information element IDs can be found above in DPI_CONFIG. The only valid DNS and SSL/TLS information element IDs are 179 and 244, respectively. 179 will enable de-duplication of DNS queries only. See the DNS_DEDUP block for more information on de-duplicating on DNS responses. 244 will de-duplicate SSL certificate chains used by a particular IP. To de-duplicate on all SSL certificates, see the SSL_DEDUP option in the SSL_CONFIG block.

DEDUP_CONFIG "exporter1"
    PREFIX "useragent" [111]
    PREFIX "host" DIP [120]
    PREFIX "p0f" [36, 37, 107]
    PREFIX "dns" [179]
    PREFIX "ssl" [244]
DEDUP_CONFIG END

MERGE_TRUNCATED

If present, super_mediator will be less strict in deduplicating values by merging truncated values into complete cached records. For example, the following records:

2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|1|Mozilla/5.0 (X11; U; CrOS i686
2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|1|Mozilla/5.0 (X11; U; CrOS i686 0.10.
2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|4|Mozilla/5.0 (X11; U; CrOS i686 0.10.146;
2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|18|Mozilla/5.0 (X11; U; CrOS i686 0.10.146; en-US)

will be collapsed into one record:

1296251215151|1296252025761|10.10.1.6|24|Mozilla/5.0 (X11; U; CrOS i686 0.10.146; en-US)

FLUSH_TIME FLUSH_SECONDS

If present, the super_mediator will flush a dedup record from the cache when a new record has not been seen for over FLUSH_SECONDS. super_mediator will write the record when either the FLUSH_TIME or MAX_HIT_COUNT condition has been met. The default FLUSH_SECONDS is 300 seconds, or 5 minutes.

MAX_HIT_COUNT COUNT

If present, super_mediator will flush a dedup record from the cache when the internal hit count reaches COUNT. The record will be written at this time. The default COUNT is 5000.

ADD_EXPORTER_NAME

If present, super_mediator will add an extra column onto the TEXT record with the name of the EXPORTER associated with DEDUP_CONFIG block. If the EXPORTER is an IPFIX EXPORTER, the observationDomainName IE field will be populated with the exporter name.

MAP Blocks

The first command below is used at the top-level to begin a MAP block, and the remaining commands are accepted within the MAP context. The information from the MAP block is used by super_mediator in conjunction with the DEDUP_CONFIG block to determine which fields to perform de-deduplication on and how to aggregate the data. The use of map blocks are optional. They are only used when there is a need to aggregate data on VLAN(s) or observation domain(s) in addition to IP address. This block should precede any DEDUP_CONFIG block that intends to use the defined MAP.

[VLAN|OBID]_MAP MAPNAME

The VLAN_MAP or OBID_MAP command begins a new map definition block which continues to the VLAN_MAP END or OBID_MAP END command. The argument to the VLAN_MAP or OBID_MAP command is the name of the MAP. If VLAN_MAP is used, any value defined in the VLAN_MAP block will be compared with the IPFIX information element vlanId. If OBID_MAP is used, any value defined in the OBID_MAP block will be compared with the observation domain field in the IPFIX header.

[VLAN|OBID]_MAP END

The VLAN_MAP END or OBID_MAP END command ends the definition of a map. Following this command, top-level commands are again accepted.

GROUP_NAME [INTEGER VALUE LIST] or OTHER

This command adds integer values to a MAP group, where each integer is treated as a VLAN ID or observation domain ID. This command may appear multiple times in a MAP block. Each integer value may be between 0 and 65535. If the OTHER keyword is present instead of an INTEGER VALUE LIST, then the GROUP_NAME will be used for any value not in one of the lists within the MAP block.

VLAN_MAP networks
  "S1" [1,2,3]
  "S2" [4,5,6]
  "S3" [7,8,9]
  "other" OTHER
VLAN_MAP END

DISCARD_OTHER

If this command is present within a VLAN_MAP or OBID_MAP block, then any flow received that does not match one of the values in one of the lists, will be discarded.

OBID_MAP "mymap"
   "S1" [1,2,3]
   "S2" [4,5,6]
   "S3" [7,8,9]
   DISCARD_OTHER
OBID_MAP END

Tombstone Block

The first command below is used at the top-level to begin a TOMBSTONE_CONFIG block, and the remaining commands are accepted within the TOMBSTONE_CONFIG context. The information from the TOMBSTONE_CONFIG block is used by super_mediator to decide if and how to send tombstone records to exporters. Tombstone records are IPFIX options template records that are to be used to extract information about the delay and operation of a chain of tools. They carry a creation timestamp and a timestamp from certain supported programs that forward the record.

TOMBSTONE_CONFIG: The TOMBSTONE_CONFIG command begins a tombstone configuration block and it continues until the TOMBSTONE_CONFIG END command. There should be at most one tombstone configuration block per configuration file.
TOMBSTONE_CONFIG END: The TOMBSTONE_CONFIG END command ends the definition of a tombstone configuration block. Following a TOMBSTONE_CONFIG END command, top-level commands are again accepted.
GEN TOMBSTONE: Tells super mediator to generate and output tombstone records. Tombstone records will be sent to any exporter that is configured to receive stats. See other documentation for the configuration of stats.
TOMBSTONE_CONFIGURED_ID TOMBSTONE_CONFIGURED_ID: If present, the TOMBSTONE_CONFIGURED_ID command will override the default id value of 0 with the given TOMBSTONE_CONFIGURED_ID. This value must be specified in decimal and must be less than 65535.

SILK_CONFIG Block

The optional SILK_CONFIG block allows flow records to be labeled with a Sensor and a Type similar to how SiLK's rwflowpack(8) tool categorizes records. The command was added in super_mediator 1.9.0 and it is only allowed when super_mediator has been compiled with SiLK Type and Sensor labeling support.

The SILK_CONFIG block provides the file pathnames of the silk.conf(5) and sensor.conf(5) files as defined by SiLK. Only the sensor blocks in the sensor.conf file are used; any probe blocks are ignored.

When a SILK_CONFIG block is present, it applies to every EXPORTER in the configuration file. However, for records to be labeled, the records must be read by a COLLECTOR that contains a SILK_PROBE statement. If the COLLECTOR does not have a SILK_PROBE statement, its records are not labeled. As a reminder, the SILK_PROBE statement is

SILK_PROBE PROBE_NAME VLAN|INGRESS

When processing a flow record, the EXPORTER gets the PROBE_NAME from the record's COLLECTOR and finds every sensor block in the sensor.conf file that uses that PROBE_NAME. For each of these sensor blocks, the flow labeled with that sensor's name and the definitions of the internal, external, and null networks are used to assign a flow-type to the record.

The networks may be defined by IP addresses or by integer values (labeled "interface" values in the sensor.conf file) based on either the VLAN IDs (vlanId, reverseVlanId) or the SNMP interfaces (ingressInterface, egressInterface). Which value to use is determined by the second argument to the SILK_PROBE statement in the COLLECTOR block.

Note that SiLK records are uni-directional and a request may be labeled as incoming (in) and the response as outgoing (out). IPFIX records are typically bi-directional and the flow-type assigned by super_mediator applies to the forward direction of the record. There is an implicit label in the opposite direction for a bidirectional flow.

The logic in super_mediator to label flow records is loosely based on how rwflowpack categorized data for the "twoway" site, packlogic-twoway(3). Note that data that crosses the border is only labeled as "in" or "out", the "inweb" and "outweb" types are not used.

The syntax of the SILK_CONFIG block is described next.

SILK_CONFIG: The SILK_CONFIG command is given in the top-level context and it begins a SILK_CONFIG block. There may only be one SILK_CONFIG block per configuration.
SILK_CONFIG END: This command closes the SILK_CONFIG block and sets context back to top level.
SILK_CONF_PATH FILE_PATHNAME: Sets the location of the silk.conf(5) file to use for labeling flow records. This command is required.
SENSOR_CONF_PATH FILE_PATHNAME: Sets the location of the sensor.conf(5) file to use for labeling flow records. This command is required.

The record added to the multi-list uses this template:

silkSensorName CERT (PEN 6871) IE 941, variable length, string: The name of the silkSensorId as given in the sensor.conf file.
silkSensorDescription CERT (PEN 6871) IE 942, variable length, string: The description of of the silkSensorId as specified in the silk.conf file.
silkTypeName CERT (PEN 6871) IE 940, variable length, string: The type name of the silkFlowtypeId.
silkSensorId CERT (PEN 6871) IE 31, 2 octets, unsigned: The numeric identifier of the sensor named by silkSensorName. The silk.conf contains the mapping between silkSensorId and silkSensorName.
silkFlowtypeId CERT (PEN 6871) IE 30, 1 octet, unsigned: The numeric identifier of the silkTypeName as defined by the silk.conf file.

When exporting as JSON from super_mediator, the values are given at the top-level if exactly one label exists:

{
  "flows": {
    "flowStartMilliseconds": "2012-06-30 14:07:30.085",
    "flowEndMilliseconds": "2012-06-30 14:07:31.226",
    ...
    "silkSensorName": "S1",
    "silkSensorDescription": "",
    "silkTypeName": "in",
    "silkSensorId": 1,
    "silkFlowtypeId": 0
  }
}

If multiple labels are present, they are presented in an array:

{
  "flows": {
    "flowStartMilliseconds": "2012-06-30 14:07:30.085",
    "flowEndMilliseconds": "2012-06-30 14:07:31.226",
    ...
    "smSilkTypeSensorList": [
      {
        "silkSensorName": "S1",
        "silkSensorDescription": "",
        "silkTypeName": "in",
        "silkSensorId": 1,
        "silkFlowtypeId": 0
      },
      {
        "silkSensorName": "S2",
        "silkSensorDescription": "Optional description for sensor S2",
        "silkTypeName": "out",
        "silkSensorId": 2,
        "silkFlowtypeId": 1
      }
    ]
  }
}

EXAMPLES

TCP Collector

COLLECTOR TCP
    HOST "127.0.0.1"
    PORT 18000
COLLECTOR END

Spread Collector with name "SP0"

COLLECTOR SPREAD SP0
    DAEMON "4803"
    GROUP TEST2
COLLECTOR END

Poll-Directory Collector with Port 53 Filter

COLLECTOR DIR
    ANY_PORT == 53
    PATH "/tmp/flow/yaf-*"
    POLL 30
    MOVE "/data/flow/done"
    LOCK
COLLECTOR END

Filter on Collection (applied to all Collectors)

FILTER
    APPLICATION == 80
    ANY_PORT == 80
FILTER END

TCP Exporter (rwflowpack) with name "SiLK"

EXPORTER TCP SiLK
    HOST "localhost"
    PORT 18001
    FLOW_ONLY
EXPORTER END

TCP Exporter for flows that were collected by COLLECTOR "C1"

EXPORTER TCP
    COLLECTOR == C1
    HOST "localhost"
    PORT 18001
    FLOW_ONLY
EXPORTER END

De-duplicated DNS to CSV files

EXPORTER TEXT
    PATH "/data/dns/yaf2dns"
    ROTATE 120
    LOCK
    DNS_DEDUP_ONLY
    DELIMITER ","
EXPORTER END

Example Data in "/data/dns/yaf2dns-20120504-0001.txt":

2012-04-10 04:41:54.194,2,wikimedia.org.,ns2.wikimedia.org.

Export to TEXT files with filter

EXPORTER TEXT
    PATH "/data/http-only.txt"
    APPLICATION == 80
    DPI_ONLY
EXPORTER END

Example Data in /data/http-only.txt:

flow|1441601726|1207802496583|1.2.3.4|208.80.152.2|6|1360|80|0
http|115|1441929406|1207802496582|http://en.wikipedia.org/wiki/Http
http|114|1441601726|1207802496583|HTTP/1.0
http|117|1441601726|1207802496583|en.wikipedia.org

Spread Exporter with filters for each group

EXPORTER SPREAD
    DAEMON "4803"
    GROUP TEST1
    GROUP TEST2
EXPORTER END

GROUP TEST1
    ANY_PORT == 53
GROUP END

GROUP TEST2
    ANY_PORT == 80
GROUP END

De-duplication options

DNS_DEDUP
    FLUSH_TIME 600
    BASE64_ENCODE
DNS_DEDUP END

Example Using BASE_64 Encode:

2010-07-21 11:51:15.166|6|bmVtby3kbGEubTlsLg==|bnMxLmTsYS5taWwu
2010-07-21 11:51:15.175|2|bm5zYy58kbGEuLWlsLg==|bnMnLmRsYS9taWwu

DPI Configuration options - MULTI_FILE Example

EXPORTER TEXT
    PATH "/data/dpi"
    ROTATE 120
    LOCK
    DPI_ONLY
    MULTI_FILES
EXPORTER END

DPI_CONFIG
    TABLE myhttp [110, 111, 112]
    TABLE mydns [1, 28]
    TABLE myp0f [36, 37]
    TABLE myssl [244,247,248,443]
DPI_CONFIG END

Example Data for MULTI_FILE Config Example:

In /data/dpi/flow.txt0:
    1441601726|1207802496583|10.10.1.172|10.10.152.2|6|1360|80|0|0
    114422227|1207802496560|10.10.1.172|10.10.1.10|17|1599|53|0|0
    4144722023|1296251240788|10.10.0.204|10.10.4.3|6|54489|443|900|0
In /data/dpi/myp0f.txt0:
    36|1441601726|1207802496583|0|Windows
    37|1441601726|1207802496583|0|2000 SP2+, XP SP1+ (seldom 98)
In /data/dpi/myhttp.txt0:
    112|1441601726|1207802496583|0|/skins-1.5/monobook/headbg.jpg
    112|1441601726|1207802496583|0|/skins-1.5/common/images/poweredby_mediawiki_88x31.png
In /data/dpi/mydns.txt0:
    114422227|1207802496560|0|Q|14728|0|0|0|1|0|meta.wikimedia.org.
    114422227|1207802496560|0|R|14728|1|0|0|1|120|rr.pmtpa.wikimedia.org.|10.10.15.2
In /data/dpi/myssl.txt0:
    10|4144722023|1296251240788|0|S|0|Apple Inc

DPI Configuration with alternative index name

EXPORTER TEXT DPIExporter
    PATH "/data/dpi/dpi_data.txt"
    DPI_ONLY
EXPORTER END

DPI_CONFIG flow_index
    TABLE kitchen_sink [110,111,112,113,2,15,186,200,201]
DPI_CONFIG END

Example Data with above configuration:

flow_index|1441208511|1207802506600|128.237.224.172|208.80.152.3|6|1370|80|0|0
kitchen_sink|111|1441208511|1207802506600|0|Mozilla/4.0(compatible; MSIE 7.0; Windows NT 5.1;)
kitchen_sink|112|1441208511|1207802506600|0|/wikipedia/commons/thumb/d/de/Www.wikipedia.org_screenshot.png/300px-Www.wikipedia.org_screenshot.png

IPset filter in Exporter

EXPORTER TEXT
    ANY_IP IN_LIST "mywatchlist.set"
    PATH "/data/dpi/dpi_data.txt"
    DPI_ONLY
EXPORTER END

Custom Field List Example

EXPORTER TEXT
    PATH "/data/flow/custom.txt"
    FIELDS stime,etime,sip,dip,sport,dport,protocol,vlan,pkts,bytes,dpi
    DPI_FIELD_LIST [111,112,110,1,2,5,6,12,244,248,247,443]
EXPORTER END

Custom SSL Field Example

EXPORTER TEXT sslcerts
   PATH "/data/flow/sslcerts.txt"
   FIELDS stime, sip, dip, sport, dport, dpi
EXPORTER END

SSL_CONFIG sslcerts
   ISSUER [*]
   SUBJECT [7, 8]
   OTHER  [247, 248]
   EXTENSIONS [14, 15]
SSL_CONFIG END

Example Data with above configuration:

2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|8|S|0|Washington
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|7|S|0|Seattle
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|247|I|0|140601000000Z
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|248|I|0|150521235959Z
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|6|I|1|US
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|11|I|1|VeriSign Trust Network

SSL Certificate De-duplication Example

EXPORTER TEXT ssldedup
    PATH "/data/ssl/ssldedup"
    SSL_DEDUP_ONLY
    ROTATE 600
    LOCK
EXPORTER END

SSL_CONFIG ssldedup
    ISSUER [*]
    SUBJECT [*]
    OTHER [*]
    EXTENSIONS [*]
    CERT_FILE "/data/ssl/certs"
    FLUSH_TIME 1200
    MAX_HIT_COUNT 10000
SSL_CONFIG END

Example Data with above configuration:

$ head -n 5 /data/ssl/certs.txt
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|6|I|0|US
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|10|I|0|VeriSign, Inc.
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|11|I|0|VeriSign Trust Network
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|11|I|0|Terms of use at https://www.verisign.com/rpa (c)10
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|3|I|0|VeriSign Class 3 International Server CA - G3

$ head -n 5 /data/ssl/ssldedup.txt
2015-04-08 19:14:14.618|2015-04-08 19:14:30.117|0x5294e23f0bfa5bb98c0|2|VeriSign Class 3 International Server CA - G3
2015-04-08 19:14:39.902|2015-04-08 19:14:39.915|0x009ddde63d7dc9573067e|2|EssentialSSL CA
2015-04-08 19:14:39.902|2015-04-08 19:14:39.915|0x18b2cbbfc1f2f326462a4a|2|COMODO Certification Authority
2015-04-08 19:14:39.902|2015-04-08 19:14:39.915|0x2e79832eef31a6ee67a44|2|UTN - DATACorp SGC

HTTP Deduplication Example

EXPORTER TEXT "httpdedup"
    PATH "/data/dedup"
    ROTATE 120
    LOCK
EXPORTER END

DEDUP_CONFIG "httpdedup"
    PREFIX "useragent" SIP [111]
    PREFIX "referer" [115]
    PREFIX "host" DIP [117]
    MAX_HIT_COUNT 65535
    FLUSH_TIME 600
    MERGE_TRUNCATED
DEDUP_CONFIG END

Example data with the above configuration:

$ head -n 4 /data/useragent_20110128220025.txt
2015-04-08 19:15:24.645|2015-04-08 19:15:24.645|10.10.1.60|2015-04-08 19:15:24.645|2221618956|2|Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8)
2015-04-08 19:15:28.809|2015-04-08 19:15:28.809|10.10.0.205|2015-04-08 19:15:24.645|546421315|1|TwitterAndroid/1.0.5 (109) Nexus One/8 (HTC;passion)
2015-04-08 19:15:11.544|2015-04-08 19:16:18.351|10.13.0.63|2015-04-08 19:15:24.645|213547784|2|urlgrabber/3.9.1 yum/3.2.28
2015-04-08 19:16:21.632|2015-04-08 19:16:21.668|10.10.1.5|2015-04-08 19:15:24.645|1315645613|4|OpenTable/3.2 CFNetwork/485.12.7 Darwin/10.4.0

$ head -n 4 /data/referer_20110128220025.txt
2015-04-08 19:15:24.645|2015-04-08 19:15:24.645|10.27.33.66|2015-04-08 19:15:24.645|532889529|10|http://www.google.com/search?hl=en&biw=1274
2015-04-08 19:15:28.809|2015-04-08 19:15:28.809|10.10.1.45|2015-04-08 19:15:24.645|654654123|4|http://www.ustream.tv/socialstream/6951299
2015-04-08 19:15:11.544|2015-04-08 19:16:18.351|10.11.0.139|2015-04-08 19:15:24.645|212754153|2|http://reviews.opentable.com/0938/33364/reviews.htm
2015-04-08 19:16:21.632|2015-04-08 19:16:21.668|10.10.1.31|2015-04-08 19:15:24.645|3264312556|1|http://www.northerntool.com/shop/tools/product.htm

$ head -n 4 /data/host_20110128220025.txt
2015-04-08 19:15:24.645|2015-04-08 19:15:24.645|10.10.0.196|2015-04-08 19:15:24.645|3251463421|1|www.funtrivia.com
2015-04-08 19:15:28.809|2015-04-08 19:15:28.809|172.16.0.163|2015-04-08 19:15:24.645|134313131|1|twitter.com
2015-04-08 19:15:11.544|2015-04-08 19:16:18.351|10.10.0.247|2015-04-08 19:15:24.645|313546131319|16|reviews.opentable.com
2015-04-08 19:16:21.632|2015-04-08 19:16:21.668|10.10.1.45|2015-04-08 19:15:24.645|210564613203|7|a2.twimg.com

Deduplication with a OBID_MAP

OBID_MAP "sensors"
    S1 [1, 2, 3]
    S2 [4, 5, 6]
    S3 [7, 8, 9]
    "S_OTHER" OTHER
OBID_MAP END

EXPORTER TEXT "dedup"
    PATH "/data/dedup/"
EXPORTER END

DEDUP_CONFIG "dedup"
    PREFIX "useragent" SIP MAP(sensors) [111]
    MAX_HIT_COUNT 65535
    FLUSH_TIME 600
    MERGE_TRUNCATED
DEDUP_CONFIG END

Example with the above configuration:

$ head -n 4 /data/dedup/useragent.txt
2011-01-28 21:45:53.130|2011-01-28 21:49:33.630|10.10.1.13|2011-01-28 21:49:33.518|327929330|8|Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B367 Safari/531.21.10|S_OTHER
2011-01-28 21:48:45.775|2011-01-28 21:48:45.775|10.10.0.209|2011-01-28 21:48:43.713|2118954148|1|Google Update/1.2.183.39;winhttp|S2
2011-01-28 21:48:47.163|2011-01-28 21:48:47.163|10.10.0.209|2011-01-28 21:48:46.905|1985639067|1|Microsoft NCSI|S1

IPFIX Exporter with multiple protocol deduplication

EXPORTER FILEHANDLER "multi-dedup"
    PATH "/data/sm/super"
    ROTATE 120
    GZIP_FILES
    MOVE "/data/sm/done"
    DNS_DEDUP
    SSL_DEDUP
    NO_FLOW
EXPORTER_END

DNS_DEDUP "multi-dedup"
    MAX_HIT_COUNT 5000
    FLUSH_TIME 600
    LAST_SEEN
DNS_DEDUP END

DEDUP_CONFIG "multi-dedup"
    PREFIX "httpua" [111]
    PREFIX "httplang" [121]
    PREFIX "httpserver" DIP [110]
    PREFIX "httphost" DIP [117]
    PREFIX "https" DIP [244]
    MERGE_TRUNCATED
    MAX_HIT_COUNT 10000
    FLUSH_TIME 600
DEDUP_CONFIG END

Logging options

LOG "/var/log/mediator.log"

LOGLEVEL DEBUG

KNOWN ISSUES

Bug reports and feature requests may be sent via email to <netsa-help@cert.org>.

yaf presently encodes the ICMP type and code information into the destinationTransportPort information element for ICMP and ICMP6 flows. super_mediator running in TEXT output mode writes the type in the sourceTransportPort field and the ICMP code in the destinationTransportPort field.

AUTHORS

Emily Sarneso and the CERT Network Situational Awareness Group Engineering Team, <http://www.cert.org/netsa>.