SiLK

NAME

rwflowpack - Collect flow data and store it in binary SiLK Flow files

SYNOPSIS

rwflowpack [--input-mode=MODE] INPUT_MODE_SPECIFIC_SWITCHES
      [--output-mode=MODE] OUTPUT_MODE_SPECIFIC_SWITCHES
      { --log-destination=DESTINATION
        | --log-pathname=FILE_PATH
        | --log-directory=DIR_PATH [--log-basename=LOG_BASENAME]
          [--log-post-rotate=COMMAND] }
      [--no-file-locking] [--flush-timeout=VAL]
      [--file-cache-size=VAL] [--pack-interfaces]
      [--byte-order=ENDIAN] [--compression-method=COMP_METHOD]
      [--error-directory=DIR_PATH] [--archive-directory=DIR_PATH]
      [--flat-archive] [--post-archive-command=COMMAND]
      [--site-config-file=FILENAME] [--log-level=LEVEL]
      [--log-sysfacility=NUMBER] [--pidfile=FILE_PATH]
      [--no-chdir] [--no-daemon]

To collect flow data over the network or directory polling (default):

rwflowpack [--input-mode=stream] --sensor-configuration=FILE_PATH
      [--packing-logic=PLUGIN] [--sensor-name=SENSOR]
      [--polling-interval=NUMBER] ...

To collect from local files containing flows created by flowcap(8):

rwflowpack --input-mode=fcfiles --incoming-directory=DIR_PATH
      --sensor-configuration=FILE_PATH [--packing-logic=PLUGIN]
      [--polling-interval=NUMBER] ...

To collect from a single file containing NetFlow v5 PDUs:

rwflowpack --input-mode=pdufile --netflow-file=FILE_PATH
      --sensor-configuration=FILE_PATH [--packing-logic=PLUGIN]
      [--sensor-name=SENSOR] ...

To respool SiLK Flows without modifying the class, type, or sensor:

rwflowpack --input-mode=respool --incoming-directory=DIR_PATH
      [--polling-interval=NUMBER] ...

To store the SiLK Flow files on the local machine (default):

rwflowpack ... [--output-mode=local-storage]
      --root-directory=DIR_PATH ...

To create incremental files to be processed by rwflowappend(8):

rwflowpack ... --output-mode=incremental-files
      --incremental-directory=DIR_PATH ...

To create incremental files to be processed by rwflowappend (deprecated):

rwflowpack ... --output-mode=sending --sender-directory=DIR_PATH
      --incremental-directory=DIR_PATH ...

Help options:

rwflowpack --sensor-configuration=FILE_PATH [--packing-logic=PLUGIN]
      { --verify-sensor-config | --verify-sensor-config=VERBOSE }

rwflowpack --help

rwflowpack --version

DESCRIPTION

rwflowpack is a daemon that runs as part of the SiLK flow collection and packing tool-chain. The primary job of rwflowpack is to convert each incoming flow record to the SiLK Flow format, categorize each incoming flow record (e.g., as incoming or outgoing), set the sensor value for the record, and determine which hourly file will ultimately store the record.

The settings that rwflowpack uses to categorize each flow record are determined by two textual configuration files and compiled code that is referred to as packing logic.

The first of the configuration files is silk.conf(5) which specifies the classes, types, and sensors to use at your site. There are several different ways to specify the location of this file as detailed in the "FILES" section below.

The second configuration file is the sensor.conf(5) file, whose location is specified via the --sensor-configuration switch. This file contains multiple sensor blocks, where each block contains information used to categorize flow records captured at that particular sensor. This file also contains probe blocks which specify how to collect NetFlow v5, IPFIX, and/or NetFlow v9 flow records, and a mapping of probes to sensors. See the sensor.conf(5) manual page for details.

The packing logic uses the information from the silk.conf and sensor.conf files to categorize each flow record. By categorizing a flow record, we mean determine whether the record is incoming or outgoing and assign a class/type pair (also called a flowtype) to the record. The flowtype along with the starting hour of the record and the sensor where the record was collected form a triple which determines into which file a flow record is stored. The files that rwflowpack produces have the form flowType-sensorName_YYYYMMDD.HH where flowType encodes the class/type pair, sensorName is the sensor where the flows were collected, and YYYYMMDD.HH is the year, month, day, and hour when the flow started.

For more information on how rwflowpack categorizes each flow record and converts data to the SiLK Flow format, see the SiLK Installation Handbook, the sensor.conf(5) manual page, and the manual page for the packing logic plug-in that rwflowpack is using (e.g., packlogic-twoway(3) is the default, packlogic-generic(3)).

The compiled code for the packing-logic is normally loaded into rwflowpack as a run-time. (The administrator may choose to compile the packing logic into rwflowpack when building SiLK. See the SiLK Installation Handbook for details.) The name of this plug-in is specified in the packing-logic statement in the silk.conf file. A different location may be specified using the --packing-logic switch.

Input Modes

There are several ways to input data to rwflowpack, and the method to use is determined by the --input-mode switch, with stream being the default when the switch is not provided.

stream

In stream input-mode, rwflowpack processes the probe statements in the sensor.conf(5) file which specify how rwflowpack is to capture data from one or more sources. The data is assigned to a sensor based on the probe-sensor mapping in the sensor.conf file. rwflowpack then categorizes the records, converts them to the SiLK Flow format, and writes them to files.

The sources of data that rwflowpack supports are:

• listening for NetFlow v5 packets on a UDP socket

• listening for IPFIX (Internet Protocol Flow Information eXport) packets on a TCP or a UDP socket

• listening for NetFlow v9 packets on a UDP socket

• listening for sFlow v5 packets on a UDP socket

• polling a directory for files containing NetFlow v5 records (see the description of the pdufile input-mode for the required format of these files)

• polling a directory for files containing IPFIX records (as generated by yaf(1))

• polling a directory for files containing SiLK Flow records (compare to the respool input-mode)

Multiple sources may be specified.

Processing of IPFIX, NetFlow v9, or sFlow is only available when SiLK is compiled with support for libfixbuf-1.7.0 or later. Processing of sFlow records was added in SiLK 3.9.0. libfixbuf is available from http://tools.netsa.cert.org/fixbuf/.

Configuration of stream input-mode is specified in the "Stream Collection Switches (--input-mode=stream)" section below.

fcfiles

Instead of having rwflowpack capture data itself, you may configure rwflowpack to work in conjunction with one or more flowcap(8) daemons by specifying fcfiles as the input-mode.

In this configuration, each flowcap uses the probe statements in the sensor.conf(5) file to determine how to collect the data. flowcap supports the network-based capture methods specified for the stream input-mode---i.e., flowcap does not support directory polling. flowcap writes the data it captures into small files and includes the probe name in each file's header.

Typically, the flowcap processes run on separate machines near the router or flow meter that is generating the records. The rwsender(8) and rwreceiver(8) daemons are often used to move the files from the flowcap machines to the rwflowpack machine.

rwflowpack polls a directory for the files created by flowcap. Once it finds a file, it assigns those records a sensor based on the probe-sensor mapping in the sensor.conf file, it categorizes the records, and it writes the records to one or more output files.

Since rwflowpack does not capture data in fcfiles input-mode, rwflowpack does not use the probe statements in the sensor.conf file, and the statements do not need to be present.

The switches that rwflowpack uses in fcfiles input-mode are given below ("Flowcap Files Collection Switches (--input-mode=fcfiles)").

pdufile

Setting the input-mode to pdufile tells rwflowpack to read a single file containing NetFlow v5 PDU records and then exit. rwflowpack does not become a daemon in this input-mode; instead it remains in the foreground, processes the NetFlow file, and exits.

The file must be in the format created by NetFlow Collector: The file's size must be an integer multiple of 1464, where each 1464 byte chunk contains a 24-byte NetFlow v5 header and space for thirty 48-byte NetFlow records. The number of valid records per chunk is specified in the chunk's header. (This is also the format that rwflowpack requires in stream input-mode when it is polling a directory for NetFlow v5 files.)

To convert single PDU file to a stream of SiLK Flow records, consider using rwpdu2silk(1).

In pdufile input-mode, the sensor.conf file must define a sensor that maps to a probe that uses the read-from-file statement. However, the argument to that statement is ignored, and the argument is typically set to /dev/null. The NetFlow v5 file's location must be specified with the --netflow-file switch. If sensor.conf contains multiple sensor blocks, the --sensor-name switch is required to tell rwflowpack which sensor to use.

See the "PDU File Collection Switches (--input-mode=pdufile)" section below for additional configuration information.

respool

Sometimes it is desirable to pull existing SiLK Flow records from one data repository and use them to create a "mini" data repository (for example, for testing). The respool input-mode is one way to accomplish this.

In this configuration, rwflowpack polls a directory for SiLK flow files and it uses the existing class/type pair and sensor values to determine where to store the flow record. That is, rwflowpack puts the data into appropriate hourly file, but it does not change any other settings on the flow records.

To contrast respool input-mode with rwflowpack processing SiLK Flow files in stream input-mode: In respool input-mode, the existing class, type, and sensor values are used to determine each record's destination. In the latter, rwflowpack treats the records as it would any other newly arrived data, assigning the data to a sensor and re-categorizing the records to assign a class/type pair to them.

Since no categorization occurs in respool input-mode, the --sensor-configuration and --packing-logic switches are not required and not allowed, and their presence causes rwflowpack to exit with an error code.

Output Modes

As mentioned above, after rwflowpack categorizes a flow record (that is, determines the sensor, class/type, and starting hour for the record), it uses those values to generate the name of the hourly file that will contain that record, and it writes the record to that file.

In order for the record in that file to be retrievable by rwfilter(1), the file must be stored in a SiLK data repository, which is a directory tree of binary SiLK Flow files. The root of this directory tree is called the SILK_DATA_ROOTDIR. The structure of the tree under the root is determined by the path-format setting in the silk.conf(5) file.

There are two ways to get the files into the SILK_DATA_ROOTDIR; which method is used is determined by rwflowpack's --output-mode switch. This switch supports the following values:

local-storage

In local-storage output-mode, rwflowpack creates the hourly SiLK Flow files directly in the data repository, and it writes the records into these files. rwflowpack uses local-storage output-mode when the --output-mode switch is not provided.

incremental-files

When the output-mode is incremental-files, rwflowpack does not create hourly data files directly. Instead, rwflowpack creates smaller files (called incremental files), and rwflowpack relies on the rwflowappend(8) daemon to combine the incremental files into hourly files in the final data repository.

In incremental-files output-mode, rwflowpack uses a single destination directory whose location is specified by the --incremental-directory switch. In this directory, rwflowpack creates a zero-byte place holder file and a corresponding working file. The name of the place holder file has a unique, random suffix, and the name of the working file is a dot (.) followed by the name of the place holder file.

rwflowpack first writes records to the working files. Periodically (as determined by the value specified to --flush-timeout), rwflowpack closes all the working files and moves them over the place holder files. Once rwflowpack has closed and moved a working file, it no longer takes responsibility for it.

The rwflowappend process may poll the incremental-directory itself, or another process may poll that directory and pass the files to rwflowappend. If rwflowpack and rwflowappend are on different machines, an rwsender(8)/rwreceiver(8) pair may be used to move the files between the machines.

Additional reasons one may want to use incremental-files output-mode are to support having multiple data repositories or to allow additional processing of the SiLK Flow records (such as by the Analysis Pipeline (http://tools.netsa.cert.org/analysis-pipeline/)).

Note: This output-mode was introduced in SiLK 3.6.0. This mode is similar to the sending output-mode and is meant to replace it. In incremental-files output-mode, all writing occurs within the incremental-directory, while the sending output-mode uses two directories.

sending

This output-mode works similarly to incremental-files. The difference is that when rwflowpack flushes the open files, it moves the files from the incremental-directory and to the directory specified by the --sender-directory switch. Once a file is moved to the sender-directory, rwflowpack no longer takes responsibility for it.

As of SiLK 3.6.0, the sending output-mode is deprecated.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

Input and Output Mode

As described in the "Input Modes" section above, rwflowpack has multiple ways of getting data, and that data may be stored in one of two methods (c.f., "Output Modes"). Choosing the modes and configuring each mode are described in the sections below.

--input-mode=MODE

Determine how rwflowpack gathers data. The default input MODE is stream. The available input-modes are stream, fcfiles, pdufile, and respool.

--output-mode=MODE

Determine what rwflowpack does with the data as it is packed into SiLK binary files. The default output MODE is local-storage. The available output-modes are local-storage, incremental-files and sending.

Stream Collection Switches (--input-mode=stream)

In stream input-mode, rwflowpack uses the probe statements in the sensor.conf file to capture data, and then rwflowpack categorizes the data. The stream input-mode is the default when the --input-mode switch is not provided. This input-mode accepts the following switches; the --sensor-configuration switch is required, and all other switches are optional.

--sensor-configuration=FILE_PATH

Give the path to the configuration file that specifies how rwflowpack is to capture data and that tells rwflowpack whether a record represents an incoming or outgoing flow. The complete syntax of the configuration file is described in the sensor.conf(5) manual page; see also the SiLK Installation Handbook.

--packing-logic=PLUGIN

Specify the plug-in that rwflowpack should load, where the plug-in provides functions that determine into which class and type each flow record will be categorized and the format of the files that rwflowpack will write. When SiLK has been configured with hard-coded packing logic (i.e., when --enable-packing-logic was specified to the configure script), this switch will not be present on rwflowpack. A default value for this switch may be specified in the silk.conf(5) site configuration file (see the description of the --site-config-file switch). When PLUGIN does not contain a slash (/), rwflowpack attempts to find a file named PLUGIN in the directories listed in the "FILES" section. If rwflowpack finds the file, it uses that path. If PLUGIN contains a slash or if rwflowpack does not find the file, rwflowpack relies on your operating system's dlopen(3) call to find the file. When the SILK_PLUGIN_DEBUG environment variable is non-empty, rwflowpack prints status messages to the standard error as it attempts to find and open each of its plug-ins. rwflowpack exits if it does not have access to the packing logic functions.

--sensor-name=SENSOR

Cause rwflowpack to ignore all probes in the sensor configuration file except the probes for SENSOR. Only data for SENSOR is collected. This allows a common configuration file to be used by multiple rwflowpack invocations, yet also allow each rwflowpack instance only collect data for a single sensor. There must be a sensor definition for SENSOR in the configuration file. When this switch is not present, rwflowpack collects and packs data for all sensors.

--polling-interval=NUMBER

Specify the number of seconds rwflowpack waits between scans of the directories specified by the poll-directory settings of the probes in the sensor.conf file. The default is 15 seconds.

Flowcap Files Collection Switches (--input-mode=fcfiles)

As described in the "Input Modes" section above, in fcfiles input-mode, rwflowpack processes files created by the flowcap(8) daemon. In fcfiles input-mode, the --sensor-configuration and --incoming-directory switches are required.

--sensor-configuration=FILE_PATH

Give the path to the configuration file that rwflowpack consults to determine whether a record represents an incoming or outgoing flow. The complete syntax of the configuration file is described in the sensor.conf(5) manual page; see also the SiLK Installation Handbook.

--incoming-directory=DIR_PATH

Periodically scan the directory DIR_PATH for files that have been created by flowcap. As rwflowpack scans DIR_PATH, it ignores a file if its name begins with a dot (.) or if its size is 0 bytes. When a file is first detected, rwflowpack records its size, and the file must have the same size for two consecutive scans before rwflowpack processes it. After the file is successfully processed, rwflowpack either moves it to the archive-directory or deletes it. The interval between scans is set by --polling-interval. DIR_PATH must be a complete directory path.

--packing-logic=PLUGIN

Specify the plug-in that rwflowpack should load for the packing logic. For more detail, see the description above.

--polling-interval=NUMBER

Specify the number of seconds rwflowpack waits between polls of the incoming-directory for new files created by flowcap. If not given, the default value is 15 seconds.

PDU File Collection Switches (--input-mode=pdufile)

In this input-mode, rwflowpack stays in the foreground, processes a single file of NetFlow v5 data, and exits. The --sensor-configuration and --netflow-file switches are required. The --sensor-name switch is also required when the sensor.conf file contains more than one sensor. The following switches are available in pdufile input-mode:

--sensor-configuration=FILE_PATH

Give the path to the configuration file that rwflowpack consults to determine whether a record represents an incoming or outgoing flow.

--netflow-file=FILE_PATH

Name the full path of the file from which rwflowpack reads NetFlow v5 PDUs. This switch is required in PDU File input-mode.

--sensor-name=SENSOR

Cause rwflowpack to ignore all probes in the sensor configuration file except the probes for SENSOR. There must be a sensor definition for SENSOR in the configuration file. This switch is required in this input-mode unless the sensor.conf file only defines a single sensor.

--packing-logic=PLUGIN

Specify the plug-in that rwflowpack should load for the packing logic. For more detail, see the description of this switch in the stream input-mode.

Respooling Switches (--input-mode=respool)

When the --input-mode=respool switch is provided, rwflowpack polls a directory for SiLK Flow files, and writes the records it finds into new hourly files, leaving the sensor and class/type values unchanged in the records. (See "Input Modes" above for additional details.) The first of the following switches is required:

--incoming-directory=DIR_PATH

Periodically scan the directory DIR_PATH for SiLK Flow files to process. As rwflowpack scans DIR_PATH, it ignores a file if its name begins with a dot (.) or if its size is 0 bytes. When a file is first detected, rwflowpack records its size, and the file must have the same size for two consecutive scans before rwflowpack will process it. After the file is successfully processed, rwflowpack either moves it to the archive-directory or deletes it. The interval between scans is set by --polling-interval. DIR_PATH must be a complete directory path.

--polling-interval=NUMBER

Specify the number of seconds rwflowpack waits between polls of the incoming-directory. If not given, the default value is 15 seconds.

Local Storage Switches (--output-mode=local-storage)

In local-storage output-mode, rwflowpack stores SiLK Flow records directly in the data repository. This is the default when the --output-mode switch is not provided. When operating in this output-mode, the following switch is required:

--root-directory=DIR_PATH

Name the full path of the SILK_DATA_ROOTDIR; that is, the directory under which the files containing the packed SiLK Flow records are stored. rwflowpack creates subdirectories below DIR_PATH based on the data received and the path-format setting in the silk.conf(5) file.

Incremental-Files Switches (--output-mode=incremental-files)

As described in the "Output Modes" section above, the incremental-files output-mode tells rwflowpack to write incremental-files. The rwflowappend(8) daemon combines these incremental-files to create a SiLK data repository. When running in incremental-files output-mode, the following switch must be provided:

--incremental-directory=DIR_PATH

Name the full path of the destination directory where incremental-files are both created and stored to await action by another process such as rwflowappend or rwsender. It is recommended that only a single rwflowpack process write to this directory.

Sending Mode Switches (--output-mode=sending)

This output-mode is deprecated as of SiLK 3.6.0. This output-mode works similarly to the incremental-files output-mode, except rwflowpack moves the files to a second directory periodically. Both the following switches are required in sending output-mode:

--incremental-directory=DIR_PATH

Name the full path of the working directory under which incremental-files are initially created. Periodically (as determined by the --flush-timeout switch), rwflowpack closes the files in this directory and moves them to the sender-directory. An rwflowpack invocation assumes it has full control over the files in this directory. When rwflowpack starts, any files in this directory are moved to the sender-directory.

--sender-directory=DIR_PATH

Name the full path of the destination directory where incremental-files are moved to await action by another process such as rwflowappend or rwsender. Once rwflowpack moves files to this directory, it no longer takes responsibility for them. The other process (e.g., rwsender) is responsible for removing files from this directory. Multiple rwflowpack invocations may use a single sender-directory.

General Configuration

The following switches are optional:

--no-file-locking

Do not use advisory write locks. Normally, rwflowpack obtains a write lock on an data file prior to writing records to it. The write lock prevents two instances of rwflowpack from writing to the same data file simultaneously. However, attempting to use a write lock on some file systems causes rwflowpack to exit with an error, and this switch may be used when writing data to these file systems.

--flush-timeout=VAL

Set the timeout for flushing any in-memory records to disk to VAL seconds. If not specified, the default is 2 minutes (120 seconds). When the output-mode is local-storage, this value specifies how often the files are flushed to disk to ensure that any records in memory are written to disk. When using the incremental-files or sending output-mode, this value specifies how often to close and move the incremental files. See the "Output Modes" section for details.

--file-cache-size=VAL

Set the maximum number of data files to have open for writing at any one time to VAL. If not specified, the default is 128 files. The minimum file cache size is 4. This switch also determines how many files rwflowpack reads from simultaneously when using probes that poll directories for files (see sensor.conf(5)). The maximum number of input files open at any one time is limited to one eighth of VAL (with a minimum of 2), and the number of directory polling operations to perform simultaneously is limited to one sixteenth of VAL (minimum is 1).

--pack-interfaces

Allow one to override the default file output formats of the packed SiLK Flow files that rwflowpack writes. When this switch is present, rwflowpack writes additional information into the packed files: the router's SNMP input and output interfaces and the next-hop IP address. (When the sensor.conf file contains an interface-values attribute whose value is vlan, the input and output fields contain the vlan IDs instead of SNMP interface values.) The extra data produced by this switch is useful for determining why traffic is being stored in certain files. Note that this switch only affects newly created files. New records are always appended to an existing file in the file's current output format to maintain file integrity.

--byte-order=ENDIAN

Set the byte order for newly created SiLK Flow files. When appending records to an existing file, the byte order of the file is maintained. The argument is one of the following:

native

Use the byte order of the machine where rwflowpack is running. This is the default.

big

Use network byte order (big endian) for the flow files.

little

Write the flow files in little endian format.

--compression-method=COMP_METHOD

Specify the compression library to use when creating new files. When this switch is not given, newly created files are compressed using the default chosen when SiLK was compiled. When appending records to an existing file, the compression method of the file is maintained. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

none

Do not compress the output using an external library.

zlib

Use the zlib(3) library for compressing the output. Using zlib produces the smallest output files at the cost of speed.

lzo1x

Use the lzo1x algorithm from the LZO real time compression library for compression. This compression provides good compression with less memory and CPU overhead.

snappy

Use the snappy library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead. Since SiLK 3.13.0.

best

Use lzo1x if available, otherwise use snappy if available, otherwise use zlib if available.

--site-config-file=FILENAME

Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwflowpack searches for the site configuration file in the locations specified in the "FILES" section.

Disposal of Input Flow Files

The following switches determine how rwflowpack handles input files once it has processed them. These switches have no effect when rwflowpack reads all of its data directly from the network. Otherwise, the switches affect the named --netflow-file in pdufile input-mode, the files read from the --incoming-directory in fcfiles and respool input-mode, and files read from probes that have a poll-directory statement (c.f. sensor.conf(5)) in stream input-mode.

--error-directory=DIR_PATH

Move input files that cannot be opened, have an unexpected format, contain an unrecognized probe name in fcfiles input-mode, or are not successfully processed into the directory DIR_PATH. DIR_PATH must be a complete directory path. If this switch is not provided, problem files remain in place and cause rwflowpack to exit.

--archive-directory=DIR_PATH

Move input files that rwflowpack processes successfully into the directory DIR_PATH. DIR_PATH must be a complete directory path. When this switch is not provided and the input-mode is pdufile, the original NetFlow source file is not modified, moved, or deleted. In all other input-modes, no --archive-directory switch causes rwflowpack to delete each input file after successfully processing it. When the --flat-archive switch is also provided, incoming files are moved into the top of DIR_PATH; when --flat-archive is not given, each file is moved to a subdirectory based on the current UTC time: DIR_PATH/YEAR/MONTH/DAY/HOUR/. Removing files from the archive-directory is not the job of rwflowpack; the system administrator should implement a separate process to clean this directory. This switch is required when the --post-archive-command switch is present.

--flat-archive

When archiving input files via the --archive-directory switch, move the files into the top of the archive-directory, not into subdirectories of the archive-directory. This switch has no effect if --archive-directory is not also specified. This switch may be used to allow another process to watch for new files appearing in the archive-directory.

--post-archive-command=COMMAND

Run COMMAND on each input file after rwflowpack has successfully processed the file and moved the file into the archive-directory. Each occurrence of the string %s in COMMAND is replaced with the full path to the input file in the archive-directory, and each occurrence of %% is replaced with %. If any other character follows %, rwflowpack exits with an error. When using this feature, the --archive-directory switch must be specified. See also the rwpollexec(8) daemon.

Logging and Daemon Configuration

One of the following mutually-exclusive switches is required:

--log-destination=DESTINATION

Specify the destination where logging messages are written. When DESTINATION begins with a slash /, it is treated as a file system path and all log messages are written to that file; there is no log rotation. When DESTINATION does not begin with /, it must be one of the following strings:

none

Messages are not written anywhere.

stdout

Messages are written to the standard output.

stderr

Messages are written to the standard error.

syslog

Messages are written using the syslog(3) facility.

both

Messages are written to the syslog facility and to the standard error (this option is not available on all platforms).

--log-directory=DIR_PATH

Use DIR_PATH as the directory where the log files are written. DIR_PATH must be a complete directory path. The log files have the form

DIR_PATH/LOG_BASENAME-YYYYMMDD.log

where YYYYMMDD is the current date and LOG_BASENAME is the application name or the value passed to the --log-basename switch when provided. The log files are rotated: At midnight local time, a new log is opened, the previous file is closed, and the command specified by --log-post-rotate is invoked on the previous day's log file. (Old log files are not removed by rwflowpack; the administrator should use another tool to remove them.) When this switch is provided, a process-ID file (PID) is also written in this directory unless the --pidfile switch is provided.

--log-pathname=FILE_PATH

Use FILE_PATH as the complete path to the log file. The log file is not rotated.

The following set of switches is optional:

--log-level=LEVEL

Set the severity of messages that are logged. The levels from most severe to least are: emerg, alert, crit, err, warning, notice, info, debug. The default is info.

--log-sysfacility=NUMBER

Set the facility that syslog(3) uses for logging messages. This switch takes a number as an argument. The default is a value that corresponds to LOG_USER on the system where rwflowpack is running. This switch produces an error unless --log-destination=syslog is specified.

--log-basename=LOG_BASENAME

Use LOG_BASENAME in place of the application name in the name of log files in the log directory. See the description of the --log-directory switch. This switch does not affect the name of the process-ID file.

--log-post-rotate=COMMAND

Run COMMAND on the previous day's log file after log rotation. When this switch is not specified, the previous day's log file is compressed with gzip(1). When the switch is specified and COMMAND is the empty string, no action is taken on the log file. Each occurrence of the string %s in COMMAND is replaced with the full path to the log file, and each occurrence of %% is replaced with %. If any other character follows %, rwflowpack exits with an error. Specifying this switch without also using --log-directory is an error.

--pidfile=FILE_PATH

Set the complete path to the file in which rwflowpack writes its process ID (PID) when it is running as a daemon. No PID file is written when --no-daemon is given. When this switch is not present, no PID file is written unless the --log-directory switch is specified, in which case the PID is written to LOGPATH/rwflowpack.pid.

--no-chdir

Do not change directory to the root directory. When rwflowpack becomes a daemon process, it changes its current directory to the root directory so as to avoid potentially running on a mounted file system. Specifying --no-chdir prevents this behavior, which may be useful during debugging. The application does not change its directory when --no-daemon is given.

--no-daemon

Force rwflowpack to run in the foreground---it does not become a daemon process. This may be useful during debugging.

Help Options

--verify-sensor-config

--verify-sensor-config=VERBOSE

Verify that the syntax of the sensor configuration file is correct and then exit rwflowpack. If the file is incorrect or if it does not define any sensors, an error message is printed and rwflowpack exits abnormally. If the file is correct and no argument is provided to the --verify-sensor-config switch, rwflowpack simply exits with status 0. If an argument (other than the empty string and 0) is provided to the switch, the names of the probes and sensors found in the sensor configuration file are printed to the standard output, and then rwflowpack exits.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

ENVIRONMENT

SILK_IPFIX_PRINT_TEMPLATES

When set to 1, rwflowpack writes messages to the log file describing each IPFIX and NetFlow v9 template it receives. This is equivalent to adding show-templates to the log-flags setting for each probe in the sensor.conf file. See the sensor.conf(5) manual page for the format of these messages. Since SiLK 3.8.2.

SILK_LIBFIXBUF_SUPPRESS_WARNINGS

When set to 1, rwflowpack disables all warning messages generated by libfixbuf. These warning messages include out-of-sequence packets, data records not having a corresponding template, record count discrepancies, and issues decoding list elements. Since SiLK 3.10.0.

SILK_CONFIG_FILE

This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR

This environment variable specifies the root directory of data repository. When the output-mode is sending, rwflowpack may use this environment variable when searching for the SiLK site configuration file. See the "FILES" section for details.

SILK_PATH

This environment variable gives the root of the install tree. When searching for configuration files, rwflowpack may use this environment variable. See the "FILES" section for details.

SILK_PLUGIN_DEBUG

When set to 1, rwflowpack print status messages to the standard error as it tries to open the packing logic plug-in.

FILES

sensor.conf

The location of this file must be specified by the --sensor-configuration switch. This file specifies probe blocks that tell rwflowpack how to capture data when the --input-mode is stream. The file also contains sensor blocks that map probes to sensors and that the packing-logic uses to determine the category of each flow record. The syntax of this file is described in the sensor.conf(5) manual page.

${SILK_CONFIG_FILE}

ROOT_DIRECTORY/silk.conf

${SILK_PATH}/share/silk/silk.conf

${SILK_PATH}/share/silk.conf

/usr/share/silk/silk.conf

/usr/share/silk.conf

Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided. When rwflowpack is running in local-storage output-mode, ROOT_DIRECTORY/ is the directory specified to the --root-directory switch. When the output-mode is sending, ROOT_DIRECTORY/ is either the value specified in the SILK_DATA_ROOTDIR environment variable or the default data repository directory compiled into rwflowpack (/data).

${SILK_PATH}/lib64/silk/

${SILK_PATH}/lib64/

${SILK_PATH}/lib/silk/

${SILK_PATH}/lib/

/usr/lib64/silk/

/usr/lib64/

/usr/lib/silk/

/usr/lib/

Directories that rwflowpack checks when attempting to load the packing-logic plug-in.

SEE ALSO

SiLK Installation Handbook, sensor.conf(5), silk.conf(5), packlogic-twoway(3), packlogic-generic(3), flowcap(8), rwfilter(1), rwflowappend(8), rwreceiver(8), rwsender(8), rwpollexec(8), rwpdu2silk(1), rwpackchecker(8), silk(7), gzip(1), yaf(1), dlopen(3), zlib(3), syslog(3)

NOTES

As SiLK 3.6.0, the incremental-files output-mode should be used in place of the sending output-mode that existed in prior releases of rwflowpack. See "Output Modes" for details.

For administrators that use the sending output-mode in SiLK 3.5 or older and upgrade to SiLK 3.6 or later: Any incremental files that the older version of rwflowpack leaves in the incremental-directory will not be moved to the sender-directory by the new version of rwflowpack. Those files will need to be moved by hand.

rwflowpack does not check the integrity of the data file before appending records to it.

When the disk becomes full or other write errors occur, rwflowpack may leave partially written records or partially written compressed blocks in the files it has open. For each file where a partially written compressed block remains and additional compressed blocks are appended, the newly appended compressed blocks are not readable by other SiLK tools. For each file where a partially written record remains and additional records are appended, other SiLK tools will read the unaligned data as if it were aligned and produce garbage records. Partially writes may also occur if rwflowpack is suddenly killed (e.g., by kill -9).

When a write error occurs, rwflowpack may leave a zero byte file in the data repository. Such files do affect the exit status of rwfilter(1), though rwfilter warns about being unable to read the header from the file.

rwflowpack obtains an advisory write lock on the hourly file it is writing, allowing multiple rwflowpack processes to write to the same hourly file. File locking may be disabled by using the --no-file-locking switch. If this switch is enabled, the administrator must ensure that multiple rwflowpack processes do not attempt to write to the same hourly file simultaneously.