Monitoring for Large-Scale Networks

SiLK

flowcap

NAME

flowcap - Capture network flow data and write it to temporary files

SYNOPSIS

flowcap --destination-directory=DIR_PATH
      --sensor-configuration=FILENAME [--probes=NAME[,NAME...]]
      --max-file-size=SIZE [--fc-version=NUM]
      [--timeout=TIMEOUT] [--clock-time[=OFFSET]]
      [--freespace-minimum=SIZE] [--space-maximum-percent=NUM]
      [--compression-method=COMP_METHOD]
      { --log-destination=DESTINATION
        | --log-pathname=FILE_PATH
        | --log-directory=DIR_PATH [--log-basename=LOG_BASENAME]
          [--log-post-rotate=COMMAND] }
      [--log-level=LEVEL] [--log-sysfacility=NUMBER]
      [--pidfile=FILE_PATH] [--no-chdir] [--no-daemon]

Help options:

flowcap --sensor-configuration=FILE_PATH
      { --verify-sensor-config | --verify-sensor-config=VERBOSE }

flowcap --help

flowcap --version

DESCRIPTION

flowcap is a daemon that collects records from routers, flow meters, and devices that produce network flow data. The records are written in the SiLK Flow record format to temporary files on disk. flowcap may collect NetFlow records (versions 5 or 9), IPFIX records (Internet Protocol Flow Information eXport) such as those generated by yaf(1), or sFlow records.

The SiLK Flow files produced by flowcap are meant to be used only for temporary storage. For longer-term storage, the records should processed by the rwflowpack(8) daemon which assigns values to each record depending on where it was collected and writes the record to an hourly file that is stored in a directory tree.

As flowcap receives flow records, it stores them in files in the location specified by the --destination-directory switch. These files are closed on quantum boundaries, with one file per flow source per quantum. A quantum is either the amount of time represented by the --timeout switch or the file size represented by the --max-file-size switch, whichever is reached first.

To transfer the files to rwflowpack, flowcap works in tandem with the rwsender(8) program. rwsender polls the storage directory and sends the files it finds there to an rwreceiver(8) process for processing by rwflowpack.

flowcap produces files that are named PROBE_YYYYMMDDhhmmss.XXXXXX, where PROBE is the name of the probe, YYYY is the current year, MM is the current month, DD is the current day, hh is the current hour, mm is the current minute, ss is the current second, and XXXXXX is a random six-character string.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

For the following options, a SIZE may be given as an ordinary integer, or as a real number followed by a suffix K, M, G, or T, which represents the numerical value multiplied by 1,024 (kilo), 1,048,576 (mega), 1,073,741,824 (giga), and 1,099,511,627,776 (tera), respectively. For example, 1.5K represents 1,536 bytes, or one and one-half kilobytes.

General Configuration Switches

--destination-directory=DIR_PATH

Store aggregated packed flow files in this directory for processing by rwsender. DIR_PATH must be a complete directory path. This switch is required.

--sensor-configuration=FILENAME

Give the path to the configuration file that flowcap consults to determine how to collect flow records. The complete syntax of the configuration file is described in the sensor.conf(5) manual page; see also the SiLK Installation Handbook. This switch is required.

--probes=NAME[,NAME...]

Choose which of the probes described in the sensor configuration file will be used by flowcap. The default is to use all of the probes defined in the configuration file. This switch instructs flowcap to only use the specifically named probes.

--max-file-size=SIZE

Set the approximate maximum size of flowcap files to SIZE bytes. If a flowcap file exceeds SIZE bytes, it is closed and a new file will be created and used. In addition, before opening an output file, flowcap ensures there are SIZE bytes of free space available, and exits if there is not. This switch is required.

--timeout=TIMEOUT

Set the maximum duration that a flowcap output file remains open to TIMEOUT seconds. When the --clock-time switch is given, the first duration may be less than TIMEOUT seconds. If the --timeout switch is not specified, flowcap uses a default of 60 seconds.

--clock-time[=OFFSET]

Force flowcap to close its files at predictable times. When this switch is provided, flowcap closes its output files at OFFSET seconds after midnight (UTC of the current day) and at every TIMEOUT seconds thereafter. The default value of OFFSET is 0. For example, --timeout=900 --clock-time=300 causes flowcap to close its output files at the 05, 20, 35, and 50 minute points in each hour. Even with this switch, files are still be closed if they exceed the size specified by --max-file-size.

--fc-version=NUM

Choose the record version for the files of IPv4 flow records that flowcap produces. Valid values are 2, 3, 4, and 5, and the default is 5. This switch is ignored for probes that support IPv6 addresses.

--freespace-minimum=SIZE

Set the minimum free space to maintain on the file system where the --destination-directory is located. By default, flowcap assumes that it has full rein over the file system on which it writes its files. The default is to leave 1GB of free space. If flowcap fills this space, it exits. Flows arriving during this time will be lost. See also --space-maximum-percent.

--space-maximum-percent=NUM

Use no more than this percentage of the file system containing the --destination-directory. The default is to use no more than 98% of the file system. If flowcap fills this space, it exits. See also --freespace-minimum.

--compression-method=COMP_METHOD

Specify the compression library to use when writing output files. When no compression method is specified, flowcap files are compressed using the best method, regardless of the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

none: Do not compress the output using an external library.
zlib: Use the zlib(3) library for compressing the output. Using zlib produces the smallest output files at the cost of speed.
lzo1x: Use the lzo1x algorithm from the LZO real time compression library for compression. This compression provides good compression with less memory and CPU overhead.
snappy: Use the snappy library for compression. This compression provides good compression with less memory and CPU overhead. Since SiLK 3.13.0.
best: Use lzo1x if available, otherwise use snappy if available, otherwise use zlib if available.

--verify-sensor-config

--verify-sensor-config=VERBOSE

Verify that the syntax of the sensor configuration file is correct and then exit flowcap. If the file is incorrect or if it does not define any probes, an error message is printed and flowcap exits abnormally. If the file is correct and no argument is provided to the --verify-sensor-config switch, flowcap simply exits with status 0. If an argument (other than the empty string and 0) is provided to the switch, the names of the probes found in the sensor configuration file are printed to the standard output, and then flowcap exits.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

Logging and Daemon Configuration Switches

The switches in this section determine the type of log messages that flowcap generates and where those messages are written.

One of the following switches are required:

--log-destination=DESTINATION

Specify the destination where logging messages are written. When DESTINATION begins with a slash /, it is treated as a file system path and all log messages are written to that file; there is no log rotation. When DESTINATION does not begin with /, it must be one of the following strings:

none: Messages are not written anywhere.
stdout: Messages are written to the standard output.
stderr: Messages are written to the standard error.
syslog: Messages are written using the syslog(3) facility.
both: Messages are written to the syslog facility and to the standard error (this option is not available on all platforms).

--log-directory=DIR_PATH

Use DIR_PATH as the directory to which the log files are written; DIR_PATH must be a complete directory path. The log files have the form

DIR_PATH/LOG_BASENAME-YYYYMMDD.log

where YYYYMMDD is the current date and LOG_BASENAME is the application name or the value passed to the --log-basename switch when provided. The log files are rotated: At midnight local time, a new log is opened, the previous file is closed, and the command specified by --log-post-rotate is invoked on the previous day's log file. (Old log files are not removed by flowcap; the administrator should use another tool to remove them.) When this switch is provided, a process-ID file (PID) is also written in this directory unless the --pidfile switch is provided.

--log-pathname=FILE_PATH

Use FILE_PATH as the complete path to the log file. The log file is not rotated.

The following switches are optional:

--log-level=LEVEL: Set the severity of messages that are logged. The levels from most severe to least are: emerg, alert, crit, err, warning, notice, info, debug. The default is info.
--log-sysfacility=NUMBER: Set the facility that syslog(3) uses for logging messages. This switch takes a number as an argument. The default is a value that corresponds to LOG_USER on the system where flowcap is running. This switch produces an error unless --log-destination=syslog is specified.
--log-basename=LOG_BASENAME: Use LOG_BASENAME in place of the application name in the name of log files in the log directory. See the description of the --log-directory switch. This switch does not affect the name of the process-ID file.
--log-post-rotate=COMMAND: Run COMMAND on the previous day's log file after log rotation. When this switch is not specified, the previous day's log file is compressed with gzip(1). When the switch is specified and COMMAND is the empty string, no action is taken on the log file. Each occurrence of the string %s in COMMAND is replaced with the full path to the log file, and each occurrence of %% is replaced with %. If any other character follows %, flowcap exits with an error. Specifying this switch without also using --log-directory is an error.
--pidfile=FILE_PATH: Set the complete path to the file in which flowcap writes its process ID (PID) when it is running as a daemon. No PID file is written when --no-daemon is given. When this switch is not present, no PID file is written unless the --log-directory switch is specified, in which case the PID is written to LOGPATH/flowcap.pid.
--no-chdir: Do not change directory to the root directory. When flowcap becomes a daemon process, it changes its current directory to the root directory so as to avoid potentially running on a mounted file system. Specifying --no-chdir prevents this behavior, which may be useful during debugging. The application does not change its directory when --no-daemon is given.
--no-daemon: Force flowcap to run in the foreground---it does not become a daemon process. This may be useful during debugging.

ENVIRONMENT

SILK_IPFIX_PRINT_TEMPLATES: When set to 1, flowcap writes messages to the log file describing each IPFIX and NetFlow v9 template it receives. This is equivalent to adding show-templates to the log-flags setting for each probe in the sensor.conf file. See the sensor.conf(5) manual page for the format of these messages. Since SiLK 3.8.2.
SILK_LIBFIXBUF_SUPPRESS_WARNINGS: When set to 1, flowcap disables all warning messages generated by libfixbuf. These warning messages include out-of-sequence packets, data records not having a corresponding template, record count discrepancies, and issues decoding list elements. Since SiLK 3.10.0.

FILES

sensor.conf: The location of this file must be specified by the --sensor-configuration switch. This file specifies probe blocks that tell flowcap how to capture data. The syntax of this file is described in the sensor.conf(5) manual page.