NAME

sensor.conf - Sensor Configuration file for rwflowpack and flowcap

DESCRIPTION

As part of collecting flow data, the rwflowpack(8) and flowcap(8) daemons need to know what type of data they are collecting and how to collect it (e.g., what port they need to open to collect the data). In addition, the rwflowpack daemon needs information on how to categorize the flow (for example, to label the flows collected at a border router as incoming or outgoing). The Sensor Configuration file, sensor.conf, contains this information, and this manual page describes the syntax of the file.

The sensor.conf file may have any name, and it does not have to reside in a particular location. The location of the the file is specified by the --sensor-configuration switch to rwflowpack and flowcap.

The Sensor Configuration File defines the following concepts:

probe: A probe collects a stream of flow-data from a flow generator, where the generator could be a YAF sensor listening to live traffic, a Cisco router producing NetFlow v5, or software creating flow information from packet capture (pcap(3)) files generated by tcpdump(1). When defining a probe, you must specify a name for the probe, the probe's type, and the address and port the probe uses to collect flow data.
sensor: A sensor represents a logical collection point for the purposes of analyis. Usually one sensor corresponds to one probe; however, a sensor could be comprised of multiple probes, or the flow data collected at a single proble could be divided into multiple sensors. The sensor also contains the information that rwflowpack uses to categorize each flow depending on how the flow moves between networks at the collection point. Since the sensors and the categories (known as class and type) are also used for analysis, they are defined in the Site Configuration file, as described silk.conf(5). In the Sensor Configuration file, you specify how sensors map to probes, and give the rules required to categorize the data.

SYNTAX

When parsing the Sensor Configuration file, blank lines are ignored. At any location in a line, the character # indicates the beginning of a comment, which continues to the end of the line. These comments are ignored.

All other lines begin with optional leading whitespace, a command name, and one or more arguments to the command. Command names are a sequence of non-whitespace characters, not including the character #. Arguments are textual atoms: any sequence of non-whitespace, non-# characters, including numerals and punctuation.

There are three contexts for commands: top-level, probe block, and sensor block. The probe block and sensor block contexts are used to describe individual features of probes and sensors.

The valid commands for each context are described below.

Top-Level Commands

In addition to the commands to begin a probe or sensor block, the top-level context supports the follow command.

include ``path'': The include command is used to include the contents of another file whose location is path. This may be used to separate large configurations into logical units.

Probe Block

The first command below is used at top-level to begin a probe definition block, and the remaining commands are accepted within the probe context. Note that one and only one of listen-on-port, listen-on-unix-socket, or read-from-file must be specified.

probe probe-name probe-type

The probe command begins a new probe block and it continues to the end probe command. The arguments to the probe command are the name of the probe being defined and the probe type. The probe-name must be unique among all probes. It must begin with a letter, and it may not contain whitespace characters or the character /. When a probe is associated with a single sensor, it is good practice to give the probe the same name as the sensor; however, when rwflowpack from SiLK-1.0 is reading flowcap files created by earlier versions of SiLK, the probe name must follow the rules specified in the rwflowpack(8) manual page. The probe-type must be one of the following:

netflow: A NetFlow probe processes NetFlow V5 protocol data units (PDU) that the daemon reads from a UDP port or from a file. NetFlow may be generated by a router or by software that reads packet capture (pcap(3)) data and generates NetFlow V5 records.
ipfix: An IPFIX probe processes Internal Protocol Flow Information eXchange records that the daemon reads from an IPFIX source such as YAF. To support IPFIX probes, SiLK must be built with support for the external library libfixbuf. Both YAF and libfixbuf are available from http://tools.netsa.cert.org/.

end probe

The end probe command ends the definition of a probe. Following an end probe command, top-level commands are again accepted.

accept-from-host host-name

This optional command specifies the IP or name of the host that is allowed to connect to the port where the probe is listening. When this command is not present, any host may connect. The command may only be specified when the listen-on-port command is also present. When listening for NetFlow, this attribute would be the IP address of the router as seen from the machine running rwflowpack or flowcap.

listen-as-host ip-address

This optional command is used on a multi-homed machine to specify the address the probe should listen on. Its value is a network IP address in dotted-decimal notation. If not present, the program will listen on all the machine's addresses. The command may only be specified when the listen-on-port command is also present. For listening to NetFlow, the value would be the ip-address given by Cisco ISO command

  ip flow-export [ip-address] [port]

listen-on-port port

This command specifies the network port where the probe should collect flow data. When listening to NetFlow from a Cisco router, this is the port given by the Cisco ISO command

  ip flow-export [ip-address] [port]

listen-on-unix-socket path-to-unix-socket

The value contains the path name to a UNIX domain socket where the flow generator writes its data. The parent directory of path-to-unix-socket must exist.

log-flags { none | all | { bad | missing ... } }

This optional command gives specific logging instructions for the probe. Some probe types will keep track of the sequence numbers in the packets they receive; these probes will write messages to the log file about missing or bad packets. By default, all messages are written by the probes that keep track of this information, but sometimes it is useful to reduce the verbosity of this information or eliminate it entirely. The log-flags command accepts a list of the following values:

none: Log nothing.
all: Log all the conditions listed below; this is the default.
bad: Write messages about bad packets. An example of a bad packet is a NetFlow v5 PDU that reports no flow records.
missing: Examine the sequence numbers and write messages about missing packets.

priority value

This optional command is used by flowcap when it is running as a server. The value may be an integer between 1 and 100, inclusive, where a value of 1-50 is considered low priority, and 51-100 is high. flowcap accumulates flow data from probes into files, and these files are closed after a certain amount of time. Closed high-priority files are always sent to a client before closed low-priority files.

protocol { tcp | udp | sctp }

This command, required when listen-on-port is given, specifies whether the port on which to listen is an sctp, tcp, or udp port. IPFIX probes support all three types; the only permitted value for all other probe types is udp.

read-from-file { /dev/null | file-path }

When this command is given, rwflowpack will read NetFlow v5 records from the specified path on the file system. flowcap does not use this value. The file's length should be an integer multiple of 1464 bytes, where 1464 is the maximum length of the NetFlow v5 PDU. Each 1464 block should contain the 24-byte NetFlow v5 header and space for thirty 48-byte flow records, even if fewer NetFlow records are valid. You can override this command by giving the --netflow-file switch to rwflowpack, but the command must still be present in the Sensor Configuration file. In this case, we recommend you use /dev/null as the file name.

Sensor Block

The information from the sensor block is used by rwflowpack to determine how to categorize a flow; that is, in which file the flow record is stored.

When the Sensor Configuration file is used with flowcap, no sensors need to be defined. In fact, flowcap completely ignores all text inside the Sensor Block.

The logic to categorize a flow for the twoway site is described below in the PACKING LOGIC section. The packing logic is normally loaded into rwflowpack as a run-time plug-in. The packing logic for your version of rwflowpack may be different.

The first command below is used at top-level to begin a sensor configuration block, and the remaining commands are accepted within the sensor block.

sensor sensor-name: The sensor command begins a new sensor configuration block. It takes as an argument the name of the sensor being configured, and that sensor must be defined in the Site Configuration file (see silk.conf(5)). A sensor block is closed with the end sensor command.
end sensor: The end sensor command ends the configuration of a sensor. Following an end sensor command, top-level commands are again accepted.
probe-type-probes probe-name [probe-name ...]: This command associates the listed probe names of the given probe type with the sensor. The probes do not have to be defined before they are used. (Note this also means that a mistyped probe-name will not be detected.) For example, netflow-probes S1 says that S1 is a netflow probe; whenever flow data arrives on the S1 probe, the sensor associated with the probe notices that data is available and processes it.
network-name-ipblocks {cidr-block [cidr-block...] | remainder}: The value of this command is a list of CIDR blocks specifying the IP-space of the network named network-name. If the value is the keyword remainder, the IP-space is taken as all IPs not assigned to other networks. The remainder keyword can only appear one time within a sensor block. network-name is a string defined in the packing logic plug-in that rwflowpack loads. Most plug-ins provide the external, internal, and null names, where internal refers to network being monitored, and external is everything else.
network-name-interfaces {interface [interface...] | remainder}: This command is similar to the above. The value is a list of positive integers from 0 to 65535 that represent the SNMP interface index(es) that connect to routers or machines on the network named network-name. If the value is the keyword remainder, the interface list is computed by finding all interface values not assigned to other networks. The command is only valid on sensors that are associated with netflow probes.
isp-ip ip-address [ip-address ...]: This optional command may be used for a sensor that processes data from a netflow probe. The value to the command is a list of IP addresses in dotted-decimal notation, where the IPs are the addresses of the NICs on the router. For traffic that doesn't leave the router (and thus was sent to the router's null-interface), some packing-logic plug-ins use these IPs to distinguish legitimate traffic for the router (e.g., routing protocol traffic, whose destination address would be in this list) from traffic that violated the router's access control list (ACL).

PACKING LOGIC

The packing logic plug-in that rwflowpack loads uses the values in the Sensor Block to determine how to categorize a flow and where on disk to store it. This section describes the how a flow's category (or type) is set when the logic in the pack-twoway.so file is used.

You may categorize any flow record, regardless of the type of probe that collected it, based on the IP addresses the flow contains and the IP space of the monitored network. Flows collected by a NetFlow probe may instead be categorized using the SNMP interface information in the flow.

The pack-twoway.so packing logic requires you specify for each sensor either the IP space outside the monitored network (external-ipblock) or, for sensors that use NetFlow probes, the SNMP interfaces that face outside the network (external-interfaces). In addition to specifying the external side, you may specify the values on the internal side (internal-ipblock or internal-interfaces). When the internal value is not specified, it is taken to be all values not used on the external side. It is an error to specify both the IP space and SNMP interfaces on a sensor.

When a router generates a flow record for packets that the router did not route (that never left the router), the outgoing SNMP interface is set to the null interface. A packet may be not-routed because the packet is meant for the router itself (e.g., a BGP message) or because it matches an ACL violation. On a CISCO router, the null interface has the destination ID of 0. You may specify the null-interfaces to use this destination in your packing logic. (For consistency, there is a null-ipblock that can be set.)

Note: As of SiLK-1.0, the null-interface must be explicitly set. It is no longer set by default.

Given a flow travelling between the external, internal, and null networks, the following rules specify the type that is assigned to the flow. This description uses IP-based splitting; for SNMP-based splitting, the logic is identical, except the SNMP interfaces are checked.

ext2ext: Contains flows where both the source IP and destination IP are listed in the external-ipblock.
innull: Contains flows where the source IP is listed in the external-ipblock and the destination IP is listed in the null-ipblock.
in,inicmp,inweb: Contains flows where the source IP is listed in the external-ipblock and the destination IP is listed in the internal-ipblock. A flow's type is inicmp if the protocol is ICMP (i.e., 1). The flow's type is inweb if the protocol is TCP (6) and either the source port or destination port is one of 80, 443, 8080. Otherwise the flow's type is in.
int2int: Contains flows where both the source IP and destination IP are listed in the internal-ipblock.
outnull: Contains flows where the source IP is listed in the internal-ipblock and the destination IP is listed in the null-ipblock.
out,outicmp,outweb: Contains flows where the source IP is listed in the internal-ipblock and the destination IP is listed in the external-ipblock.
other: Contains flows where either

EXAMPLES

NetFlow Categorized by SNMP Interface

The following two blocks define a probe that listens on 9900/udp for NetFlow v5 from a router. The probe only accepts traffic originating from 172.16.22.22. The associated sensor uses the SNMP interfaces to categorize the flows, where traffic from 1 to 8 is in, 8 to 1 is out, 1 to 0 is innull, 8 to 8 is ext2ext, etc.

  probe S1 netflow
      listen-on-port 9900
      protocol udp
      accept-from-host 172.16.22.22
  end probe

  sensor S1
      netflow-probes S1
      external-interface 1
      internal-interface 8
      null-interface 0
  end sensor

NetFlow Categorized by IP Address

The probe is the same as above, but the flows are categorized based on the IP addresses. The internal network is defined as 128.2.0.0/16, and all other IPs are defined as external.

  probe S2 netflow
      listen-on-port 9901
      protocol udp
      accept-from-host 172.16.22.22
  end probe

  sensor S2
      netflow-probes S2
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder
  end sensor

IPFIX Categorized by IP Address

This example uses the same sensor configuration as above, but it uses an IPFIX probe to collect the flows on port 9902/tcp. Also, we instruct the probe to bind to address 192.168.1.92. Since there is no SNMP information for IPFIX flows, one must use CIDR blocks to categorize the flows.

  probe S3 ipfix
      listen-on-port 9902
      protocol tcp
      listen-as-host 192.168.1.92
  end probe

  sensor S3
      ipfix-probes S3
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder
  end sensor

NetFlow From a File Categorized by SMNP Interfaces

Instead of listening on a UDP port for NetFlow traffic, you can configure the probe to process a file containing NetFlow v5 PDUs. This example assumes you are running rwflowpack with the switches --input-mode=file --netflow-file=FILENAME. The --netflow-file switch overrides the read-from-file command on the probe.

  probe S4 netflow
      log-flags bad                     # ignore missing pkts
      read-from-file /dev/null          # use --netflow-file=<file>
  end probe

  sensor S4
      netflow-probes S4
      external-interface 2
      internal-interface 9
      null-interface 0
  end sensor

NOTES

You may need to use specific probe names when rwflowpack is processing files created by a pre-SiLK-1.0 version of flowcap. See the NOTES section of the rwflowpack man page for details.

The syntax of sensor.conf differs greatly between SiLK-0.11.x and SiLK-1.0. As of SiLK-1.0, defining temporary sensors in sensor.conf is no longer supported. Instead, the sensors should be defined in the silk.conf file.

SiLK-1.0 provides the update-sensor-conf script to upgrade older versions of sensor.conf to the new syntax. When making the change by hand, this old sensor-probe block:

 sensor-probe S01
      probe-name netflow
      probe-type netflow
      priority 8
      log-flags none all bad missing
      listen-on-port 9999
      protocol udp
      listen-as-host 192.168.1.1
      listen-on-unix-domain-socket /tmp/sock
      read-from-file /var/tmp/flow-file
      accept-from-host 172.16.22.22
      null-index 0
      isp-ip 10.10.10.10,10.10.10.11
      external-index 1,2,3,4
      internal-index 8,9
      null-ipblock 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder

becomes:

 probe S01 netflow
      priority 8
      log-flags none all bad missing
      listen-on-port 9999
      protocol udp
      listen-as-host 192.168.1.1
      listen-on-unix-domain-socket /tmp/sock
      read-from-file /var/tmp/flow-file
      accept-from-host 172.16.22.22
 end probe

 sensor S01
      netflow-probes S01
      isp-ip 10.10.10.10,10.10.10.11
      null-interface 0
      external-interface 1,2,3,4
      internal-interface 8,9
      null-ipblock 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder
 end sensor

Note the following:

probe-type

-probes

-index

-indexes

-interface

rwflowpack

null-interface

protocol

listen-on-port

end