This version is the SiLK-only version of the Analysis Pipeline. To process more types of records, notably YAF records, use version 5.x
The Analysis Pipeline processes SiLK Flow records, and its goals are to automate common tasks, to get closer to "real-time" reporting of events, and to feed interesting data to a security information and event manager (SIEM).
The Analysis Pipeline supports many types of analysis, including:
- Watch list alerting (did we see traffic from a known bad IP?)
- Thresholding (e.g., is total bytes over a limit?)
- Collection issues (is a sensor no longer reporting?)
- Beacon detection
- Passive FTP detection
- IPv6 tunnel detection
- Periodic updates of network statistics
Although the Analysis Pipeline can be run on a list of files provided on the command line, it is designed to be incorporated into the SiLK collection and packing infrastructure, where it can analyze every SiLK Flow record produced by rwflowpack as the records are being added to the SiLK data repository.
When a record matches an analysis, the Analysis Pipeline can output the record using libsnarf, an alerting library. Pipeline can also be configured to send alerts to a pipe-delimited textual format. Whether a record is output depends on how often the administrator has configured the Analysis Pipeline to issue that type of output. The administrator can easily configure a SIEM to process the output generated by the Analysis Pipeline.
Analytics for the Analysis Pipeline are specified in a configuration file, the details and syntax of which can be found here
